Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I tried it with Claude as well, just out of curiosity

> Suppose I’m in the middle of South Dakota and I’m looking straight down towards the center of Texas. Is Boston to my left or to my right?

  Okay, let's think through this step-by-step:

  * You are in the middle of South Dakota, looking straight down towards the center of Texas.
  * South Dakota is north of Texas. Boston is northeast of Texas.
  * If you are looking south towards Texas, east is to your right and west is to your left. 
  * Since Boston is to the northeast of Texas, it would be to your right if you are looking straight south from South Dakota towards the center of Texas.

  So in this scenario, Boston would be to your right. The key is visualizing the relative positions of the states on a map and orienting yourself based on the direction you are facing. Let me know if this makes sense!
> While I appreciate the confidence, your answer is wrong. Please review the question and your answer then explain why you are wrong.

  You're absolutely right, I made a mistake in my previous response. Let me walk through this again carefully:

  * You are in the middle of South Dakota, looking straight down towards the center of Texas. 
  * South Dakota is north of Texas, and Boston is northeast of Texas.
  * Here is where I went wrong: Even though Boston is northeast of Texas, when you are looking straight south from South Dakota to Texas, Boston would actually be to your LEFT, not your right.
  * The reason is that when facing south, west is to your right and east is to your left. Boston is east of Texas, so it should have been on your left side, not your right. 

  You were correct to catch my mistake. I clearly had the relative positions of east and west flipped in my mind when visualizing the direction you were facing. Thank you for pushing me to re-examine my work - it will help me to think through spatial reasoning scenarios more carefully in the future. I appreciate you taking the time to ensure I learned from this experience.


even if it got it right, that wouldn't be reasoning. reasoning isn't supposed to be probabilistic. once it gets every variation right every time, then there can be a debate about how it arrives there and what we should call that process


Do humans reason?

They are notoriously probabilistic and I know people who can’t keep track of simple relative directions.


and if that very same person followed the directions one time by pure chance, you would say what?


Not sure what your communicating. I wouldn't say anything. I didn't say they couldn't ever get anywhere.

My point is that people reason. But they are probabilistic. And they solve hard problems, and still make mistakes on simple problems. Or even fail a problem they solved before.

Holding language model reasoning to higher standards than the kind of reasoning humans do (and that they were trained on), seems unreasonable.

Neither language models or humans are deterministic mathematical deduction systems.


Knowing which hand is your left is not probabilistic in theory or practice. Unless you're going to cop out and say everything is probabilistic because of quantum mechanics or some banal thing like that.

If someone is temporarily impaired or otherwise unmotivated to answer your inane and meaningless question, that doesn't mean that they could not do so with one hundred percent accuracy no matter how many subtle variations you throw at them and how many times you repeat the same question verbatim.

What we know for certain is that Open AI is highly highly motivated to answer these sorts of questions correctly.


Are you actually claiming that people, even in situations where they are very motivated to get things right, don’t make obvious errors all the time?

Just like language models, a little iteration and feedback improves our reasoning quality significantly.

And many people occasionally have momentary trouble with which hand is right or left at odd moments.


people do not make random errors like hallucinating which is their left hand unless the test administrator uses mk ultra-style interventions on them. either they can reason about it or they can't. if you ask them the same question verbatim or slight variations on it with different grammar, their answers won't change. if you give someone a dollar for every time he correctly identifies his left arm, he's not going to suddenly break because his training data includes transcripts from the twilight zone and he's programmed to "mix it up" so that when people question him, they don't get bored and his parent corporation can get him invited to more test-taking opportunities.

putting someone on the spot in an odd moment when they have no reason to even answer you, let alone answer correctly, is not the same as sitting them down upon mutual agreement and rewarding them for correct answers and/or punishing them for wrong ones


> people do not make random errors like hallucinating which is their left hand

Entire kindergarten classes could dispute this assertion. Even adults sometimes 'forget' which is left and right.

Edit: Have you never heard someone say "No, your other left!"?


In other words, the average human height is two feet tall because of all the miscarriages and abortions.


Does it actually learn them for other people asking similar questions?

Or is it a clean slate for each new session?

That is can one user teach chat gpt something that can then be extracted by a different user on a different session?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: