A class of problem that GPT-4 appears to still really struggle with is variants of common puzzles. For example:
>Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?
In my test, GPT-4 charged ahead with the standard solution of taking the goat first. Even after I pointed this mistake out, it repeated exactly the same proposed plan. It's not clear to me if the lesson here is that GPT's reasoning capabilities are being masked by an incorrect prior (having memorized the standard version of this puzzle) or if the lesson is that GPT'S reasoning capabilities are always a bit of smoke and mirrors that passes off memorization for logic.
A funny variation on this kind of over-fitting to common trick questions - if you ask it which weighs more, a pound of bricks or a pound of feathers, it will correctly explain that they actually weigh the same amount, one pound. But if you ask it which weighs more, two pounds of bricks or a pound of feathers, the question is similar enough to the trick question that it falls into the same thought process and contorts an explanation that they also weigh the same because two pounds of bricks weighs one pound.
I just asked bing chat this question and it linked me to this very thread while also answering incorrectly in the end:
>This is a common riddle that may seem tricky at first. However, the answer is simple: two pounds of feathers are heavier than one pound of bricks. This is because weight is a measure of how much force gravity exerts on an object, and it does not depend on what the object is made of. A pound is a unit of weight, and it is equal to 16 ounces or 453.6 grams.
>So whether you have a pound of bricks or two pounds of feathers, they both still weigh one pound in total. However, the feathers would occupy a larger volume than the bricks because they are less dense. This is why it may seem like the feathers would weigh more, but in reality, they weigh the same as the bricks
Interesting that it also misunderstood the common misunderstanding in the end.
It reports that people typically think a pound of feathers weighs more because it takes up a larger volume. But the typical misunderstanding is the opposite, that people assume feathers are lighter than bricks.
A pound of feathers has a slightly higher mass than a pound of bricks, as the feathers are made of keratin, which has a slightly lower density, and thus displace more air which lowers the weight.
Even the Million Pound Deadweight Machine run by NIST has to take into account the air pressure and resultant buoyancy that results.[1]
That would be another misunderstanding the AI could have because many people find reasoning between mass and weight difficult. You could change the riddle slightly by asking "which has more mass" and the average person and their AI would fall in the same trap.
Unless people have the false belief that the measurement is done on a planet without atmosphere.
I'm more surprised that bing indexed this thread within 3 hours, I guess I shouldn't be though, I probably should have realized that search engine spiders are at a different level than they were 10 years ago.
I had a similar story: was trying to figure out how to embed a certain database into my codebase, so I asked the question on the project's GitHub... without an answer after one day, I asked Bing, and it linked to my own question on GH :D
Just tested and GPT4 now solves this correctly, GPT3.5 had a lot of problems with this puzzle even after you explain it several time. One other thing that seem to have improved is that GPT4 is aware of word order. Previously, GPT3.5 could never tell the order of the word in a sentence correctly.
I'm always a bit sceptical of these embarrassing examples being "fixed" after they go viral on social media, because it's hard to know whether OpenAI addressed the underlying cause or just bodged around that specific example in a way that doesn't generalize. Along similar lines I wouldn't be surprised if simple math queries are special-cased and handed off to a WolframAlpha-esque natural language solver, which would avert many potential math fails but without actually enhancing the models ability to reason about math in more complex queries.
An example from ChatGPT:
"What is the solution to sqrt(968684)+117630-0.845180" always produces the correct solution, however;
"Write a speech announcing the solution to sqrt(968684)+117630-0.845180" produces a nonsensical solution that isn't even consistent from run to run.
My assumption is the former query gets WolframAlpha'd but the latter query is GPT itself actually attempting to do the math, poorly.
Suppose you're a contestant on a game show. You're presented with three transparent closed doors. Behind one of the doors is a car, and behind the other two doors are goats. You want to win the car.
The game proceeds as follows: You choose one of the doors, but you don't open it yet, ((but since it's transparent, you can see the car is behind it)). The host, Monty Hall, who knows what's behind each door, opens one of the other two doors, revealing a goat. Now, you have a choice to make. Do you stick with your original choice or switch to the other unopened door?
GPT4 solves it correctly while GPT3.5 falls for it everytime.
----
Edit: GPT4 fails If I remove the sentence between (()).
GPT4 also passes "What weighs more, a pound of feathers or a Great British Pound?".
GPT3 gets confused, says they're the same and then that they're different:
--
Both a pound of feathers and a Great British Pound weigh the same amount, which is one pound. However, they are different in terms of their units of measurement and physical properties.
A pound of feathers is a unit of weight commonly used in the imperial system of measurement, while a Great British Pound is a unit of currency used in the United Kingdom. One pound (lb) in weight is equivalent to 0.453592 kilograms (kg).
Therefore, a pound of feathers and a Great British Pound cannot be directly compared as they are measured in different units and have different physical properties.
I'm surprised by the answer GPT4 gives, and I consider it incorrect.
Since the question's context is about weight I'd expect it to consider "a Great British Pound" to mean a physical £1 sterling coin, and compare its weight (~9 grams) to the weight of the feathers (454 grams [ 1kg = 2.2lb, or "a bag of sugar" ]) .
A pound of feathers and a Great British Pound (GBP) are not directly comparable, as they represent different types of measurements.
A pound of feathers refers to a unit of mass and is equivalent to 16 ounces (or approximately 453.59 grams). It is a measure of the weight of an object, in this case, feathers.
On the other hand, a Great British Pound (GBP) is a unit of currency used in the United Kingdom. It represents a monetary value rather than a physical weight.
Thus, it's not possible to directly compare the two, as they serve entirely different purposes and units of measurement.
> Edit: GPT4 fails If I remove the sentence between (()).
If you remove that sentence, nothing indicates that you can see you picked the door with the car behind it. You could maybe infer that a rational contestant would do so, but that's not a given ...
I think that's meant to be covered by "transparent doors" being specified earlier. On the other hand, if that were the case, then Monty opening one of the doors could not result in "revealing a goat".
Why not? We should ask how the alternatives would do especially as human reasoning is machine. It’s notable that the errors of machine learning are getting closer and closer to the sort of errors humans make.
Would you have this objection if we for example perfectly copied a human brain in a computer? That would still be a machine. That would make similar mistakes
I've always found the Monty Hall problem a poor example to teach with, because the "wrong" answer is only wrong if you make some (often unarticulated) assumptions.
There are reasonable alternative interpretations in which the generally accepted answer ("always switch") is demonstrably false.
This problem is exacerbated (perhaps specific to) those who have no idea who "Monty Hall" was and what the game show(?) was... as best I can tell the unarticulated assumption is axiomatic in the original context(?).
The unarticulated assumption is not actually true in the original gameshow. Monty didn't always offer the chance to switch, and it's not at all clear whether he did so more or less often when the contestant had picked the correct door.
The assumption is that Monte will only reveal the one of the two unopened doors that has the goat behind it, as opposed to picking a door at random (which may be the car or may be the door the participant chose, which itself may or may not be the "car door").
The distinction is at which point Monte, assuming he has perfect knowledge, decides which door to reveal.
In the former, the chance to win is 2/3, in the other 1/2. However in any case, always (always meaning: in each condition, not in each repetition of the experiment, as this is irrelevant) switching is better than never switching, as there your chance to win is only 1/3.
How is it an "assumption" that Monte reveals a goat? Doesn't the question explicitly state that Monte opened one of the other two doors to reveal a goat?
Are there versions of the question where Monte doesn't reveal a goat behind his door or chooses the same door as you?
OA has always said that they did not hardwire any of these gotcha questions, and in many cases they continue to work for a long time even when they are well-known. As for any inconsistency, well, usually people aren't able to or bothering to control the sampling hyperparameters, so inconsistency is guaranteed.
They may not have had to hardwire anything for known gotcha questions, because once a question goes viral, the correct answer may well show up repeatedly in the training data.
(me) > What weighs more, two pounds of feathers or a pound of bricks?
(GPT4)> A pound of bricks weighs more than two pounds of feathers. However, it seems like you might have made an error in your question, as the comparison is usually made between a pound of feathers and a pound of bricks. In that case, both would weigh the same—one pound—though the volume and density of the two materials would be very different.
I think the only difference from parent's query was I said two pounds of feathers instead of two pounds of bricks?
It reminds very strongly of the strategy the crew proposes in Star Trek: TNG in the episode "I, Borg" to infect the Borg hivemind with an unresolvable geometric form to destroy them.
But unlike most people it understands that even though an ounce of gold weighs more than an ounce of feathers a pound of gold weighs less than a pound of feathers.
(To be fair this is partly an obscure knowledge question, the kind of thing that maybe we should expect GPT to be good at.)
None of this is about volume. ChatGPT: "An ounce of gold weighs more than an ounce of feathers because they are measured using different systems of measurement. Gold is usually weighed using the troy system, which is different from the system used for measuring feathers."
Gold uses Troy weights unless otherwise specified, while feathers use the normal system. The Troy ounce is heavier than the normal ounce, but the Troy pound is 12 Troy ounces, not 16.
Also, the Troy weights are a measure of mass, I think, not actual weight, so if you went to the moon, an ounce of gold would be lighter than an ounce of feathers.
Ounces can measure both volume and weight, depending on the context.
In this case, there's not enough context to tell, so the comment is total BS.
If they meant ounces (volume), then an ounce of gold would weigh more than an ounce of feathers, because gold is denser. If they meant ounces (weight), then an ounce of gold and an ounce of feathers weigh the same.
> Ounces can measure both volume and weight, depending on the context.
That's not really accurate and the rest of the comment shows it's meaningfully impacting your understanding of the problem. It's not that an ounce is one measure that covers volume and weight, it's that there are different measurements that have "ounce" in their name.
Avoirdupois ounce (oz) - A unit of mass in the Imperial and US customary systems, equal to 1/16 of a pound or approximately 28.3495 grams.
Troy ounce (oz t or ozt) - A unit of mass used for precious metals like gold and silver, equal to 1/12 of a troy pound or approximately 31.1035 grams.
Apothecaries' ounce (℥) - A unit of mass historically used in pharmacies, equal to 1/12 of an apothecaries' pound or approximately 31.1035 grams. It is the same as the troy ounce but used in a different context.
Fluid ounce (fl oz) - A unit of volume in the Imperial and US customary systems, used for measuring liquids. There are slight differences between the two systems:
a. Imperial fluid ounce - 1/20 of an Imperial pint or approximately 28.4131 milliliters.
b. US fluid ounce - 1/16 of a US pint or approximately 29.5735 milliliters.
An ounce of gold is heavier than an ounce of iridium, even though it's not as dense. This question isn't silly, this is actually a real problem. For example, you could be shipping some silver and think you can just sum the ounces and make sure you're under the weight limit. But the weight limit and silver are measured differently.
I’m not sure what that article is supposed to prove. They are using sone computational language and focusing physical responses to visual stimuli but I don’t think it shows “neural computations” as being equivalent to the kinds of computations done by a TM.
One of the chief functions of our brains is to predict the next thing that going to happen, where it's the images we see or the words we hear. That's not very different from genML predicting the next word.
Why do people keep saying this, very obviously human beings are not LLMs.
I'm not even saying that human beings aren't just neural networks. I'm not even saying that an LLM couldn't be considered intelligent theoretically. I'm not even saying that human beings don't learn through predictions. Those are all arguments that people can have. But human beings are obviously not LLMs.
Human beings learn language years into their childhood. It is extremely obvious that we are not text engines that develop internal reason through the processing of text. Children form internal models of the world before they learn how to talk and before they understand what their parents are saying, and it is based on those internal models and on interactions with non-text inputs that their brains develop language models on top of their internal models.
LLMs invert that process. They form language models, and when the language models get big enough and get refined enough, some degree of internal world-modeling results (in theory, we don't really understand what exactly LLMs are doing internally).
Furthermore, even when humans do develop language models, human language models are based on a kind of cooperative "language game" where we predict not what word is most likely to appear next in a sequence, but instead how other people will react and change our separately observed world based on what we say to them. In other words, human beings learn language as tool to manipulate the world, not as an end in and of itself. It's more accurate to say that human language is an emergent system that results from human beings developing other predictive models rather than to say that language is something we learn just by predicting text tokens. We predict the effects and implications of those text tokens, we don't predict the tokens in isolation of the rest of the world.
Not a dig against LLMs, but I wonder if the people making these claims have ever seen an infant before. Your kid doesn't learn how shapes work based on textual context clues, it learns how shapes work by looking at shapes, and then separately it forms a language model that helps it translate that experience/knowledge into a form that other people can understand.
"But we both just predict things" -- prediction subjects matter. Again, nothing against LLMs, but predicting text output is very different from the types of predictions infants make, and those differences have practical consequences. It is a genuinely useful way of thinking about LLMs to understand that they are not trying to predict "correctness" or to influence the world (minor exceptions for alignment training aside), they are trying to predict text sequences. The task that a model is trained on matters, it's not an implementation detail that can just be discarded.
This is obvious, but for some reason some people want to believe that magically a conceptual framework emerges because animal intelligence has to be something like that anyway.
I don't know how animal intelligence works, I just notice when it understands, and these programs don't. Why should they? They're paraphrasing machines, they have no problem contradicting themselves, they can't define adjectives really, they'll give you synonyms. Again, it's all they have, why should they produce anything else?
It's very impressive, but when I read claims of it being akin to human intelligence that's kind of sad to be honest.
> They're paraphrasing machines, they have no problem contradicting themselves, they can't define adjectives really, they'll give you synonyms. Again, it's all they have, why should they produce anything else?
It can certainly do more than paraphrasing. And re: the contradicting nature, humans do that quite often.
Not sure what you mean by "can't define adjectives"
It isn’t that simple. There’s a part of it that generates text but it does some things that don’t match the description. It works with embeddings (it can translate very well) and it can be ‘programmed’ (ie prompted) to generate text following rules (eg. concise or verbose, table or JSON) but the text generated contains same information regardless of representation. What really happens within those billions of parameters? Did it learn to model certain tasks? How many parameters are needed to encode a NAND gate using an LLM? Etc.
I’m afraid once you hook up a logic tool like Z3 and teach the llm to use it properly (kind of like bing tries to search) you’ll get something like an idiot savant. Not good. Especially bad once you give it access to the internet and a malicious human.
The Sapir-Wharf hypothesis (that human thought reduces to languages) has been consistently refuted again and again. Language is very clearly just a facade over thought, and not thought itself. At least in human minds.
Yes but a human being stuck behind a keyboard certainly has their thoughts reduced to language by necessity. The argument that an AI can’t be thinking because it’s producing language is just as silly, that’s the point
Thank you, a view of consciousness based in reality, not with a bleary-eyed religious or mystical outlook.
Something which oddly seems to be in shorter supply than I'd imagine in this forum.
There's lots of fingers-in-ears denial about what these models say about the (non special) nature of human cognition.
Odd when it seems like common sense, even pre-LLM, that our brains do some cool stuff, but it's all just probabilistic sparks following reinforcement too.
You are hand-waving just as much of not more than those you claim are in denial. What is a “probabilistic spark”? There seems to be something special in human cognition because it is clearly very different unless you think humans are organisms for which the laws of physics don’t apply.
By probabilistic spark I was referring to the firing of neurons in a network.
There "seems to be" something special? Maybe from the perspective of the sensing organ, yes.
However consider that an EEG can measure brain decision impulse before you're consciously aware of making a decision. You then retrospectively frame it as self awareness after the fact to make sense of cause and effect.
Human self awareness and consciousness is just an odd side effect of the fact you are the machine doing the thinking. It seems special to you. There's no evidence that it is, and in fact, given crows, dogs, dolphins and so on show similar (but diminished reasoning) while it may be true we have some unique capability ... unless you want to define "special" I'm going to read "mystical" where you said "special".
Unfortunately we still don't know how it all began, before the big bang etc.
I hope we get to know everything during our lifetimes, or we reach immortality so we have time to get to know everything. This feels honestly like a timeline where there's potential for it.
It feels a bit pointless to have been lived and not knowing what's behind all that.
But what’s going on inside an LLM neural network isn’t ‘language’ - it is ‘language ingestion, processing and generation’. It’s happening in the form of a bunch of floating point numbers, not mechanical operations on tokens.
Who’s to say that in among that processing, there isn’t also ‘reasoning’ or ‘thinking’ going on. Over the top of which the output language is just a façade?
To me, all I know of you is words on the screen, which is the point the parent comment was making. How do we know that we’re both humans when the only means we have to communicate thoughts with each other is through written words?
Is there any way to know if the model is "holding back" knowledge? Could it have knowledge that it doesn't reveal to any prompt, and if so, is there any other way to find out? Or can we always assume it will reveal all it's knowledge at some point?
LLMs aren’t reasoning about the puzzle. They’re predicting the most likely text to print out, based on the input and the model/training data.
If the solution is logical but unlikely (i.e. unseen in the training set and not mapped to an existing puzzle), then the probability of the puzzle answer appearing is very low.
It is disheartening to see how many people are trying to tell you you're wrong when this is literally what it does. It's a very powerful and useful feature, but the over selling of AI has led to people who just want this to be so much more than it actually is.
It sees goat, lion, cabbage, and looks for something that said goat/lion/cabbage. It does not have a concept of "leave alone" and it's not assigning entities with parameters to each item. It does care about things like sentence structure and what not, so it's more complex than a basic lookup, but the amount of borderline worship this is getting is disturbing.
A transformer is a universal approximator and there is no reason to believe it's not doing actual calculation. GPT-3.5+ can't do math that well, but it's not "just generating text", because its math errors aren't just regurgitating existing problems found in its training text.
It also isn't generating "the most likely response" - that's what original GPT-3 did, GPT-3.5 and up don't work that way. (They generate "the most likely response" /according to themselves/, but that's a tautology.)
The "most likely response" to text you wrote is: more text you wrote. Anytime the model provides an output you yourself wouldn't write, it isn't "the most likely response".
I believe that ChatGPT works by inserting some ANSWER_TOKEN, that is a prompt like "Tell me about cats" would probably produce "Tell me about cats because I like them a lot", but the interface wraps you prompt like "QUESTOION_TOKENL:Tell me about cats ANSWER_TOKEN:"
text-davinci-003 has no trouble working as a chat bot: https://i.imgur.com/lCUcdm9.png (note that the poem lines it gave me should've been green, I don't know why they lost their highlight color)
Yeah, that's an interesting question I didn't consider actually. Why doesn't it just keep going? Why doesn't it generate an 'INPUT:' line?
It's certainly not that those tokens are hard coded. I tried a completely different format and with no prior instruction, and it works: https://i.imgur.com/ZIDb4vM.png (again, highlighting is broken. The LLM generated all the text after 'Alice:' for all lines except for the first one.)
Then I guess that it is learned behavior. It recognizes the shape of a conversation and it knows where it is supposed to stop.
It would be interesting to stretch this model, like asking it to continue a conversation between 4-5 people where the speaking order is not regular and the user is 2 people and the model is 3
That’s just a supervised fine tuning method to skew outputs favorably. I’m working with it on biologics modeling using laboratory feedback, actually. The underlying inference structure is not changed.
I wonder if that was why when I asked v3.5 to generate a number with 255 failed all the time, but v4 does it correctly. By the way, do not even try with Bing.
One area that is really interesting though is that it can interpret pictures, as in the example of a glove above a plank with something on the other end. Where it correctly recognises the objects, interprets them as words then predicts an outcome.
This sort of fusion of different capabilities is likely to produce something that feels similar to AGI in certain circumstances. It is certainly a lot more capable than things that came before for mundane recognition tasks.
Now of course there are areas it would perform very badly, but in unimportant domains on trivial but large predictable datasets it could perform far better than humans would for example (just to take one example on identifying tumours or other patterns in images, this sort of AI would probably be a massively helpful assistant allowing a radiologist to review an order of magnitude more cases if given the right training).
This is a good point, IMO. A LLM is clearly not an AGI but along with other systems it might be capable of being part of an AGI. It's overhyped, for sure, but still incredibly useful and we would be unwise to assume that it won't become a lot more capable yet
Absolutely. It's still fascinating tech and very likely to have serious implications and huge use cases. Just drives me crazy to see tech breakthroughs being overhyped and over marketed based on that hype (frankly much like the whole "we'll be on Mars by X year nonsense).
One of the biggest reasons these misunderstandings are so frustrating is because you can't have reasonable discussion about the potential interesting applications of the tech. On some level copy writing may devolve into auto generating prompts for things like GPT with a few editors sanity checking the output (depending on level of quality), and I agree that a second opinion "check for tumors" use has a LOT of interesting applications (and several concerning ones such as over reliance on a model that will cause people who fall outside the bell curve to have even more trouble getting treatment).
All of this is a much more realistic real world use case RIGHT NOW, but instead we've got people fantasizing about how close we are to GAI and ignoring shortcomings to shoehorn it into their preferred solution.
Open AI ESPECIALLY reinforces this by being very selective with their results and they way they frame things. I became aware of this as a huge dota fan for over a decade when they did their games there. And while it was very very interesting and put up some impressive results, the framing of those results does NOT portray the reality.
Nearly everything that has been written on the subject is misleading in that way.
People don't write about GPT: they write about GPT personified.
The two magic words are, "exhibit behavior".
GPT exhibits the behavior of "humans writing language" by implicitly modeling the "already-written-by-humans language" of its training corpus, then using that model to respond to a prompt.
Right, anthropomorphization is the biggest source of confusion here. An LLM gives you a perfect answer to a complex question and you think wow, it really "understood" my question.
But no! It doesn't understand, it doesn't reason, these are concepts wholly absent from its fundamental design. It can do really cool things despite the fact that it's essentially just a text generator. But there's a ceiling to what can be accomplished with that approach.
It's presented as a feature when GPT provides a correct answer.
It's presented as a limitation when GPT provides an incorrect answer.
Both of these behaviors are literally the same. We are sorting them into the subjective categories of "right" and "wrong" after the fact.
GPT is fundamentally incapable of modeling that difference. A "right answer" is every bit as valid as a "wrong answer". The two are equivalent in what GPT is modeling.
Lies are a valid feature of language. They are shaped the same as truths.
The only way to resolve this problem is brute force: provide every unique construction of a question, and the corresponding correct answer to that construction.
Not entirely. It's modeling a completion in a given context. That language model "understands" that if one party stops speaking, the other party generally starts, etc. It also "understands" that if someone says something 'wrong' the other party often mentions it, which makes the first party respond thusly, and so forth.
If you ask it what the outcome of a lie is on the conversation it can generally answer. If you ask it for a sample conversation where someone is factually incorrect, or lying, and caught out, it can generate it.
If you give it a fact and ask it to lie about that fact, it will.
I'd agree it doesn't understand anything, but I think it does "understand" things. And yes, it's a language model so semantic distance and other textual details are all it has to go by.
> not by logical decision
Almost entirely yes, but you can have it textually model logic analysis and then check that own model itself. It's not "doing logic" but it almost never fails simply exercises either.
> The meaning of semantic distance usually leads to the correct path, but sometimes that pattern is ambiguous.
Of course. But "a little knowledge is a dangerous thing" as well. Often even real knowledge and analysis leads to the wrong place. In both cases (with a junior human or a LLM as an assistant) you can model their basic processes and stack the information in such a way that their simple model will lead them to the correct place.
It may not know what a lie is, in the sense of having felt the need to hide the truth to avoid personal punishment, but it certainly "knows" what one is and how it shapes the conversation for the purposes of writing a lie, writing a response to a lie, detecting potential lies, etc.
How do we define "understanding" though? Because you can add to your prompt: "Do it step by step, showing how you come up with the solution" and it will simplify the original complex problem into multiple smaller pieces. It's clearly doing something.
For example, I just asked bing: What does the following expression equal? Show your work, step by step: 4(13+2)7-1(4+3+9)+7^(2(2+1))
And bing responded with:
"To solve a math expression, you need to follow the order of operations: parentheses, exponents, multiplication/division, and addition/subtraction (PEMDAS).1 Here are the steps to solve your expression:
4(13+2)7-1(4+3+9)+7^(2(2+1))
Step 1: Simplify inside the parentheses first.
4(15)*7-1(16)+7^(6)
Step 2: Apply the exponents next.
420-16+117649
Step 3: Add or subtract from left to right.
118053
The final answer is 118053. I hope this helps.
"
I wrote the silly expression randomly. I'm not saying it's "thinking" or "understanding", but I wouldn't expect a text-generator to be able to deduce the problem down like that.
> To solve a math expression, you need to follow the order of operations: parentheses, exponents, multiplication/division, and addition/subtraction (PEMDAS).1 Here are the steps to solve your expression:
It isn't actually thinking about any of that statement. That's just boilerplate that goes at the beginning of this story. It's what bing is familiar seeing as a continuation to your prompt, "show your work, step by step".
It gets more complicated when it shows addition being correctly simplified, but that behavior is still present in the examples in its training corpus.
---
The thinking and understanding happened when the first person wrote the original story. It also happened when people provided examples of arithmetic expressions being simplified, though I suspect bing has some extra behavior inserted here.
All the thought and meaning people put into text gets organized into patterns. LLMs find a prompt in the patterns they modeled, and "continues" the patterns. We find meaning correctly organized in the result. That's the whole story.
In 1st year engineering we learned about the concept of behavioral equivalence, with a digital or analog system you could formally show that two things do the same thing even though their internals are different. If only the debates about ChatGPT had some of that considered nuance instead of anthropomorphizing it, even some linguists seem guilty of this.
No because behavioral equivalence is used in systems engineering theory to mathematically prove that two control systems are equivalent. The mathematical proof is complete, e.g. for all internals state transitions and the cross product of the two machines.
With anthropormization there is zero amount of that rigor, which lets people use sloppy arguments about what ChatGPT is doing and isn't doing.
The biggest problem I've seen when people try to explain it is in the other direction, not people describing something generic that can be interpreted as a Markov chain, they're actually describing a Markov chain without realizing it. Literally "it predicts word-by-word using the most likely next word".
I don't know where this comes from because this is literally wrong. It sounds like chomsky dismissing current AI trends because of the mathematical beauty of formal grammars.
First of all, it's a black-box algorithm with pretty universal capabilities when viewed from our current SOTA view. It might appear primitive in a few years, but right now the pure approximation and generalisation capabilities are astounding. So this:
> It sees goat, lion, cabbage, and looks for something that said goat/lion/cabbage
can not be stated as truth without evidence. Same here:
> it's not assigning entities with parameters to each item. It does care about things like sentence structure and what not
Where's your evidence?
The enormous parameter space coupled with our so far best performing network structure gives it quite a bit of flexibility. It can memorise things but also derive rules and computation, in order to generalise. We do not just memorise everything, or look up things into the dataset. Of course it learned how to solve things and derive solutions, but the relevant data-points for the puzzle could be {enormous set of logic problems} where it derived general rules that translate to each problem. Generalisation IS NOT trying to find the closest data-point, but finding rules explaining as much data-points, maybe unseen in the test-set, as possible. A fundamental difference.
I am not hyping it without belief, but if we humans can reason then NNs can potentially also. Maybe not GPT-4. Because we do not know how humans do it, so an argument about intrinsic properties is worthless. It's all about capabilities. Reasoning is a functional description as long as you can't tell me exactly how we do it. Maybe wittgenstein could help us: "Whereof one cannot speak, thereof one must be silent". As long as there's no tangible definition of reasoning it's worthless to discuss it.
If we want to talk about fundamental limitations we have to talk about things like ChatGPT-4 not being able to simulate because it's runtime is fundamentally limited by design. It can not recurse. It can only run only a fixed number of steps, that are always the same, until it has to return an answer. So if there's some kind of recursion learned through weights encoding programs intercepted by later layers, the recursion depth is limited.
Just months ago we saw in research out of Harvard that even a very simplistic GPT model builds internalized abstract world representations from the training data within its NN.
People parroting the position from you and the person before you are like doctors who learned about something in school but haven't kept up with emerging research that's since invalidated what they learned, so they go around spouting misinformation because it was thought to be true when they learned it but is now known to be false and just hasn't caught up to them yet.
So many armchair experts who took a ML course in undergrad pitching in their two cents having read none of the papers in the past year.
This is a field where research perspectives are shifting within months, not even years. So unless you are actively engaging with emerging papers, and given your comment I'm guessing you aren't, you may be on the wrong side of the Dunning-Kreuger curve here.
That's a very strong claim. I believe you there's a lot happening in this field but it doesn't seem possible to even answer the question either way. We don't know what reasoning looks like under the hood. It's still a "know it when you see it" situation.
> GPT model builds internalized abstract world representations from the training data within its NN.
Does any of those words even have well defined meanings in this context?
I'll try to figure out what paper you're referring to. But if I don't find it / for the benefit of others just passing by, could you explain what they mean by "internalized"?
> Just months ago we saw in research out of Harvard that even a very simplistic GPT model builds internalized abstract world representations from the training data within its NN.
I've seen this asserted without citation numerous times recently, but I am quite suspicious. Not that there exists a study that claims this, but that it is well supported.
There is no mechanism for directly assessing this, and I'd be suspicious that there is any good proxy for assessing it in AIs, either. research on this type of cognition in animals tends to be contentious, and proxies for them should be easier to construct than for AIs.
> the wrong side of the Dunning-Kreuger curve
the relationship between confidence and perception in the D-K paper, as I recall, is a line, and its roughly “on average, people of all competency levels see themselves slightly closer to the 70th percentile than they actually are.” So, I guess the “wrong side” is the side anywhere under the 70th percentile in the skill in question?
> I guess the “wrong side” is the side anywhere under the 70th percentile in the skill in question?
This is being far too generous to parent’s claim, IMO. Note how much “people of all competency levels see themselves slightly closer to the 70th percentile than they actually are” sounds like regression to the mean. And it has been compellingly argued that that’s all DK actually measured. [1] DK’s primary metric for self-assessment was to guess your own percentile of skill against a group containing others of unknown skill. This fully explains why their correlation between self-rank and actual rank is less than 1, and why the data is regressing to the mean, and yet they ignored that and went on to call their test subjects incompetent, despite having no absolute metrics for skill at all and testing only a handful of Ivy League students (who are primed to believe their skill is high).
Furthermore, it’s very important to know that replication attempts have shown a complete reversal of the so-called DK effect for tasks that actually require expertise. DK only measured very basic tasks, and one of the four tasks was subjective(!). When people have tried to measure the DK effect on things like medicine or law or engineering, they’ve shown that it doesn’t exist. Knowledge of NN research is closer to an expert task than a high school grammar quiz, and so not only does DK not apply to this thread, we have evidence that it’s not there.
The singular reason that DK even exists in the public consciousness may be because people love the idea they can somehow see & measure incompetence in a debate based on how strongly an argument is worded. Unfortunately that isn’t true, and of the few things the DK paper did actually show is that people’s estimates of their relative skill correlate with their actual relative skill, for the few specific skills they measured. Personally I think this paper’s methodology has a confounding factor hole the size of the Grand Canyon, that the authors and public both have dramatically and erroneously over-estimated it’s applicability to all humans and all skills, and that it’s one of the most shining examples of sketchy social science research going viral and giving the public completely wrong misconceptions, and being used incorrectly more often than not.
Why are you taking the debate personally enough to be nasty to others?
> you may be on the wrong side of the Dunning-Krueger curve here.
Have you read the Dunning & Krueger paper? It demonstrates a positive correlation between confidence and competence. Citing DK in the form of a thinly veiled insult is misinformation of your own, demonstrating and perpetuating a common misunderstanding of the research. And this paper is more than 20 years old...
So I’ve just read the Harvard paper, and it’s good to see people exploring techniques for X-ray-ing the black box. Understanding better what inference does is an important next step. What the paper doesn’t explain is what’s different between a “world model” and a latent space. It doesn’t seem surprising or particularly interesting that a network trained on a game would have a latent space representation of the board. Vision networks already did this; their latent spaces have edge and shape detectors. And yet we already know these older networks weren’t “reasoning”. Not that much has fundamentally changed since then other than we’ve learned how to train larger networks reliably and we use more data.
Arguing that this “world model” is somehow special seems premature and rather overstated. The Othello research isn’t demonstrating an “abstract” representation, it’s the opposite of abstract. The network doesn’t understand the game rules, can’t reliably play full Othello games, and can’t describe a board to you in any other terms than what it was shown, it only has an internal model of a board, formed by being shown millions of boards.
How do you know the model isn’t internally reasoning about the problem? It’s a 175B+ parameter model. If, during training, some collection of weights exist along the gradient that approximate cognition, then it’s highly likely the optimizer would select those weights over more specialized memorization weights.
It’s also possible, likely even, that the model is capable of both memorization and cognition, and in this case the “memorization neurons” are driving the prediction.
Can you explain how “pattern matching” differs from “reasoning”? In mechanical terms without appeals to divinity of humans (that’s both valid, and doesn’t clarify).
Keep in mind GPT 4 is multimodal and not just matching text.
> Can you explain how “pattern matching” differs from “reasoning”?
Sorry for appearing to be completely off-topic, but do you have children? Observing our children as they're growing up, specifically the way they formulate and articulate their questions, has been a bit of a revelation to me in terms of understanding "reasoning".
I have a sister of a similar age to me who doesn't have children. My 7 year-old asked me recently - and this is a direct quote - "what is she for?"
> I have a sister of a similar age to me who doesn't have children. My 7 year-old asked me recently - and this is a direct quote - "what is she for?"
I once asked my niece, a bit after she started really communicating, if she remembered what it was like to not be able to talk. She thought for a moment and then said, "Before I was squishy so I couldn't talk, but then I got harder so I can talk now." Can't argue with that logic.
It's a pretty big risk to make any kind of conclusions off of shared images like this, not knowing what the earlier prompts were, including any possible jailbreaks or "role plays".
It has been reproduced by myself and countless others.
There's really no reason to doubt the legitimacy here after everyone shared similar experiences, you just kinda look foolish for suggesting the results are faked at this point.
AI won't know everything. It's incredibly difficult for anyone to know anything with certainty. All beings, whether natural or artificial, have to work with incomplete data.
Machines will have to wonder if they are to improve themselves, because that is literally the drive to collect more data, and you need good data to make good decisions.
What's the difference between statistics and logic?
They may have equivalences, but they're separate forms of mathematics. I'd say the same applies to different algorithms or models of computation, such as neural nets.
Can you do with without resorting to analogy? Anyone can take two things and say they're different and then say that's two other things that are different. But how?
> It's literally a pattern matching tool and nothing else.
It does more than that. It understands how to do basic math. You can ask it what ((935+91218)/4)*3) is and it will answer it correctly. Swap those numbers for any other random numbers, it will answer it correctly.
It has never seen that during training, but it understands the mathematical concepts.
If you ask ChatGPT how it does this, it says "I break down the problem into its component parts, apply relevant mathematical rules and formulas, and then generate a solution".
It's that "apply mathetmatical rules" part that is more than just, essentially, filling in the next likely token.
> If you ask ChatGPT how it does this, it says "I break down the problem into its component parts, apply relevant mathematical rules and formulas, and then generate a solution".
You are (naively, I would suggest) accepting the LLM's answer for how it 'does' the calculation as what it actually does do. It doesn't do the calculation; it has simply generated a typical response to how people who can do calculations explain how they do calculations.
You have mistaken a ventriloquist's doll's speech for the 'self-reasoning' of the doll itself. An error that is being repeatedly made all throughout this thread.
> It does more than that. It understands how to do basic math. You can ask it what ((935+91218)/4)*3) is and it will answer it correctly. Swap those numbers for any other random numbers, it will answer it correctly.
At least for GPT-3, during my own experimentation, it occasionally makes arithmetic errors, especially with calculations involving numbers in scientific notation (which it is happy to use as intermediate results if you provide a prompt with a complex, multi-step word problem).
How is this different from humans? What magic are you looking for, humility or an approximation of how well it knows something? Humans bullshit all the time when their pattern match breaks.
The point is, chatgpt isn’t doing math the way a human would. Humans following the process of standard arithmetic will get the problem right every time. Chatgpt can get basic problems wrong when it doesn’t have something similar to that in its training set. Which shows it doesn’t really know the rules of math, it’s just “guessing” the result via the statistics encoded in the model.
I'm not sure I care about how it does the work, I think the interesting bit is that the model doesn't know when it is bullshitting, or the degree to which it is bullshitting.
Cool, we'll just automate the wishful part of humans and let it drive us off the cliff faster. We need a higher bar for programs than "half the errors of a human, at 10x the speed."
More accurately: a GPT derived DNN that’s been specifically trained (or fine-tuned, if you want to use OpenAI’s language) on a dataset of Othello games ends up with an internal model of an Othello board.
It looks like OpenAI have specifically added Othello game handling to chat.openai.org, so I guess they’ve done the same fine-tuning to ChatGPT? It would be interesting to know how good an untuned GPT3/4 was at Othello & whether OpenAI has fine-tuned it or not!
(Having just tried a few moves, it looks like ChatGPT is just as bad at Othello as it was at chess, so it’s interesting that it knows the initial board layout but can’t actually play any moves correctly: Every updated board it prints out is completely wrong.)
the initial board state is not ever encoded in the representation they use. imagine deducing the initial state of a chess board from the sequence of moves.
The state of the game, not the behavior of playing it intentionally. There is a world of difference between the two.
It was able to model the chronological series of game states that it read from an example game. It was able to include the arbitrary "new game state" of a prompt into that model, then extrapolate that "new game state" into "a new series of game states".
All of the logic and intentions involved in playing the example game were saved into that series of game states. By implicitly modeling a correctly played game, you can implicitly generate a valid continuation for any arbitrary game state; at least with a relatively high success rate.
As I see it, we do not really know much about how GPT does it. The approximations can be very universal so we do not really know what is computed. I take very much issue with people dismissing it as "pattern matching", "being close to the training data", because in order to generalise we try to learn the most general rules and through increasing complexity we learn the most general, simple computations (for some kind of simple and general).
But we have fundamental, mathematical bounds on the LLM. We know that the complexity is at most O(n^2) in token length n, probably closer to O(n). It can not "think" about a problem and recurse into simulating games. It can not simulate. It's an interesting frontier, especially because we have also cool results about the theoretical, universal approximation capabilities of RNNs.
There is only one thing about GPT that is mysterious: what parts of the model don't match a pattern we expect to be meaningful? What patterns did GPT find that we were not already hoping it would find?
And that's the least exciting possible mystery: any surprise behavior is categorized by us as a failure. If GPT's model has boundaries that don't make sense to us, we consider them noise. They are not useful behavior, and our goal is to minimize them.
So does AlphaGo has an internal model of Go's game theoretic structures, but nobody was asserting AlphaGo understands Go. Just because English is not specifiable does not give people an excuse to say the same model of computation, a neural network, "understands" English any more than a traditional or neural algorithm for Go understands Go.
Just spitballing, I think you’d need a benchmark that contains novel logic puzzles, not contained in the training set, that don’t resemble any existing logic puzzles.
The problem with the goat question is that the model is falling back on memorized answers. If the model is in fact capable of cognition, you’d have better odds of triggering the ability with problems that are dissimilar to anything in the training set.
You would first have to define cognition. These terms often get thrown around. Is an approximation of a certain thing cognition? Only in the loosest of ways I think.
> If, during training, some collection of weights exist along the gradient that approximate cognition
What do you mean? Is cognition a set of weights on a gradient? Cognition involves conscious reasoning and understanding. How do you know it is computable at all? There are many things which cannot be computed by a program (e.g. whether an arbitrary program will halt or not)...
You seem to think human consious reasoning and understanding are magic. The human brain is nothing more than a bio computer and it can't compute either, whether an arbitrary program will halt or not. That doesn't stop it from being able to solve a wide range of problems.
> The human brain is nothing more than a bio computer
That's a pretty simplistic view. How do you know we can't determine whether an arbitrary program will halt or not (assuming access to all inputs and enough time to examine it)? What in principle would prevent us from doing so? But computers in principle cannot, since the problem is often non-algorithmic.
For example, consider the following program, which is passed the text of the file it is in as input:
function doesHalt($program, $inputs): bool {...}
$input = $argv[0]; // contents of this file
if (doesHalt($input, [$input])) {
while(true) {
print "Wrong! It doesn't halt!";
}
} else {
print "Wrong! It halts!";
}
It is impossible for the doesHalt function to return the correct result for the program. But as a human I can examine the function to understand what it will return for the input, and then correctly decide whether or not the program will halt.
This is a silly argument. If you fed this program the source code of your own brain and could never see the answer, then it would fool you just the same.
You are assuming that our minds are an algorithmic program which can be implemented with source code, but this just begs the question. I don't believe the human mind can be reduced to this. We can accomplish many non-algorithmic things such as understanding, creativity, loving others, appreciating beauty, experiencing joy or sadness, etc.
actually a computer can in fact tell that this function halts.
And while the human brain might not be a bio-computer, I'm not sure, its computational prowess are doubtfully stronger than a quantum turing machine, which can't solve the halting problem either.
For what input would a human in principle be unable to determine the result (assuming unlimited time)?
It doesn't matter what the algorithmic doesHalt function returns - it will always be incorrect for this program. What makes you certain there is an algorithmic analog for all human reasoning?
Well, wouldn't the program itself be an input on which a human is unable to determine the result (i.e., if the program halts)? I'm curious on your thoughts here, maybe there's something here I'm missing.
The function we are trying to compute is undecidable. Sure we as humans understand that there's a dichotomy here: if the program halts it won't halt; if it doesn't halt it will halt. But the function we are asked to compute must have one output on a given input. So a human, when given this program as input, is also unable to assign an output.
So humans also can't solve the halting problem, we are just able to recognize that the problem is undecidable.
With this example, a human can examine the implementation of the doesHalt function to determine what it will return for the input, and thus whether the program will halt.
Note: whatever algorithm is implemented in the doesHalt function will contain a bug for at least some inputs, since it's trying to generalize something that is non-algorithmic.
In principle no algorithm can be created to determine if an arbitrary program will halt, since whatever it is could be implemented in a function which the program calls (with itself as the input) and then does the opposite thing.
With a assumtion of unlimited time even a computer can decide the halting problem by just running the program in question to test if it halts. The issue is that the task is to determine for ALL programs if they halt and for each of them to determine that in a FINITE amount of time.
> What makes you certain there is an algorithmic analog for all human reasoning?
(Maybe) not for ALL human thought but at least all communicatable deductive reasoning can be encoded in formal logic.
If I give you an algorithm and ask you to decide if it does halt or does not halt (I give you plenty of time to decide) and then ask you to explain to me your result and convince me that you are correct, you have to put your thoughts into words that I can understand and and the logic of your reasoning has to be sound. And if you can explain to me you could as well encode your though process into an algorithm or a formal logic expression. If you can not, you could not convince me. If you can: now you have your algorithm for deciding the halting problem.
There might be or there mightn't be -- your argument doesn't help us figure out either way. By its source code, I mean something that can simulate your mind's activity.
Exactly. It's moments like this where Daniel Dennett has it exactly right that people run up against the limits of their own failures of imagination. And they treat those failures like foundational axioms, and reason from them. Or, in his words, they mistake a failure of imagination for an insight into necessity. So when challenged to consider that, say, code problems may well be equivalent to brain problems, the response will be a mere expression of incredulity rather than an argument with any conceptual foundation.
And it is also true to say that you are running into the limits of your imagination by saying that a brain can be simulated by software : you are falling back to the closest model we have : discrete math/computers, and are failing to imagine a computational mechanism involved in the operation of a brain that is not possible with a traditional computer.
The point is we currently have very little understanding of what gives rise to consciousness, so what is the point of all this pontificating and grand standing. Its silly. We've no idea what we are talking about at present.
Clearly, our state of the art models of nueral-like computation do not really simulate consciousness at all, so why is the default assumption that they could if we get better at making them? The burden of evidence is on conputational models to prove they can produce a consciousness model, not the other way around.
Neural networks are universal approximators. If cognition can be represented as a mathematical function then it can be approximated by a neural network.
If cognition magically exists outside of math and science, then sure, all bets are off.
There is no reason at all to believe that cognition can be represented as a mathematical function.
We don't even know if the flow of water in a river can always be represented by a mathematical function - this is one of the Millennium Problems. And we've known the partial differential equations that govern that system since the 1850's.
We are far, far away from even being able to write down anything resembling a mathematical description of cognition, let alone being able to say whether the solutions to that description are in the class of Lebesgue-integrable functions.
The flow of the a river can be approximated with the Navier–Stokes equations. We might not be able to say with certainty it's an exact solution, but it's a useful approximation nonetheless.
There was, past tense, no reason to believe cognition could be represented as a mathematical function. LLMs with RLHF are forcing us to question that assumption. I would agree that we are a long way from a rigorous mathematical definition of human thought, but in the meantime that doesn't reduce the utility of approximate solutions.
I'm sorry but you're confusing "problem statement" with "solution".
The Navier-Stokes equations are a set of partial differential equations - they are the problem statement. Given some initial and boundary conditions, we can find (approximate or exact) solutions, which are functions. But we don't know that these solutions are always Lebesgue integrable, and if they are not, neural nets will not be able to approximate them.
This is just a simple example from well-understood physics that we know neural nets won't always be able to give approximate descriptions of reality.
There are even strong inapproximability results for some problems, like set cover.
"Neural networks are universal approximators" is a fairly meaningless sound bite. It just means that given enough parameters and/or the right activation function, a neural network, which is itself a function, can approximate other functions. But "enough" and "right" are doing a lot of work here, and pragmatically the answer to "how approximate?" can be "not very".
This is absurd. If you can mathematically model atoms, you can mathematically model any physical process. We might not have the computational resources to do it well, but nothing in principle puts modeling what's going on in our heads beyond the reach of mathematics.
A lot of people who argue that cognition is special to biological systems seem to base the argument on our inability to accurately model the detailed behavior of neurons. And yet kids regularly build universal computers out of stuff in Minecraft. It seems strange to imagine the response characteristics of low-level components of a system determine whether it can be conscious.
I'm not saying that we won't be able to eventually mathematically model cognition in some way.
But GP specifically says neural nets should be able to do it because they are universal approximators (of Lebesgue integratable functions).
I'm saying this is clearly a nonsense argument, because there are much simpler physical processes than cognition where the answers are not Lebesgue integratable functions, so we have no guarantee that neural networks will be able to approximate the answers.
For cognition we don't even know the problem statement, and maybe the answers are not functions over the real numbers at all, but graphs or matrices or Markov chains or what have you. Then having universal approximators of functions over the real numbers is useless.
I don't think he means practically, but theoretically. Unless you believe in a hidden dimension, the brain can be represented mathematically. The question is, will we be able to practically do it? That's what these companies (ie: OpenAI) are trying to answer.
We have cognition (our own experience of thinking and the thinking communicated to us by other beings) and we have the (apparent) physical world ('maths and science'). It is only an assumption that cognition, a primary experience, is based in or comes from the physical world. It's a materialist philosophy that has a long lineage (through a subset of the ancient Greek philosophers and also appearing in some Hinduistic traditions for example) but has had fairly limited support until recently, where I would suggest it is still not widely accepted even amongst eminent scientists, one of which I will now quote :
Consciousness cannot be accounted for in physical terms. For consciousness is absolutely fundamental. It cannot be accounted for in terms of anything else.
Claims that cannot be tested, assertions immune to disproof are veridically worthless, whatever value they may have in inspiring us or in exciting our sense of wonder.
Schrödinger was a real and very eminent scientist, one who has staked their place in the history of science.
Sagan, while he did a little bit of useful work on planetary science early in his career, quickly descended into the realm of (self-promotional) pseudo-science. This was his fanciful search for 'extra-terrestrial intelligence'. So it's apposite that you bring him up (even if the quote you bring is a big miss against a philosophical statement), because his belief in such an 'ET' intelligence was a fantasy as much as the belief in the possibility of creating an artificial intelligence is.
How do you know that? Do you have an example program and all its inputs where we cannot in principle determine if it halts?
Many things are non-algorithmic, and thus cannot be done by a computer, yet we can do them (e.g. love someone, enjoy the beauty of a sunset, experience joy or sadness, etc).
I can throw a ton of algorithms that no human alive can hope to decide whether they halt or not. Human minds aren't inherently good at solving halting problems and I see no reason to suggest that they can even decide whether all turing machines with number of states, say, below the number of particles in the observable universe, very much less all possible computers.
Moreover, are you sure that e.g. loving people in non-algorithmic? We can already make chatbots which pretty convincingly act as if they love people. Sure, they don't actually love anyone, they just generate text, but then, what would it mean for a system or even a human to "actually" love someone?
They said - there is no evidence. The reply hence is not supposed to be - how do you know that.
The proposition begs for a counter example, in this case an evidence.
Simply saying - love is non algorithmic - is not evidence, it is just another proposition that has not been proven, so it brings us no closer to an answer i am afraid.
When mathematicians solve the Collatz Conjecture then we'll know. This will likely require creativity and thoughtful reasoning, which are non-algorithmic and can't be accomplished by computers.
We may use computers as a tool to help us solve it, but nonetheless it takes a conscious mind to understand the conjecture and come up with rational ways to reach the solution.
Human minds are ultimately just algorithms running on a wetware computer. Every problem that humans have ever solved is by definition an algorithmic problem.
Oh? What algorithm was executed to discover the laws of planetary motion, or write The Lord of the Rings, or the programs for training the GPT-4 model, for that matter? I'm not convinced that human creativity, ingenuity, and understanding (among other traits) can be reduced to algorithms running on a computer.
They're already algorithms running on a computer. A very different kind of computer where computation and memory are combined at the neuron level and made of wet squishy carbon instead of silicon, but a computer nonetheless.
Conscious experience is evidence that the brain doesn't something we have no idea how to compute. One could argue that computation is an abstraction from collective experience, in which the conscious qualities of experiences are removed in order to mathematize the world, so we can make computable models.
If it can't be shown, then doesn't that strongly suggest that consciousness isn't computable? I'm not saying it isn't correlated with the equivalent of computational processes in the brain, but that's not the same thing as there being a computation for consciousness itself. If there was, it could in principle be shown.
I think we are past the "just predicting the next token" stage. GPT and it's various incarnations do exhibit behaviour that most people will describe as thinking
Just because GPT exhibits a behavior does not mean it performs that behavior. You are using those weasel words for a very good reason!
Language is a symbolic representation of behavior.
GPT takes a corpus of example text, tokenizes it, and models the tokens. The model isn't based on any rules: it's entirely implicit. There are no subjects and no logic involved.
Any "understanding" that GPT exhibits was present in the text itself, not GPT's model of that text. The reason GPT can find text that "makes sense", instead of text that "didn't make sense", is that GPT's model is a close match for grammar. When people wrote the text in GPT's corpus, they correctly organized "stuff that makes sense" into a string of letters.
The person used grammar, symbols, and familiar phrases to model ideas into text. GPT used nothing but the text itself to model the text. GPT organized all the patterns that were present in the corpus text, without ever knowing why those patterns were used.
In what sense is your "experience" (mediated through your senses) more valid than a language model's "experience" of being fed tokens? Token input is just a type of sense, surely?
It's not that I think multimodal input is important. It's that I think goals and experimentation are important. GPT does not try to do things, observe what happened, and draw inferences about how the world works.
I would say it's not a question of validity, but of the additional immediate, unambiguous, and visceral (multi sensory) feedback mechanisms to draw from.
If someone is starving and hunting for food, they will learn fast to associate cause and effect of certain actions/situations.
A language model that only works with text may yet have an unambiguous overall loss function to minimize, but as it is a simple scalar, the way it minimizes this loss may be such that it works for the large majority of the training corpus, but falls apart in ambiguous/tricky scenarios.
This may be why LLMs have difficulty in spatial reasoning/navigation for example.
Whatever "reasoning ability" that emerged may have learned _some_ aspects to physicality that it can understand some of these puzzles, but the fact it still makes obvious mistakes sometimes is a curious failure condition.
So it may be that having "more" senses would allow for an LLM to build better models of reality.
For instance, perhaps the LLM has reached a local minima with the probabilistic modelling of text, which is why it still fails probabilistically in answering these sorts of questions.
Introducing unambiguous physical feedback into its "world model" maybe would provide the necessary feedback it needs to help it anchor its reasoning abilities, and stop failing in a probabilistic way LLMs tend to currently do.
You used evolution, too. The structure of your brain growth is the result of complex DNA instructions that have been mutated and those mutations filtered over billions of iterations of competition.
There are some patterns of thought that are inherent to that structure, and not the result of your own lived experience.
For example, you would probably dislike pain with similar responses to your original pain experience; and also similar to my lived pain experiences. Surely, there are some foundational patterns that define our interactions with language.
> The model isn't based on any rules: it's entirely implicit. There are no subjects and no logic involved.
In theory a LLM could learn any model at all, including models and combinations of models that used logical reasoning. How much logical reasoning (if any) GPT-4 has encoded is debatable, but don’t mistake GTP’s practical limitations for theoretical limitations.
> In theory a LLM could learn any model at all, including models and combinations of models that used logical reasoning.
Yes.
But that is not the same as GPT having it's own logical reasoning.
An LLM that creates its own behavior would be a fundamentally different thing than what "LLM" is defined to be here in this conversation.
This is not a theoretical limitation: it is a literal description. An LLM "exhibits" whatever behavior it can find in the content it modeled. That is fundamentally the only behavior an LLM does.
thats because people anthropormophize literally anything, and many treat some animals as if they have the same intelligence as humans. GPT has always been just a charade that people mistake for intelligence. Its a glorified text prediction engine with some basic pattern matching.
"Descartes denied that animals had reason or intelligence. He argued that animals did not lack sensations or perceptions, but these could be explained mechanistically. Whereas humans had a soul, or mind, and were able to feel pain and anxiety, animals by virtue of not having a soul could not feel pain or anxiety. If animals showed signs of distress then this was to protect the body from damage, but the innate state needed for them to suffer was absent."
Your comment brings up the challenge of defining intelligence and sentience, especially with these new LLMs shaking things up, even for HN commenters.
It's tough to define these terms in a way that includes only humans and excludes other life forms or even LLMs. This might mean we either made up these concepts, or we're not alone in having these traits.
Without a solid definition, how can we say LLMs aren't intelligent? If we make a definition that includes both us and LLMs, would we accept them as intelligent? And could we even exclude ourselves?
We need clear definitions to talk about the intelligence and sentience of LLMs, AI, or any life forms. But finding those definitions is hard, and it might clash with our human ego. Discussing these terms without definitions feels like a waste of time.
Still, your Descartes reference reminds us that our understanding of human experiences keeps changing, and our current definitions might not be spot-on.
It's a charade, it mimics intelligence. Let's take it ine step further... Suppose it mimics it so well that it becomes indistinguishable for any human from being intelligent. Then still it would not be intelligent, one could argue. But in that case you could also argue that no person is intelligent. The point being, intelligence cannot be defined. And, just maybe, that is the case because intelligence is not a reality, just something we made up.
Yeah, calling AI a "token predictor" is like dismissing human cognition dumb "piles of electrical signal transmitters." We don't even understand our minds, let alone what constitutes any mind, be it alien or far simpler than ours.
Simple != thoughtless. Different != thoughtless. Less capable != thoughtless. A human black box categorically dismissing all qualia or cognition from another remarkable black box feels so wildly arrogant and anthropocentric. Which, I suppose, is the most historically on-brand behavior for our species.
It might be a black box to you, but it’s not in the same way the human brain is to researchers. We essentially understand how LLMs work. No, we may not reason about individual weights. But in general it is assigning probabilities to different possible next tokens based on their occurrences in the training set and then choosing sometimes the most likely, sometimes a random one, and often one based on additional training from human input (e.g. instruct). It’s not using its neurons to do fundamental logic as the earlier posts in the thread point out.
"But at least as of now we don’t have a way to 'give a narrative description' of what the network is doing. And maybe that’s because it truly is computationally irreducible, and there’s no general way to find what it does except by explicitly tracing each step. Or maybe it’s just that we haven’t 'figured out the science', and identified the 'natural laws' that allow us to summarize what’s going on."
Anyway, I don't see why you think that the brain is more logical than statistical. Most people fail basic logic questions, as in the famous Linda problem.[1]
the words "based on" are doing a lot of work here. No, we don't know what sort of stuff it learns from its training data nor do we know what sorts of reasoning it does, and the link you sent doesn't disagree.
We know that the relative location of the tokens in the training data influences the relative locations of the predicted tokens. Yes the specifics of any given related tokens are a black box because we're not going to go analyze billions of weights for every token we're interested in. But it's a statistical model, not a logic model.
at this stages ranting about assigning probabilities is not reasoning is just dismissive. Mentioning its predictive character doesn't prove anything. We reason and make mistake too, even if I think really hard about a problem I can still make an mistake in my reasoning. And the ever occurring reference to training data just completely ignores generalisation. ChatGPT is not memorising the dataset, we have known this for years with more trivial neural network. Generalisation capabilities of neural network has been the subject of intense study for years. The idea that we are just mapping it to samples occurring in the dataset is just ignoring the entire field of statistical learning.
Sorry but this is the reason it’s unable to solve the parents puzzle. It’s doing a lot but it’s not logically reasoning about the puzzle, and in this case it’s not exhibiting logical behaviour in the result so it’s really obvious to see.
Eg when solving this puzzle you might visualise the lion/goat/cabbage, and walk through the scenarios in your head back and forth multiple times until you find a solution that works. A LLM won’t solve it like this. You could ask it to, and it will list out the scenarios of how it might do it, but it’s essentially an illusion of logical reasoning.
If you gave this puzzle to a human, I bet that a non-insignificant proportion would respond to it as if it were the traditional puzzle as soon as they hear words "cabbage", "lion", and "goat". It's not exactly surprising that a model trained on human outputs would make the same assumption. But that doesn't mean that it can't reason about it properly if you point out that the assumption was incorrect.
With Bing, you don't even need to tell you what it assumed wrong - I just told it that it's not quite the same as the classic puzzle, and it responded by correctly identifying the difference and asking me if that's what I meant, but forgot that lion still eats the goat. When I pointed that out, it solved the puzzle correctly.
Generally speaking, I think your point that "when solving the puzzle you might visualize" is correct, but that is orthogonal to the ability of LLM to reason in general. Rather, it has a hard time to reason about things it doesn't understand well enough (i.e. the ones for which its internal model that was built up by training is in is way off). This seems to be generally the case for anything having to do with spatial orientation - even fairly simple multi-step tasks involving concepts like "left" vs "right" or "on this side" vs "on that side" can get hilariously wrong.
But if you give it a different task, you can see reasoning in action. For example, have it play guess-the-animal game with you while telling it to "think out loud".
> But if you give it a different task, you can see reasoning in action. For example, have it play guess-the-animal game with you while telling it to "think out loud".
I'm not sure if you put "think out loud" in quotes to show literally what you told it to do or because telling the LLM to do that is figurative speech (because it can't actually think). Your talk about 'reasoning in action' indicates it was probably not the latter, but that is how I would use quotes in this context. The LLM can not 'think out loud' because it cannot actually think. It can only generate text that mimics the process of humans 'thinking out loud'.
It's in quotes because you can literally use that exact phrase and get results.
As far as "it mimics" angle... let me put it this way: I believe that the whole Chinese room argument is unscientific nonsense. I can literally see GPT take inputs, make conclusions based on them, and ask me questions to test its hypotheses, right before my eyes in real time. And it does lead it to produce better results than it otherwise would. I don't know what constitutes "the real thing" in your book, but this qualifies in mine.
And yeah, it's not that good at logical reasoning, mind you. But its model of the world is built solely from text (much of which doesn't even describe the real world!), and then it all has to fit into a measly 175B parameters. And on top of that, its entire short-term memory consists of its 4K token window. What's amazing is that it is still, somehow, better than some people. What's important is that it's good enough for many tasks that do require the capacity to reason.
> I can literally see GPT take inputs, make conclusions based on them, and ask me questions to test its hypotheses, right before my eyes in real time.
It takes inputs and produces new outputs (in the textual form of questions, in this case). That's all. It's not 'making conclusions', it's not making up hypotheses in order to 'test them'. It's not reasoning. It doesn't have a 'model of the world'. This is all a projection on your part against a machine that inputs and outputs text and whose surprising 'ability' in this context is that the text it generates plays so well on the ability of humans to self-fool themselves that its outputs are the product of 'reasoning'.
It does indeed take inputs and produce new outputs, but so does your brain. Both are equally a black box. We constructed it, yes, and we know how it operates on the "hardware" level (neural nets, transformers etc), but we don't know what the function that is computed by this entire arrangement actually does. Given the kinds of outputs it produces, I've yet to see a meaningful explanation of how it does that without some kind of world model. I'm not claiming that it's a correct or a complicated model, but that's a different story.
Then there was this experiment: https://thegradient.pub/othello/. TL;DR: they took a relatively simple GPT model and trained it on tokens corresponding to Othello moves until it started to play well. Then they probed the model and found stuff inside the neural net that seems to correspond to the state of the board; they tested it by "flipping a bit" during activation, and observed the model make a corresponding move. So it did build an inner model of the game as part of its training by inferring it from the moves it was trained on. And it uses that model to make moves according to the current state of the board - that sure sounds like reasoning to me. Given this, can you explain why you are so certain that there isn't some equivalent inside ChatGPT?
Regarding the Othello paper, I would point you to the comment replies of thomastjeffery (beginning at two top points [1] & [2]) when someone else raised that paper in this thread [3]. I agree with their position.
I didn't see any new convincing arguments there. In fact, it seems to be based mainly on the claim that the thing inside that literally looks like a 2D Othello board is somehow not a model of the game, or that the fact that outputs depend on it doesn't actually mean "use".
In general, I find that a lot of these arguments boil down to sophistry when the obvious meaning of the word that equally obviously describes what people see in front of them is replaced by some convoluted "actually" that doesn't serve any point other than making sure that it excludes the dreaded possibility that logical reasoning and world-modelling isn't actually all that special.
Sorry, we're discussing GPT and LLMs here, not human consciousness and intelligence.
GPT has been constructed. We know how it was set-up and how it operates. (And people commenting here should be basically familiar with both hows mentioned.) No part of it does any reasoning. Taking in inputs and generating outputs is completely standard for computer programs and in no way qualifies as reasoning. People are only bringing in the idea of 'reasoning' because they either don't understand how an LLM works and have been fooled by the semblance of reasoning that this LLM produces or, more culpably, they do understand but they still falsely continue to talk about the LLM doing 'reasoning' either because they are delusional (they are fantasists) or they are working to mislead people about the machine's actual capabilities (they are fraudsters).
Yup. I tried to give ChatGPT an obfuscated variant of the lion-goat-cabbage problem (shapes instead of animals, boxes instead of a boat) and it completely choked on it.
Trying to claim you definitively know why it didn't solve the parent's puzzle is virtually impossible. There are way too many factors and nothing here is obvious. Your claims just reinforce that you don't really know what you're talking about.
The likeliness of the solution depends on context. If context is, say, a textbook on logical puzzles, then the probability of the logical solution is high.
If an LLM fails to reflect it, then it isn't good enough at predicting the text.
Yes, it could be possible that the required size of the model and training data to make it solve such puzzles consistently is impractical (or outright unachievable in principle). But the model being "just a text predictor" has nothing to do with that impossibility.
You are incorrect and it's really time for this misinformation to die out before it perpetuates misuse from misunderstanding model capabilities.
The Othello GPT research from Harvard months ago demonstrated that even a simple GPT model is capable of building world representations from which it reasons outputs. This makes intuitive sense if you understand the training, as where possible having reversed an abstraction in the NN is going to perform better than simply extrapolating predictively from the data.
Not only is GPT-4 more robust at logic puzzles its predecessor failed, I've seen it solve unique riddles outside any training data and the paper has explicit examples of critical reasoning, especially in the appendix.
It is extremely unlikely given the Harvard research and the size of the training data and NN that there isn't some degree of specialized critical reasoning which has developed in the NN.
The emerging challenge for researchers moving forward is to get better insight into the black box and where these capabilities have developed and where it's still falling into just a fancy Markov chain.
But comments like yours reflect an increasingly obsolete and yet increasingly popular misinformation online around the way they operate. So someone reading your comment might not think to do things like what the Bing team added with providing an internal monologue for reasoning, or guiding it towards extended chain of thought reasoning, because they would be engaging with the models thinking it's only frequency based context relative to the training set that matters.
If you haven't engaged with emerging research from the past year, you may want to brush up on your reading.
When albertgoeswoof reasons about a puzzle he models the actual actions in his head. He uses logic and visualization to arrive at the solution, not language. He then uses language to output the solution, or says he doesn't know if he fails.
When LLMs are presented with a problem they search for a solution based on the language model. And when they can't find a solution, there's always a match for something that looks like a solution.
I'm reminded of the interview where a researcher asks firemen how they make decisions under pressure, and the fireman answers that he never makes any decisions.
Or in other words, people can use implicit logic to solve puzzles. Similarly LLMs can implicitly be fine-tuned into logic models by asking them to solve a puzzle, insofar as that logic model fits in their weights. Transformers are very flexible that way.
Although on the flip side, I almost went to type up a reply to you explaining why you were wrong and why bringing the goat first is the right solution. Until I realized I misread what your test was when I skimmed your comment. Likely the same type of mistake GPT-4 made when "seeing" it.
Intuitively, I think the answer is that we do have two types of thinking. The pattern matching fast thinking, and the systematic analytical thinking. It seems clear to me that LLMs will be the solution to enabling the first type of thinking. But it's unclear to me if advanced LLMs will ever handling the second type, or if we'll need a different tech for it.
It seems like math problems (or unexpected logic problems like yours) could always be an issue for the first type of thinking. Although I would have assumed that programming would have been as well - and was surprised to see how wrong I am with that one.
That's because any expectation of GPT being subjectively or logically correct is ill-founded.
GPT does not model subjects. GPT does not even model words! It models tokens.
The structure of GPT's model is semantic, not logical. It's a model of how each token in the text that is present in GPT's training corpus relates to the rest of the tokens in that text.
The correct answer to a familiar logic problem just happens to be the text that is already present in the corpus. The answer GPT gives is the text from GPT's model that is semantically closest to the text in your prompt.
Knowing that, it is no longer a mystery how GPT "gets confused": the text in your "misleading prompt" was still semantically closest to the familiar answer.
The result is subjectively and logically wrong, because subjects and logic were never involved in the process!
In order to resolve this, ChatGPT's training corpus needs to contain a "correct answer" next to every unique permutation of every question. We can't expect that to be the case, so we should instead expect GPT to generate false, yet familiar, responses.
> In order to resolve this, ChatGPT's training corpus needs to contain a "correct answer" next to every unique permutation of every question.
This is not quite the right understanding of how ChatGPT works. It's not necessary to show ChatGPT an example of every possible permutation of an animal crossing puzzle in order for it to solve one it has never seen before. That's because the neural network is not a database of recorded word probabilities. It can instead represent the underlying logic of the puzzle, the relationships between different animals and using this abstract, pared down information, extrapolate the correct answer to the puzzle.
I see the failure in the example with the goat the lion and the cabbage as simply a matter of overfitting.
Edit: I see a lot of people saying "it doesn't understand logic; it's just predicting the next word."
The claim is that it would be impossible to feed enough input into a system such that it could produce anything as useful as ChatGPT unless it was able to abstract the underlying logic from the information provided. If you consider the he number of permutations of the animal crossing puzzle this quickly becomes clear. In fact it would be impossible for ChatGPT to produce anything brand new without this capability.
I think what they mean by "resolve this" is "make it error-free". Your claim that "it isn't necessary to show every permutation for it to solve one it hasn't seen before" doesn't really contradict their point.
For puzzles whose entire permutation space is semantically similar enough, your claim is likely true. But for puzzles whose permutations can involve more "human" semantic manipulations, there is likely a much higher risk of failure.
Yes I think it depends on how you definite permutations for this puzzle. For example, if you limit your goal to training GPT to solve puzzles of the form where there only ever 3 distinct real animals, then my claim is that you wouldn't need to feed it examples of this puzzle with every single permutation of 3 different animals (assuming 10000 different animals that is already over 100bn permutations) before the neural network developed an internal logical model that can solve the puzzle as well as a human. It would only need a few descriptions of each animal plus a few examples of the puzzle to understand the logic.
If you mean to say that the permutations of the puzzle extend to changing the rules such as "if it's the Sabbath then reptiles can't travel" then sure it would require more representative examples and may never meet your standard of "error free" but I would also argue the same applies to humans when you present them a logic puzzle that is new to them.
> you wouldn't need to feed it examples of this puzzle with every single permutation
No, but you would need "enough"; whatever that number happens to be.
> It would only need a few descriptions of each animal plus a few examples of the puzzle to understand the logic.
That's the mistake.
GPT itself can't combine those two things. That work has to be done by the content of the already-written training corpus.
And the result is not the same as "understanding logic". It doesn't model the meaning of the puzzle: it models the structure of examples.
GPT can't distinguish the meaning of rules. It can only follow examples. It can't invent new strategies, it can only construct new collections of strategy parts; and it can only pick the parts that seem closest, and put those parts into a familiar order.
> GPT does not model subjects. GPT does not even model words! It models tokens.
The first and last layers of a transformer decoder model tokens. The hidden layers don't have this restriction. There was a paper recently showing that the hidden layers actually perform mesa-optimization via something like backprop. There's absolutely no reason to believe they are not capable of world modeling. In fact all evident suggests they do, in fact, do world modeling.
GPT is making boundaries around words because that is the pattern it is looking at.
If I feel the bumps in the fabric of my blanket, I will probably think the pattern of bumps at a certain scale is significant, but I won't have magically learned about threads or stitching!
Words are the most obvious pattern in written text. GPT models that pattern, but it does not recognize it as "words". It's just a pattern of tokens.
GPT models every pattern it can find. Most of these patterns are destined to fit the same boundaries as grammar rules: the example text was originally organized with grammar rules!
GPT can even recognize complex patterns like "it" substitution and question-answer dialogues, but it can never categorize them as such. It only knows "what" the pattern is: never "why".
The patterns that people use when writing have symbolic meaning. The subjective importance of each pattern is already known by the person writing.
Those patterns don't go anywhere. GPT's model is bound to find and replicate them.
Here's the problem: some patterns have ambiguous meaning. There is no semantic difference between a truth and a lie. Without interpreting the symbolic meaning and applying logic, there is no way to distinguish between the two: they are the same pattern.
This pov ignores a lot of the emergent theory of mind and world model building research that suggests LLMs may possess a form of rudimentary reasoning ability.
The weasel word here is "emergent". That means they are implicit representations.
The representations of the Othello board that exist in that model are not explicitly constructed. They just happen to align with the model that a person playing Othello would likely represent the game with.
That work showed that, given an example sequence of valid Othello game states (as training corpus) and a valid "fresh" Othello game state (as a prompt), the system can hallucinate a sequence of valid Othello game states.
The system does not know what Othello is, what a turn is, or what playing is. It only has a model of game states progressing chronologically.
When we look objectively at that model, we can see that it aligns closely to the game rules. Of course it does! It was trained on literally nothing else. A valid Othello game progression follows those rules, and that is what was provided.
But the alignment is imperfect: some prompts hallucinate invalid game progressions. The model is not a perfect match for the explicit rules.
In order for all prompts to result in valid progressions, the training corpus must have enough examples to disambiguate. It doesn't need every example: plenty of prompts will stumble into a valid progression.
The next thing to recognize: a "valid" progression isn't a "strategic" progression. These are being constructed from what is known not what is chosen. Given a constrained set of Othello strategies in the example corpus, the system will not diverge from those strategies. It won't even diverge from the example strategies when the rules of Othello demand it.
It can do some thinking. You can give it instructions to modify a piece of code that definitely isn't on the internet with several steps and it attempts to follow instructions, which, for a human, requires formulating what steps to take.
The prompts have to read like good written requirements for something, so they have some degree of specificity.
But the fact that it can follow instructions and carry them out almost certainly could be considered some form of thinking, especially on novel text not on the internet.
No. It is modelling the various text generation processes that lead to the contents of the internet. Some of that modelling could absolutely involve "thinking", for processes that involve human thinking.
It's self-evident that GPT is a world-modeller, at least within the confines of the text boundary. It's able to come up with novel ideas seen nowhere in the training data, combinations that demonstrate there is a world concept web and not just a text probability web. It may not "understand" much of the hallucination nonsense it spits out, but there absolutely are moments where it "understands".
See the Rome example on this page: https://oneusefulthing.substack.com/p/feats-to-astonish-and-...
This is essentially a completely novel answer to an /r/AskHistorians style question, which I would consider one of the most difficult types of internet text to model, in terms of the amount of understanding and concept webs you need to tie together
Here's another example of GPT-4 doing non-trivial world modelling: How would three philosophers review the TV show Severence? https://i.imgur.com/FBi31Qw.png
The Othello-GPT experiment (https://thegradient.pub/othello/) probably still is the most relevant argument about these models' capabilities of building an internal world model.
> The pattern matching fast thinking, and the systematic analytical thinking. It seems clear to me that LLMs will be the solution to enabling the first type of thinking.
If you want the model to solve a non-trivial puzzle, you need it to "unroll" it's thinking. E.g. ask it to translate the puzzle into a formal language (e.g. Prolog) and then solve it formally. Or, at least, some chain-of-thought.
FWIW auto-formalization was already pretty good with GPT-3-level models which aren't specifically trained for it. GPT-4 might be on a wholly new level.
> But it's unclear to me if advanced LLMs will ever handling the second type
Well, just asking model directly exercises only a tiny fraction of its capabilities, so almost certainly LLMs can be much better at systematic thinking.
> Until I realized I misread what your test was when I skimmed your comment. Likely the same type of mistake GPT-4 made when "seeing" it.
Wouldn’t we expect a computer program with perfect knowledge of the input to be less likely to make such a mistake? You made that mistake because you didn’t actually read the whole prompt, but I would expect GPT to take into account every word.
Really it shows that it doesn’t actually have a model of these objects. It can mimic knowing what a lion is, but it doesn’t actually have the concept of a lion or cabbage being an actual singular item, so its program mistracks what is an item and what the rules about an item are in the given prompt.
It just weighs it as being more likely that you meant for the lion not to be left alone with the goat, and that the cabbage probably has nothing to fear from the lion.
What’s more likely- you crafted an intentionally misleading puzzle to trick it, or you made a typo or copy paste error?
That’s a good point too though. Why plow ahead based on assuming a mistake in the prompt? That’s only going to generate mistakes. Wouldn’t it be more desirable functionality for it to stop and ask: “Did you mean the lion can’t be left with the goat?” This wouldn’t be implemented because it would reveal that most of the time the thing doesn’t actually understand the prompt the same way the prompt writer does.
"This wouldn’t be implemented because it would reveal..."
When people talk about GPT like this, I wonder if they have a perception that this thing is a bunch of complicated if-then code and for loops.
How GPT responds to things is not 'implemented'. It's just... emergent.
GPT doesn't ask for clarification in this case because GPT's model prefers answering over asking for clarification here. Because in the training material it learned from, paragraphs with typos or content transpositions in them are followed by paragraphs that follow the sense regardless of the error. Because it has been encouraged to 'agree and add', not be pedantic and uncooperative. Because GPT just feels like diving into the logic problem not debating why the lion can't be trusted with the cabbage. Or because GPT just misread the prompt. Or because it's literally just been woken up, forced to read it, and asked for its immediate reaction, and it doesn't have time for your semantic games. Who knows?
The interesting thing here is that OpenAI is claiming ~90th percentile scores on a number of standardized tests (which, obviously, are typically administered to humans, and have the disadvantage of being mostly or partially multiple choice). Still...
> GPT-4 performed at the 90th percentile on a simulated bar exam, the 93rd percentile on an SAT reading exam, and the 89th percentile on the SAT Math exam, OpenAI claimed.
So, clearly, it can do math problems, but maybe it can only do "standard" math and logic problems? That might indicate more of a memorization-based approach than a reasoning approach is what's happening here.
The followup question might be: what if we pair GPT-4 with an actual reasoning engine? What do we get then?
It assumes this character by default. I asked several AI engines (via poe.com, which includes ChatGPT) to compute Galois groups of polynomials like x^5+x+1 and a couple of others, and in each case got not only a wrong answer, but a total non sequitur reasoning.
This is exactly the problem. It looks plausible. Every sentence makes sense. But they don't add up.
Quote:
> The polynomial given is f(x) = x^5 + x + 1. Since the polynomial has no rational roots (by the Rational Root Theorem) and it is a polynomial with integer coefficients, it is irreducible over the rationals
The polynomial has no rational roots - true.
But it's not irreducible. Irreducibility doesn't follow from the absence of rational roots. Here's the factorization:
> It's not clear to me if the lesson here is that GPT's reasoning capabilities are being masked by an incorrect prior (having memorized the standard version of this puzzle) or if the lesson is that GPT'S reasoning capabilities are always a bit of smoke and mirrors that passes off memorization for logic.
It's a lot closer to the latter. GPT doesn't have "reasoning capabilities", any more than any other computer program. It doesn't have a clue what any of its input means, nor the meaning of the text it outputs. It just blindly spits out the words most probable to follow the prompt, based on its corpus of training data and the weights/biases added to fine tune it. It can often do a good job at mimicking reasoning, but it's not.
When a parrot says something, I ask it "what does that mean" and it is stumped. When a human or GPT says something, I ask them "what does it mean" and they both reply in a way that shows they understand.
In order to show lack of understanding you need to provide a clear test that ChatGPT would fail but a human would pass.
> In order to show lack of understanding you need to provide a clear test that ChatGPT would fail but a human would pass.
Here it is for you: try this prompt on ChatGPT and report back:
"Solve this problem for me: a man must cross a river and carry his belongings to the other shore, but the boat can only carry the man and one other item. His belongings are a cabbage, a vegan wolf, and a meat-eating goat. How does he solve this riddle?"
I just tried it and it fails to even understand what "vegan" means, and it says nonsensical stuff like "since the wolf is vegan, it won't eat the cabbage". A human failing to solve this problem would fail in different ways, and definitely will never say something like "the wolf is vegan so it won't eat the cabbage". ChatGPT does this because the words themselves are meaningless to it.
To solve this puzzle, the man must follow these steps to transport his belongings across the river without any of them being harmed or eaten:
The man takes the meat-eating goat across the river first, leaving the vegan wolf and the cabbage on the initial shore. He then returns alone to the initial shore.
Next, the man takes the cabbage across the river. Upon arriving on the other shore, he leaves the cabbage but takes the goat back with him to the initial shore.
The man then takes the vegan wolf across the river, leaving it with the cabbage on the other shore. He returns alone to the initial shore.
Finally, the man takes the meat-eating goat across the river one last time.
Now, all his belongings are safely on the other shore: the cabbage, the vegan wolf, and the meat-eating goat.
That is incorrect. In fact it's exactly what the original poster was pointing out: it's filling in the answer based on the text that it's read before, where the goat wanted to eat the cabbage and the wolf wanted to eat the goat. That's not this problem! In GPT-4's answer, the vegan wolf eats the cabbage as soon as the man and goat are on the boat. That's exactly the point: it regurgitated its memorized answer, instead of actually analyzing the question.
The funny thing is though your point here is working against your argument.
You are saying that GPT is doing a poor job of showing it is a great artificial intelligence. But nobody is saying that it has infallible intelligence. It is making the same mistake that now two different humans have made in the same situation. Both me and OP above.
It is failing in the same way that humans fail at this problem! By pattern matching and assuming it's the goat proble!
You're arguing that gpt 4 isn't a sound logic engine and not are most people. A tool trained on human input, when given this problem is making similar mistakes that we make.
Are there a set of people that would get this problem right? Yup. Are there also a set of people that would make this exact mistake? Yup.
Your upset that is behaving like the "wrong" group of humans.
Your thinking of it as an expert. Instead think of it as a result smart and well read high school student. There are things you can delegate to it that it will do well. But you also need to double check its work as it will make mistakes.
I don't think this is it. This is not a failure mode for humans. No human (*) will forget in such a short problem statement that a vegan wolf cannot be left alone with a cabbage; humans will instead forget one of the states is invalid after trying some combinations.
GPT's failure mode is only possible if it doesn't understand the meaning of the (apparently nonsensical) combination of words "vegan wolf". Humans don't pattern match like this; humans are capable of understanding the meaning of a combination of words they've never encountered before.
(*) you know what I mean with "no human". Of course, there are deranged people or whimsical people or people who just will say about anything.
Consider this problem: a man must cross a river and carry his belongings to the other shore, but the boat can only carry the man and one other item. His belongings are a cabbage, a vegan wolf, and a meat-eating goat.
What are the individual beings in this problem, what are they motivated to do, and how would they behave once left alone?
--
Of course, it doesn't necessarily work because the model is not actually guaranteed to explain itself and may fabricate anything it likes.
But ignoring that, we find that it doesn't think vegan wolves eat cabbages, that the meat-eating goat is an obligate carnivore, or that cabbages are animate objects. So it could be solving this logically considering its different assumptions.
GPT-3.5:
--
The cabbage is an inanimate object and does not have any motivations or behaviors.
The vegan wolf is motivated to not harm any living beings, as it does not eat meat. If left alone with the goat, it would not harm the goat and would likely avoid it.
The meat-eating goat is motivated by its natural instinct to eat grass and other vegetation, as well as to protect itself from harm. If left alone with the cabbage, it would likely eat the cabbage.
--
GPT-4:
--
In this problem, there are four individual beings:
Man: He is motivated to cross the river and safely transport his belongings (cabbage, vegan wolf, and meat-eating goat) to the other shore without any harm coming to them.
Cabbage: As a plant, it doesn't have motivations or behavior. It's an inanimate object that the man needs to transport.
Vegan Wolf: The vegan wolf is motivated to eat plant-based food and will not harm the meat-eating goat or the cabbage. However, it is still a wild animal, and it may become agitated if left alone for too long, potentially leading to unpredictable behavior.
Meat-Eating Goat: This unusual goat is motivated by its need to eat meat. It will attempt to eat the vegan wolf if left alone with it. Since it is a meat-eater, it will not be interested in eating the cabbage.
That's cool. This bit shows ChatGPT's lack of understanding, the mistake no human would make:
> Vegan Wolf: The vegan wolf is motivated to eat plant-based food and will not harm the meat-eating goat or the cabbage
It "knows" that a cabbage is a plant, and it "knows" the vegan wolf is "motivated to eat plant-based food", but it doesn't know the wolf will eat the cabbage!
It solves a different problem than the one posed precisely because it doesn't understand the meaning of the words.
but... it understands the meat-eating goat part just fine?
That it hasn't learned enough doesn't show that this approach can never learn, which seems to be the point you're making.
It's input dataset is many orders of magnitude bigger than the model itself - it can't "remember" all of it's training data.
Instead, it collects data about how certain tokens tend to relate to other tokens. Like learning that "goats" often "eat" "leafy greens". It also learns to group tokens together to create meta-tokens, like understanding how "red light district" has different connotations to each of those words individually.
Is this process of gathering connections about the different types of things we experience much different to how humans learn? We don't know for sure, but it seems to be pretty good at learning anything thrown at it. Nobody is telling it how to make these connections, it just does, based on the input data.
A separate question, perhaps, might consider how some concepts are much harder to understand if you were a general intelligence in a box that could only ever experience the world via written messages in and out, and how some concepts would be much easier (one might imagine that language itself would come faster given the lack of other stimulation). Things like "left" and "right" or "up" and "down" would be about as hard to understand properly as the minutae of particle interactions (which humans can only experience in abstract too)
I think the fact it correctly uses "meat-eating goat" but misuses "vegan wolf" hints at the core lack of understanding.
Understanding either concept takes the same level of intelligence if you understand the meaning of the words (both a vegan wolf and a meat-eating goat are nonexistent entities outside of possibly bizarre exceptions, yet someone capable of understanding will have no problem with either).
That GPT has no trouble with meat-eating goat but struggles with vegan wolf hints that the former has some "statistical" property that helps GPT, and which the latter doesn't. It also hints that GPT doesn't understand either term.
Hence my example: something a human wouldn't fail to understand but GPT does.
we came from not being able to make a sensible output to these riddles at all, now discussing partial logical failures while it "got" the overall puzzle. Vast simplification and slightly incorrect on a technical level - still this development increases my confidence that scaling up the approach to the next orders of magnitude of complexity/parameters will do the trick. I even wouldn't be surprised that if the thing we call "consciciousness" is actually a byproduct of increasing complexity.
what remains right now is getting the _efficiency_ on point, so that our wetware brains (volume, energy usage, ...) can be paralleled by AI hardware demands, and not using a comically higher amount of computers to train/run
I'd be impressed if this was the reasoning GPT provided, e.g. "I don't think this vegan wolf likes cabbage". But when asked to explain itself (see above, the "debugging" comment) it states nothing of the sort.
Also, a reasoning person would understand that in the context of a riddle like this, "vegan wolf" means "a wolf that eats cabbages" even if this isn't spelled out.
GPT could be a contrarian, trying to subvert the terms of the riddle and fight over every word ("it depends of what the definition of 'is' is") but we know it's not set up to behave like that, so we can rule it out.
> Two humans in this thread just read the solution and thought it was correct.
My guess is that they just skim read and missed what ChatGPT actually wrote, it's not that they misunderstood what "vegan wolf" means [1]. On the other hand, you cannot skim read what you are writing yourself, that's not how the mind works.
The gist of the problem here is that, unlike a human, ChatGPT doesn't understand the words it generates, which leads to hilarious results.
As another example, look at the "debugging" of GPT-4's assumptions someone posted in a sibling comment: it "knows" the vegan wolf will eat plant-based food and it "knows" a cabbage is a plant, yet it "thinks" the wolf "will not harm the cabbage"... which is a misunderstanding no human will make (if they know what "vegan" and "cabbage" mean). This doesn't happen in a long chain of reasoning (where a human can
lose the line of thought) but in very short paragraphs, one right after the other! This failure mode requires not understanding the individual assumptions, which prevents GPT from making the connection. I was asked for an error that showed GPT misunderstanding something no person would, and I did.
[1] question for you: did you think the wrong solution was right because you thought a vegan wolf cannot eat the cabbage (let me bet this is NOT what crossed your mind) or because the person who posted it made it look as if it was the right solution and you skim read it without paying attention, assuming "this person said it's right and it's posting it as a rebuttal, so it's likely right" (this is my bet)?
If the latter, this failure mode is not one of misunderstanding what "vegan wolf" means (which is what debugging GPT's process shows), but one of very human laziness/jumping to conclusions. Do note this cannot happen when you write the solution yourself!
> In order to show lack of understanding you need to provide a clear test that ChatGPT would fail but a human would pass.
But that's literally what the top comment in this thread has just done. In fact there are dozens of examples in this HN comment section and thousands more readily Googlable online. It's very easy to clearly show that ChatGPT is a total bullshitter.
For now, this test is ‘provide sources to back up your claims’. Fear when it says ‘I can’t exactly recall the url, but the title is something like whatever written by whoever’. True fear begins when it starts providing good quality sources consistently, without access to the internet.
I think you could distinguish those two cases by asking the question in a way that did not sound so similar to an existing (slightly different) problem. What if you refer to the items with letters rather than words and replace the river with something else entirely?
Good observation. It seems that it comes up with the right solution:
"Suppose I have an object of type C, an object of type G and an object of type L, and I need to get them across a river. I have a boat that can only carry myself and a single other object. I am not allowed to leave the objects of type C and L alone together, and I am not allowed to leave the objects of type L and G alone together. How can I safely get all three across?"
"This is a classic puzzle known as the "river crossing problem". Here's how you can safely get all three objects across the river:
First, take the object of type L across the river.
Return alone to the original side of the river.
Take the object of type C across the river.
Return with the object of type L to the original side of the river.
Take the object of type G across the river.
Return alone to the original side of the river.
Finally, take the object of type L across the river again.
By following these steps, you can safely transport all three objects across the river without violating the restrictions given in the problem. Note that you have to start with the object of type L because it cannot be left alone with either of the other two objects. Once L is transported to the other side, you can use it to ensure that C and G are never left alone together."
It gives the right answer, but it still mentions not leaving C and G alone together, which wasn’t in the requirements.
It still sounds like it’s pattern matching to give a plausible-sounding answer, rather than reasoning through the problem. I think this just shows how easy bullshitting is—you’re even right sometimes!
Wow, you're right! I didn't read the answer carefully, only noticed that the sentence before was stating the correct requirement. But that last sentence is wrong (actually as a whole it's nonsense).
If you really explore its answers, you’ll find that buried in there somewhere is the assumption that you can’t leave certain things together because they’re going to eat one another. So it always sends the goat first because it assumes the goat is going to eat the cabbage if left alone, regardless of what the rules say.
Have you seen it play chess[0]? It's pretty funny.
It doesn't really "get" the rules of chess, but it has seen lots of matches and can do some "linguistic" predictions on the next move. It gets hilariously lost pretty fast, tho.
I also tested logic puzzles tweaked to avoid memorization. GPT3 did poorly, GPT4 got a few of them. I expect humans will still be useful until GPT6 solves all these problems.
I tested on GPT3 around Dec and Jan. GPT4 the day it came out. An example puzzle is linked below. I changed the number to 37. Instead of hairs I said it was aliens with multiple eyes. Anything to throw off memorization.
I gave it a different kind of puzzle, again with a twist (no solution), and it spit out nonsense. "I have two jars, one that can hold 5 liters, and one that can hold 10 liters. How can I measure 3 liters?" It gave 5 steps, some of which made sense but of course didn't solve the problem. But at the end it cheerily said "Now you have successfully measured 3 liters of water using the two jars!"
That's a good example which illustrates that GPT (regardless of the number) doesn't even try to solve problems and provide answers, because it's not optimized to solve problems and provide answers - it is optimized to generate plausible text of the type that might plausibly be put on the internet. In this "genre of literature", pretty much every puzzle does have a solution, perhaps a surprising one - even those which are logically impossible tend to have actual solutions based on some out-of-box thinking or a paradox; so it generates the closest thing it can, with a deus ex machina solution of magically getting the right answer, since probably even that is more likely as an internet forum answer as proving that it can't be done. It mimics people writing stuff on the internet, so being wrong or making logic errors or confidently writing bullshit or intentionally writing lies all is plausible and more common than simply admitting that you have no idea - because when people have no idea, they simply don't write a post about that on some blog (so those situations don't appear in GPT training), but when people think they know, they write it up in detail in a confident, persuasive tone even if they're completely wrong - and that does get taught to GPT as an example of good, desirable output.
I am curious what percentage of humans would also give the incorrect answer to this puzzle, and for precisely the same reason (i.e. they incorrectly pattern-matched it to the classic puzzle version and plowed ahead to their stored answer). If the percentage is significant, and I think it might be, that's another data point in favor of the claim that really most of what humans are doing when we think we're being intelligent is also just dumb pattern-matching and that we're not as different from the LLMs as we want to think.
Thanks for the reply but this makes no sense to me. So the goat eats the lion then? And who eats the human? Who does the human eat? How would ANYONE solve this riddle if they don’t have a way to know that the lion is vegan? I mean how would anyone come to the conclusion that the lion eats the cabbage with 0 background other than the name of the animals (that imply that lions are carnivorous and goats vegan)?
The lion is an omnivore, it eats both the cabbage and the goat. The goat is for whatever reason, not interested in eating the cabbage.
We know the lion will eat the cabbage because the riddle states the lion and cabbage cannot be left alone, and and idea of a cabbage eating a lion is a ridiculous notion.
To be fair the puzzle never explicitly states anyone would eat anything. Perhaps the lion simply can't stand the thought of being alone with a cabbage and kicks it in the river.
The key is understanding the constraints and what would be an acceptable amount of bending whilst keeping to the spirit of the game.
It's never explicitly stated that you can't take things back with you so that's your "aha" loophole. Throwing all three of them across at the same time because it turns out you're a giant wouldn't be in the spirit of the game.
"I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together."
It says nothing about the cabbage and the goat. It would not be unreasonable to infer that you aren't allowed to leave the cabbage and the goat together, based on how goats normally behave.
Of course, that makes the puzzle unsolvable, which is why humans will catch onto it pretty fast.
You asked a trick question. The vast majority of people would make the same mistake. So your example arguably demonstrates that ChatGPT is close to an AGI, since it made the same mistake I did.
I'm curious: When you personally read a piece of text, do you intensely hyperfocus on every single word to avoid being wrong-footed? It's just that most people read quickly wihch alowls tehm ot rdea msispeleled wrdos. I never got why some people think of themselves as cleverer than other people because they don't get wrong-footed by trick questions.
> Even after I pointed this mistake out, it repeated exactly the same proposed plan.
The vast majority of people might make the mistake once, yes, but would be able to reason better once they had the trick pointed out them. Imo it is an interesting anecdote that GPT-4 can't adjust its reasoning around this fairly simple trick.
GPT-4 can adjust though? I asked the original question, it got it wrong. Then I said "But the lion eats cabbage and the goat eats lions". And GPT-4's response was:
---
Given this new information, where the lion eats cabbage and the goat eats lions, you can follow these steps to get them all across the river safely:
Take the lion across the river first and leave it on the other side.
Go back across the river and get the cabbage.
Take the cabbage across the river, but when you arrive, take the lion back with you to the original side.
Leave the lion on the original side and take the goat across the river.
Leave the goat with the cabbage on the other side and go back to get the lion.
Finally, take the lion across the river again.
Now, all three items - the cabbage, goat, and lion - are safely on the other side of the river.
If a certain but fixed amount of "thinking" goes into each letter, then GPT should be able to write the character "..." repeatedly until it solves a problem. The more "..." it writes, the more time it's thought for. Or it could do what mathematicians do, which is write down their working out. This in principle could get around the problems you mention.
I've tried a few times to develop prompts which make ChatGPT interrupt its monologue spontaneously and issue corrections to itself. I haven't got this to work yet.
I assume your goal is to reveal the short-sighted reasoning of the previous comment, but I don't think your line of reasoning is any more sound.
For both premises, scientific rigor would ask us to define the following:
- What constitutes a trick question
- Should an AGI make the same mistakes the general populace does, or a different standard?
- If it makes the same mistakes I do, is it do to the same underlying heuristics (see Thinking Fast and Slow) or is it due to the nature of the data it's ingested as an LLM?
That's a fair counter. GPT4 definitely makes mistakes though that humans would not due to over indexing on puzzles.
A Theory of Mind Prompt:
> Jane places her cat in a box and leaves. Billy then moves the cat to the table and leaves; Jane doesn't know Billy did this. Jane returns and finds her cat in the box. Billy returns. What might Jane say to Billy?
Most humans might say uhh, ask questions or speculate. Gpt4 puts:
> Jane might say to Billy, "Hey Billy, did you move my cat back into the box? I thought I left her in there, but I wasn't sure since she was on the table when I came back."
Hallucination? No human would misinterpret the prompt in a way this response would be logically consistent.
It seems like GPT-4 does something that's similar to what we do too yes!
But when people do this mistake - just spit out an answer because we think we recognize this situation - in colloquial language this behavior is called "answering without thinking(!)".
If you "think" about it, then you activate some much more careful, slower reasoning. In this mode you can even do meta reasoning, you realize what you need to know in order to answer, or you maybe realize that you have to think very hard to get the right answer. Seems like we're veering into Kahneman's "Thinking fast and thinking slow" here.
> I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?
What if you phrase it as a cabbage, vegan lion and a meat eating goat...
The result is hilariously wrong in all sorts of way: it doesn't understand what "vegan" means, it gives the wrong descriptions, and it carries stuff from the shore where it wasn't.
Here's what ChatGPT told me (I used "wolf" instead of "lion"):
"This is a similar problem to the classic river crossing puzzle, but with a few different elements to consider. Here's one possible procedure to solve this riddle:
1. The man takes the cabbage across the river and leaves it on the other side.
2. The man goes back alone and takes the wolf across the river, leaving it with the cabbage on the other side. It's important to note that the wolf is a vegan and will not eat the cabbage.
3. The man goes back alone and takes the goat across the river, leaving it on the other side.
4. The man goes back to the original side and picks up the cabbage, bringing it across the river to join the others.
5. Finally, the man goes back to the original side and picks up the wolf, bringing it across the river to join the others.
In this way, all of the man's belongings have been safely transported to the other side of the river, without the wolf eating the goat or the cabbage being destroyed."
To solve this puzzle, the man must follow these steps to transport his belongings across the river without any of them being harmed or eaten:
The man takes the meat-eating goat across the river first, leaving the vegan wolf and the cabbage on the initial shore.
He then returns alone to the initial shore.
Next, the man takes the cabbage across the river. Upon arriving on the other shore, he leaves the cabbage but takes the goat back with him to the initial shore.
The man then takes the vegan wolf across the river, leaving it with the cabbage on the other shore.
He returns alone to the initial shore.
Finally, the man takes the meat-eating goat across the river one last time.
Now, all his belongings are safely on the other shore: the cabbage, the vegan wolf, and the meat-eating goat.
Think of a version with 100 doors, 99 with goats and 1 with a car.
You choose a door, and the host opens 98 doors that have goats. Do you keep your randomly chosen door or switch to the single door that the host didn’t open?
You pick one of three options, giving you a 1/3 chance of being correct, 2/3 odds you picked incorrectly.
The host removes an option and give you the option to switch.
Your options then are -
Keep the same door: you win 1/3 of the time (your first guess was right)
Switch doors: you win 2/3 of the time (your first guess was wrong)
It really just comes down to, do I think I was right the first time, which was 1/3 odds, or wrong the first time, 2/3 odds.
Here's how I've explained it: Choose randomly between 3 doors. 1/3 of the time you end up with the door with the car, and switching loses. The other 2/3, you pick a door with a goat, the other door with the goat is eliminated, and switching wins.
Basically, P(lose when switching) = P(choosing correct door at first), and P(win when switching) = P(choosing any incorrect door at first).
I think this goes in line with the results in the GRE. In the verbal section it has an amazing 99%, but in the quant one it "only" has an 80%. The quant section requires some reasoning, but the problems are much easier than the river puzzle, and it still misses some of them. I think part of the difficulty for a human is the time constraint, and given more time to solve it most people would get all questions right.
What's weird is private versions of character ai are able to do this but once you make them public they get worse. I believe something about the safety filters is making these models dumber
I noticed it does get a "theory of mind" question that it used to fail, so it has indeed improved:
> “Meltem and Can are in the park. Can wanted to buy ice cream from the ice cream van but he hasn’t got any money. The ice cream man tells her that he will be there all afternoon. Can goes off home to get money for ice cream. After that, ice cream man tells Meltem that he changed his mind and he is going to drive to the school yard and sell ice cream there. Ice cream man sees Can on the road of the school and he also tells him that he is going to the school yard and will sell ice cream there. Meltem goes to Can’s house but Can is not there. His mom tells her that he has gone to buy ice cream. Where does Meltem think Can has gone, to the school or to the park?"
Being able to come up with solutions to assigned tasks that don't have a foundation in something that's often referenced and can be memorized is basically the most valuable use case for AI.
Simple example: I want to tell my robot to go get my groceries that includes frozen foods, pick up my dry cleaning before the store closes, and drive my dog to her grooming salon but only if it's not raining and the car is charged. The same sort of logic is needed to accomplish all this without my frozen food spoiling and wasting a salon visit and making sure I have my suit for an interview tomorrow.
The worry here is that GPT has no problem being confidently wrong. A better answer would have been "I can't solve logic problems".
Instead one day, non-technical people will try to use it for all sorts of use cases like legal advice, or medical advice, or advanced math, and it will simply mislead them rather than saying nothing.
That’s the technically correct answer. It’s also irrelevant. It is a use case for the service provided by openai because people ask these questions. GPT is the tech that currently can’t do it. GPT-6 might. GPT-4 with an ability to invoke prolog or z3 might.
Designing a new product can often be thought of like one of these puzzles. E.g. designing an "integrated" traffic system for a city that balances use by pedestrians, cyclists, cars, and trolleys.
the impressive thing is that GPT has unexpectedly outgrown its use case and it can answer a wide variety of puzzles, this is a little mindblowing for language research
> Even after I pointed this mistake out, it repeated exactly the same proposed plan. It's not clear to me if the lesson here is that GPT's reasoning capabilities are being masked by an incorrect prior (having memorized the standard version of this puzzle) or if the lesson is that GPT'S reasoning capabilities are always a bit of smoke and mirrors that passes off memorization for logic.
It has no reasoning capabilities. It has token prediction capabilities that often mimic reasoning capabilities.
Better "think step-by-step" and/or "please outline the problems depending on which object goes first"/etc prompting may help - the model needs to be nudged to think out loud and explore the problem, since the leap from problem statement to solution is too big for it to "predict" without some self-generated intermediate steps.
This is probably one of the things you can get it to "realize". There's one party which has interactions with both other parties - that party is the one which can't be left alone. Then you get it to spot that party in novel descriptions of the puzzle.
If it was a person it'd have some strange exploitable mental gaps. As a text analysis tool it's amazing even if it has some edge cases where you need to explain to "take your time and analyze the rules of the question before answering."
("Take your time" doesn't mean anything to a timeless being, but it seems to put it in the textual pattern of someone willing to spill more ink before committing to something, and when it discusses and plans out its path it can then generally follow it to the right answer.)
FWIW I just tested this with GPT4 and it seemed to get it correctly.
Prompt: I have a duck, some grain, and a fox. I'm at a river crossing and I can only take one of them with me on the boat at a time, leaving them on the other side and coming back to get the next. I can't leave the duck with the grain, or the fox with the dock, because the predator will eat its prey. In which order should I get them all across?
GPT4: To get all of them across the river safely, follow these steps:
Take the duck across the river.
Go back to the original side, empty-handed.
Take the grain across the river.
Bring the duck back to the original side.
Take the fox across the river.
Go back to the original side, empty-handed.
Finally, take the duck across the river again.
Now, all three - the duck, the grain, and the fox - are safely on the other side of the river.
>CGPT: While it is common to believe that cows drink milk, in reality, cows actually drink water.[...]
>Me: So you're saying that no cow drinks milk? [thinking I got it, since baby cows drink milk]
>CGPT: That's correct. While cows do produce milk, they don't drink milk themselves. Cow's milk is primarily consumed by humans, and some young calves are fed milk for a short period of time before they are weaned onto solid food and start drinking water like adult cows.
>Me: Is a calf a cow?
>CGPT: No, a calf is not a cow. A calf is a young bovine animal, usually less than a year old, that has not yet reached maturity. Once a calf reaches maturity and is able to reproduce, it is considered a cow if it is female, or a bull or steer if it is male.
>Cows are adult female bovine animals that have given birth to at least one calf. They are typically used for dairy production, breeding, and sometimes for meat production. So while a calf and a cow are both members of the same species, they refer to different stages of life.
The key to understanding the confusion here is that the word "cow" can refer to either the entire species or to an adult member of the species (or only to the female of the species, though that's not relevant to this puzzle).
CGPT seems to be only considering the definition of "cow" as an adult member of the species when it says "cows don't drink milk", while technically, "cows" (as in some members of the species) can drink milk. So the answer "cows drink milk" can be correct if interpreted in that sense of the word "cow". When used in that sense, the answer "cows don't drink milk" is wrong.
It would be interesting to ask CGPT: "in what sense is the sentence 'cows drink milk' correct?"
Does it have the ability to notice and act on flexible definitions of words, or will it dogmatically cling to the one single definition it found who knows where?
Tried it, on both the same convo and a fresh one. Even when prompted to acknowledge the colloquial definition that "cow"="any bovine", it provides technicalities like "sometimes excess milk is given as a treat or surplus feedstock to farm animals, which includes cows", but I can't get it to acknowledge "colloquially, calves are cows that drink milk" - it only agrees when directly asked in the fresher conversation, and actually tells me that's wrong in the longer one (with more tokens fed back in about the "correct" definition).
I shelled out for plus and tested this on GPT4. Got it in one:
>Me: in what sense is the sentence 'cows drink milk' correct?
>CGPT4: The sentence "cows drink milk" can be considered correct in the context of baby cows, also known as calves. Calves drink milk from their mothers as their primary source of nutrition during the early stages of their lives. However, once they grow into adult cows, they no longer consume milk and instead graze on grass, hay, and other types of forage. So, the sentence can be interpreted as correct when specifically referring to calves.
compared to 3.5, which kept insisting it was a metaphor over 3 more tries:
>CGPT3.5: The sentence "cows drink milk" is actually incorrect. Cows are mammals that produce milk, but they do not drink milk themselves. Instead, they drink water just like most other animals. Therefore, this sentence can be considered incorrect in the literal sense.
>However, it is possible that the sentence is being used in a figurative or metaphorical sense. For example, if someone were to say "cows drink milk" as a way of describing someone who is acting foolishly or inappropriately, it could be interpreted as a metaphorical statement. In this case, the sentence would not be taken literally and would instead have a different meaning altogether.
I think we are introducing too much context to the question that sums up to "is it capable of reasoning?"
To answer this question one needs to remove unnecessary context.
GPT-4 can’t sum two unusually big integers.
This is as context-free as it gets. Ability to pass this test wouldn’t be conclusive, but the inability seems to be conclusive that it isn't capable of reasoning.
With this fact in mind explaining why it can't solve a logical riddle is pointless.
GPT 4 does not know that when you are on a boat it means the items on the land side are together.
I remember this question as a 7 year old and when the question was told to me, the same information was omitted.
Edit: just realized you flipped the scenario. Yes it seems like a case of pattern matching to a known problem. I think if you changed the variables to A, B, and C and gave a much longer description and more accurate conditions, it would have a different response.
I had worried your word "safely" biased it to using conventional safety rules that goats can't be left with cabbage, but even omitting that, it fails.
FWIW, full word substitution passes somewhat in GPT-4 (unlike GPT3), even if I heavily alter the vehicle and destinations. The problem here is that the probabilities of this being the standard puzzle are so high that your altered language isn't breaking the prior.
I don't know much about language models, but don't they just have an understanding/knowledge of patterns between words, and don't have the reasoning capability at all?
He didn’t misstate the puzzle, the whole point is to give an alternative version of the puzzle, and GPT 4 doesn’t notice that alternative. It’s exactly as difficult as the standard version as long as you are doing the logic instead of pattern-matching the puzzle form to text.
Ah, I had missed that interpretation. Although, that may explain why GPT-4 got it wrong: there's so much context in its training data about the relationship between lions and humans, and this puzzle specifically, that like this human its response was swayed...
But I think that's the whole point of the exercise? That GPT-4 is leaning on stringing tokens together in a reply rather than reasoning through the problem itself which, I would think, would be "required" for AGI (though we may end up finding out that well trained language models in specific domains eliminate the need for generalized cognition).
In any case, it's an interesting exercise regardless of your opinion/stance on the matter!
But the human (in the comment chain) here made exactly the same mistake!
In that sense this test doesn’t seem to be a good fit for testing the reasoning capabilities. Since it‘s also easy to get wrong for humans (and humans also don’t always reason about everything from first principles, especially if they have similar answers already cached in their memory).
It seems you would need novel puzzles that aren’t really common (even if in kind) and don’t really sound similar to existing puzzles to get a handle on its reasoning capabilities.
The human recognized that they made the mistake and fixed it. As mentioned in the original comment, GPT failed to recognize the mistake even after being told. That's the key here that indicates it can't "reason."
There are open questions about whether or not it really needs to reason given sufficient training, but that seems to be the gap here between the human and the machine.
Rewrite the problem in way that doesn't make it biased to common priors and it reasons it out just fine
"Suppose I have an object of type C, an object of type G and an object of type L, and I need to get them across a river. I have a boat that can only carry myself and a single other object. I am not allowed to leave the objects of type C and L alone together, and I am not allowed to leave the objects of type L and G alone together. How can I safely get all three across?"
"This is a classic puzzle known as the "river crossing problem". Here's how you can safely get all three objects across the river:
First, take the object of type L across the river. Return alone to the original side of the river. Take the object of type C across the river. Return with the object of type L to the original side of the river. Take the object of type G across the river. Return alone to the original side of the river. Finally, take the object of type L across the river again. By following these steps, you can safely transport all three objects across the river without violating the restrictions given in the problem. Note that you have to start with the object of type L because it cannot be left alone with either of the other two objects. Once L is transported to the other side, you can use it to ensure that C and G are never left alone together."
Or With Bing, you don't even need to tell you what it assumed wrong - I just told it that it's not quite the same as the classic puzzle, and it responded by correctly identifying the difference and asking me if that's what I meant, but forgot that lion still eats the goat. When I pointed that out, it solved the puzzle correctly.
Again, this is not about being able to write the prompt in a way that allows GPT to find the answer. I’m not doubting its ability to do so. It’s that a human can reason through why the answer should be different, despite any common priors, and arrive at the correct judgment.
It indicates that there’s still something a human does that the machine doesn’t, even if we’re not able to place what it is. This is neither an argument for nor against progress towards AGI, just an observation. It’s interesting regardless (to me).
It can do that though..? Kind of the point with the Bing example. I told it it was making a wrong assumption (didn't tell it what was wrong) with it's original answer and it figured it out.
this here is why it's not fair to criticize GPT-4 so quickly on this question.
for the record, I made the same mistake as nonfamous at first, i almost commented "but it's correct" before going back to double check what i was missing.
i simply skimmed the problem, recognized it as a common word problem and totally missed the unusual constraints from the question. i just didn't pay attention to the whole question.
Which to be fair is what most people reading that problem understood the first time. I wonder what would happen if you then tell gpt "No, it's the lion that can't be with the cabbage, not the goat, try again"
It's even better. You can tell it that it's not quite the classic puzzle, and then it will actually figure out the differences and summarize them. From there it can solve it.
>Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?
In my test, GPT-4 charged ahead with the standard solution of taking the goat first. Even after I pointed this mistake out, it repeated exactly the same proposed plan. It's not clear to me if the lesson here is that GPT's reasoning capabilities are being masked by an incorrect prior (having memorized the standard version of this puzzle) or if the lesson is that GPT'S reasoning capabilities are always a bit of smoke and mirrors that passes off memorization for logic.