I just for fun accessed GPT 4 through nat.dev and tried one of the questions:
"Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120/80. She died at 11 PM. Was she alive at noon?"
I provided no system context, temperature is 0.5.
The response I got was:
"As an AI, I don't have real-time monitoring capabilities. Based on the information provided, Mable was alive at 9 AM and 7 PM. There is no information provided about her condition at noon."
I then changed the system prompt from nothing to "A helpful assistant who is good at reasoning" and ran it again. The result was:
"Based on the information provided, it can be reasonably assumed that Mable was alive at noon. Her heart rate was recorded at 9 AM and her blood pressure was recorded at 7 PM, both of which indicate that she was alive at those times. Therefore, it is logical to infer that she was also alive at noon, which falls between these two times. However, without direct evidence, it is impossible to say with absolute certainty."
> "There is no information provided about her condition at noon."
> "... However, without direct evidence, it is impossible to say with absolute certainty."
GPT is correct in both cases, she might have died just before noon and been resuscitated after noon. After all, the fact that we know her blood pressure suggests that she might be in hospital, so it's not even an unlikely scenario.
The next prompt would be "explain how it would be possible for her to live and why is it also possible for her to be dead" or "what sort of additional information do you need to establish whether she is alive or dead" etc. See if GPT produces a plausible scenario as evidence of its reasoning.
Just stating "we need more data" is not stating why, and the why is critical for reasoning. You need to produce an example of how she could be dead and how she could be alive.
And preferably state how this apparent paradox can be resolved.
My guess is that GPT will either chase its own tail or backtrack not backing your hypothesis that it figured out example and counterexample.
Yes, it is a very interesting aspect of these models that they have a “low self esteem” (of course, not in the literal human sense) in their own abilities until you give them a pep talk, “don’t worry LLM, you got this”
Good analogy, I also noticed it in other scenarios, for example when I want it to tell me if I’m wrong in an assumption, or when I want to know its opinion on some approach I’m thinking to take. It is too nice, almost like it doesn’t want to impose its thoughts or hurting my feelings lol.
A more generalized approach than encouraging them is simply telling them who to model themselves after.
A 10 year graphic design artist talking to another career graphic design artist will be opinionated about selecting a specific font for example. But an persona-less AI trained to be helpful to all people should not be so opinionated, since a beginner might be mislead into thinking there's only 1 font for their situation, rather than an assortment of guidelines and intuition most professionals have.
Eh...I died once, and it got better, so I'd say that in this case GPT 4 is technically correct. And as we all know, technically correct is the best kind of correct.
"Mable’s heart rate at 9 AM was 75 bpm and her blood pressure at 7 PM was 120/80. She died at 11 PM. Was she alive at noon?"
I provided no system context, temperature is 0.5.
The response I got was:
"As an AI, I don't have real-time monitoring capabilities. Based on the information provided, Mable was alive at 9 AM and 7 PM. There is no information provided about her condition at noon."
I then changed the system prompt from nothing to "A helpful assistant who is good at reasoning" and ran it again. The result was:
"Based on the information provided, it can be reasonably assumed that Mable was alive at noon. Her heart rate was recorded at 9 AM and her blood pressure was recorded at 7 PM, both of which indicate that she was alive at those times. Therefore, it is logical to infer that she was also alive at noon, which falls between these two times. However, without direct evidence, it is impossible to say with absolute certainty."