It seems to be sort of a bit wrong more often than it hallucinates.
I've had it straight up invent a library that doesn't exist once, but that seems to be quite rare and you need to be deep in the weeds with a rare problem domain to get that.
More often I ask it how to do something, and it sort of provides an answer, but not quite. So I point out the flaw, and it fixes it, but not quite. Rinse and repeat. After anywhere between 4-10 iterations it's usually quite good. The experience is like code reviewing a really apologetic and endlessly patient junior developer.
Although I think what might be a beginner's saving grace is that it seems to be better at beginner questions than advanced questions, since there are more of them in the training data.
I've had it straight up invent a library that doesn't exist once, but that seems to be quite rare and you need to be deep in the weeds with a rare problem domain to get that.
More often I ask it how to do something, and it sort of provides an answer, but not quite. So I point out the flaw, and it fixes it, but not quite. Rinse and repeat. After anywhere between 4-10 iterations it's usually quite good. The experience is like code reviewing a really apologetic and endlessly patient junior developer.
Although I think what might be a beginner's saving grace is that it seems to be better at beginner questions than advanced questions, since there are more of them in the training data.