Not programming, but I asked it to generate a list of words that end in common tlds, to see if I could come up with a nice website domain / project name. I said give me words that end in .io, .ws, .is etc.
It failed to follow this simple instruction. I tried to be more and more specific, but no matter what I did it returned words that did not end with the tlds I asked for. it just seemed to give me any old words and throw a . two places before the end.
Compared to the initial experiences I had with ChatGPT it feels like it's suddenly gotten awfully dumb.
To be fair though, I would expect that. It's always been and always will be pretty terrible for anything at the character level, because it works on tokens (i.e. chunks of words) rather than characters, and it can't see individual characters at all.
For example, try asking it to count the number of characters in a word and, no matter if it's right or wrong (seems to be a coin toss, btw -- it can just do a rough guess), ask it "Are you sure?", and keep asking that over and over every time it replies. Chances are it'll keep changing the count back and forth, apologizing every time.
It failed to follow this simple instruction. I tried to be more and more specific, but no matter what I did it returned words that did not end with the tlds I asked for. it just seemed to give me any old words and throw a . two places before the end.
Compared to the initial experiences I had with ChatGPT it feels like it's suddenly gotten awfully dumb.