Once I kept refreshing and finally got an English question, it asked me to act like a Linux terminal, and issues pwd, ls, then cd over and over until I gave up. The concept is funny, where I get to act like CrapGPT, but it needs to not get stuck asking the same thing over and over.
Maybe the role reversal breaks most of the RLHF training. The training was definitely not done in the context of role reversal, so it could be out of distribution. If so, this is a glimpse of the intelligence of the LLM core without the RL/RAG/etc tape and glue layers.
