One of the problems people have mentioned for deep learning systems generally is they tend to be maintenance nightmares.

I get the impression that openAI had a lot of resources on-hand when they released ChatGPT that they used to fix problem using reinforcement learning and methods that I'd imagine were more adhoc than the original training process. Hence it seems likely the system winds-up fairly brittle.