Good! DNNs unlock semantics (parsing, transforming, producing). That's the basis of general intelligence, not encyclopedic random string recall. Models shouldn't burn ungodly quantities of compute emulating DDR5 with their working memory. We need machines that think better, not memorize well. We already have plenty of those.
Massive context windows, and their needle tests, are misguided. We won't reach human-level AGI by basically inventing a natural language RDBMS. Our resources should primarily target better reasoning systems for our models, reinforcement learning, etc.
If we can build a GPT4-level problem solving system that coincidentally also can't remember telephone numbers, I'll consider it major progress.
Memorization usually refers to training data. It's often useful to have something that can utilize instructions losslessly, which is the distinction between these models.
Massive context windows, and their needle tests, are misguided. We won't reach human-level AGI by basically inventing a natural language RDBMS. Our resources should primarily target better reasoning systems for our models, reinforcement learning, etc.
If we can build a GPT4-level problem solving system that coincidentally also can't remember telephone numbers, I'll consider it major progress.