I think you're discounting efficiency gains — through a series of individually minor breakthroughs in LLM tech I think we could end up with things like 100M+ token context windows
We've already seen this sort of incrementalism over the past couple of years, the initial buzz started without much more than a 2048 context window and we're seeing models with 1M out there now that are significantly more capable.
We've already seen this sort of incrementalism over the past couple of years, the initial buzz started without much more than a 2048 context window and we're seeing models with 1M out there now that are significantly more capable.