>And if it somehow does, no sane government will permit uncontrolled research towards AGI
If there was an effective, distributed means for training LLMs, enough people are passionate about LLMs that the only way governments could stop it is if every country in the world turned into communist China with respect to internet restrictions.
> If there was an effective, distributed means for training LLMs
Is it reasonable to believe that this is possible? Distributed training requires extremely high-bandwidth and low-latency interconnect. The internet ain't that.
Believe me, I dearly want the open-source community to "win". A future where only governments have a monopoly on AI research is absolutely terrifying. Given the known parameters though, that future seems inevitable.
>Is it reasonable to believe that this is possible? Distributed training requires extremely high-bandwidth and low-latency interconnect. The internet ain't that.
The same can be said about piracy and bittorrent. The community torrents is so large that I would bet a lot money they have more computing+bandwidth resources combined than openAI by a wide margin, probably orders of magnitude.
Why low latency? Is the latency here a meaningfully annoying limit in training, that can't be reasonably offset by just adding more compute nodes to the network?
Latency is the limit in the end, but I feel that there's plenty of easy-ish wins to have in redesigning the architecture and training approach to make it an irrelevant in practice.
If there was an effective, distributed means for training LLMs, enough people are passionate about LLMs that the only way governments could stop it is if every country in the world turned into communist China with respect to internet restrictions.