The number of people who can actually pre-train a true LLM is very small.

It remains a major feat with many tweaks and tricks. Case in point: the 114 pages of OPT175B logbook [1]

[1] https://github.com/facebookresearch/metaseq/blob/main/projec...