Afaik it's an open question what's innate and not. But it's a perfectly reasonable position that very little of it is. It seems more like what's innate is a tremendous ability to learn and shape the brain as it grows.

And it's definitely the case that human brains can learn a lot more per amount of knlut data than current neural nets. And that's the big elephant in the room with these language models. It's a huge unsolved problem in machine learning.

One clue to this is that it takes a pretty hefty artificial neural net to simulate a single cortical neuron. Neurons are a lot more intricate than a parameter in a linear equation.

My feeling is that a fundamentally new approach to neural nets is needed for them to learn as efficiently as animal brains do.