I personally think there are multiple reasons for this situation.
* It is fun to learn initially, to see something new working. Once you get something working not many people see fun in spending a lot of time in getting it to work better.
* There is huge amount of introductory materials (guides, tutorials, examples) but the amount of available materials falls drastically as you start to progress.
* Only some people are able to actually think in abstract terms required to "create" new knowledge based on existing facts. Beginners can advance quickly by "recreating" -- executing tutorials, copying existing code, etc. It is relatively easy to use these as building blocks for a simple application. But as you progress you have to figure out more and more new knowledge, understand underlying principles. This is what many people either don't feel comfortable doing or don't feel is necessary to do or are just plainly incapable.
* It gets more difficult to work with other people in your team as you create knowledge in a given topic. There is tendency to push back when team member tries to introduce something new that is not clearly recreation of accomplishment of somebody else available on the Internet.
* Creating new knowledge in the topic is a huge risk in that it is unknown payoff for large amount of honest work. There is not much risk in following existing tutorials, it is pretty much guaranteed that it is possible to recreate accomplishment others did. When you create new knowledge (for example new patterns, principles, guidelines) it is likely you are going to make mistakes. This fact may cause people uneasy and dissuade them from further advancement in the topic.
* Getting mediocre in any specific topic is frequently seen as good return on investment. For example, as an architect I would maybe not want to get expert at any of the frameworks I know. I see this as a reasonable tradeoff which allows me to tackle other problems as soon as I think I know "enough" about particular topic.
> * There is huge amount of introductory materials (guides, tutorials, examples) but the amount of available materials falls drastically as you start to progress.
Aye. My coworker described it as being able to find 100 guides on how to hammer nails and build a small bench. "Shed building for beginners" or something. But then the next exercise is "build a house" and the one after that is "build a shopping mall".
I was assigned to work on machine learning three years ago. Literally an overnight "the project changed, you're doing this now, here's the project update memo for your contract."
Your post exactly describes ML on the internet: first you find a billion "check out my intro to ML" Medium articles and git repos that spin up OpenCV on Tensorflow ("small bench"), then you move to the Tensorflow docs ("build a house"), then you move to Arxiv.org ("theories of city planning"). Once you step into that middle zone, you're basically stuck reading academic texts. As a 50-something engineer, it was a kick in the balls getting started with this stuff. Fortunately this contract is related to hardware optimization, which I've been doing for decades, but this particular variant of it was a curve-ball.
It is an interesting back-and-forth: I find myself exchanging information with lots of new-college-grads (permanent hires, not contractors) that all have sharply coiffed beards and heavy denim jeans. Two of them are pretentious and fight to be top dog (which is hilarious from my old, jaded POV), but the rest are genuinely good kids that want to learn, and vice versa, teach me what they know. It's fun. I'll miss 'em when my contract ends.
I think the biggest reason is demographic in nature: The rapidly expanding population of professional programmers means that programmers are regularly building their skills on third-hand advice while simultaneously encountering situations nobody they know has seen.
In such a climate, they aren't really competing for the job on the basis of merit, so much as they are posturing that they are a good candidate - because they learned that that's what one does to get work and protect their family. And there is so little expertise around that nobody can call them out. When they get hired, the posturing turns into gatekeeping in short order. They aren't curious about the work, they just want to keep the position. When you fill the ranks with uncurious individuals you get stagnation, because nobody is pushing them forward.
In other professions the process of education and certification cuts most of these people out early on, which comes with its own downsides but does impose some minimum standard of capability which they may go on to use or misuse. But the problem in programming is that the tools of programming address a multitude of problem domains, outstripping what could be reasonably certified - that's why the profession has gotten so big, so quickly. It's all research, all the time - independent, unverified, unscientific research.
In response to this we've ended up with the whiteboard coding interview, which basically aims to do the task of certification within the span of a few hours. It's not very good at that - these interviews are run in an inconsistent and often unprofessional manner too.
* It is fun to learn initially, to see something new working. Once you get something working not many people see fun in spending a lot of time in getting it to work better.
* There is huge amount of introductory materials (guides, tutorials, examples) but the amount of available materials falls drastically as you start to progress.
* Only some people are able to actually think in abstract terms required to "create" new knowledge based on existing facts. Beginners can advance quickly by "recreating" -- executing tutorials, copying existing code, etc. It is relatively easy to use these as building blocks for a simple application. But as you progress you have to figure out more and more new knowledge, understand underlying principles. This is what many people either don't feel comfortable doing or don't feel is necessary to do or are just plainly incapable.
* It gets more difficult to work with other people in your team as you create knowledge in a given topic. There is tendency to push back when team member tries to introduce something new that is not clearly recreation of accomplishment of somebody else available on the Internet.
* Creating new knowledge in the topic is a huge risk in that it is unknown payoff for large amount of honest work. There is not much risk in following existing tutorials, it is pretty much guaranteed that it is possible to recreate accomplishment others did. When you create new knowledge (for example new patterns, principles, guidelines) it is likely you are going to make mistakes. This fact may cause people uneasy and dissuade them from further advancement in the topic.
* Getting mediocre in any specific topic is frequently seen as good return on investment. For example, as an architect I would maybe not want to get expert at any of the frameworks I know. I see this as a reasonable tradeoff which allows me to tackle other problems as soon as I think I know "enough" about particular topic.