So, as tenured faculty I don't know about NSF advocating for careers in STEM. Maybe that's a good thing maybe its a bad thing.
What I do see driving this ultimately is something broader and more systemic, which is universities being underfunded, administrator heavy, and being run as for-profit businesses.
In many ways, this all reduces to how grants are awarded and funded, and the history of grant funding.
Through the 80s and 90s, and to some extent early 2000s, federal grant funding increased. There was sort of a heydey in that time, depending on field, where there was good funding opportunities, in general, for faculty.
At the same time, state funding to universities started decreasing, and costs started increasing due to the higher ed bubble. Because of indirect costs on grants, which allow universities to charge the federal gov a blanket percentage of a grant, universities were able to skim off a profit almost automatically. It was a sort of win-win it seemed because faculty got to fund their research, unis got a cut to offset decreased funding.
The problems are many, though. First, federal funding went down after a peak, so this money pool dried up. Second, this system sort of led to the pyramid scheme you mention, where grad students get recruited to maintain grants and research programs with nowhere to go, lest universities and faculty lose indirect funding. Third, it prioritizes research based on grant receipts rather than utility, which distorts research priorities in a perverse way (especially given nepotism in grant culture, which has been written about extensively elsewhere). Finally, because of the history of all of this, you kind of end up with a system in which those running the system are either beneficiaries of the 90s and 2000s, and don't realize what it is like now, or came up through that system and have a kind of survivorship bias.
Agencies like NSF and NIH are basically just maintaining any semblance of research in the US at the moment, making up for all the deficiencies that are occurring at the state level.
There are problems with NSF and NIH, but they're not with advocating for STEM grads. They're with how grants are funded. Basically, review panels need to randomly rotate in some kind of jury pool system almost; grants need to be awarded based on some kind of lottery system (proposed by former heads of NIH); indirect cost charges need to be eliminated or made line-item justified; some grants need to be awarded on the basis of publication record without regard to a specific proposal (ala Hungary); universities need to be adequately funded outside of the grant system; and tenure needs to be protected (I say this as someone who is probably going to step down from my tenured position soon).
The problems facing grad students and postdocs are just the tip of the iceberg. They don't end, even when you get a tenure-track position, and even when you get tenure. The system is broken, it's a mess that favors hype and popularity over rigor. What you end up with is the current reproducibility crisis, academic fraud, people manipulating the system to overclaim credit, and many more problems that society is unaware of because they have a stereotyped idea of what happens in science--the sort of "great man in the lab with a eureka moment" which is a false model of scientific process.
I think you're not giving enough weight to the massive increase (as a ratio) of people seeking postgraduate education. Today about 12% of people have at least a masters. And that number continues to increase as education, even in fields without meaningful career prospects, is continually evangelized. By contrast in years from 1975-1995 the total percent of people age 25-29 who had even a bachelors degree was only in the range of 21-25%. That number today is 36%, and again - growing. I doubt the ratio of gross 'revenue' (including grants) to total faculty today is less today than it was in the 'heyday.' Given the massive increases in tuition, I would be quite surprised if it's not vastly higher.
I completely and absolutely agree with you about the perversion of research directions, but I think this can be pin pointed more precisely to just a single point you said - universities are increasingly being run as for-profit businesses. That is really the root of all of these problems.
Yes, and the massive increase in people seeking postgrad education was preceded by a massive increase in people offering postgrad education. AND at the same time, you remind me of working in a college where a project was started to launch a grad program, and the justification made in committee meetings was, "we need to get some grad students around here to teach the classes and free up our time. Otherwise we'll never get ahead."
You are correct about how the current system is the result of a draw down of past heavy funding. The timing of the draw down was different in different fields. An NSF official explained to me in 1990 that, in the 1960's, 90% of grant applications were funded. By 1990, 90% were rejected. The same is true for hiring: in 1945-1970, with the G.I. bill and the explosion of new universities (e.g., in California), almost everyone with a Ph.D. was hired for a tenure track position, and many tenure track positions were held by professors without Ph.D.'s. As you say, faculty only very slowly learns that the systems they came up in are gone, and so they over-prepare young people for a world that is gone. The future may be a combination of the older tradition that professors start with inherited wealth and a new pattern that an academic career begins with 10 years building wealth in business.
What I do see driving this ultimately is something broader and more systemic, which is universities being underfunded, administrator heavy, and being run as for-profit businesses.
In many ways, this all reduces to how grants are awarded and funded, and the history of grant funding.
Through the 80s and 90s, and to some extent early 2000s, federal grant funding increased. There was sort of a heydey in that time, depending on field, where there was good funding opportunities, in general, for faculty.
At the same time, state funding to universities started decreasing, and costs started increasing due to the higher ed bubble. Because of indirect costs on grants, which allow universities to charge the federal gov a blanket percentage of a grant, universities were able to skim off a profit almost automatically. It was a sort of win-win it seemed because faculty got to fund their research, unis got a cut to offset decreased funding.
The problems are many, though. First, federal funding went down after a peak, so this money pool dried up. Second, this system sort of led to the pyramid scheme you mention, where grad students get recruited to maintain grants and research programs with nowhere to go, lest universities and faculty lose indirect funding. Third, it prioritizes research based on grant receipts rather than utility, which distorts research priorities in a perverse way (especially given nepotism in grant culture, which has been written about extensively elsewhere). Finally, because of the history of all of this, you kind of end up with a system in which those running the system are either beneficiaries of the 90s and 2000s, and don't realize what it is like now, or came up through that system and have a kind of survivorship bias.
Agencies like NSF and NIH are basically just maintaining any semblance of research in the US at the moment, making up for all the deficiencies that are occurring at the state level.
There are problems with NSF and NIH, but they're not with advocating for STEM grads. They're with how grants are funded. Basically, review panels need to randomly rotate in some kind of jury pool system almost; grants need to be awarded based on some kind of lottery system (proposed by former heads of NIH); indirect cost charges need to be eliminated or made line-item justified; some grants need to be awarded on the basis of publication record without regard to a specific proposal (ala Hungary); universities need to be adequately funded outside of the grant system; and tenure needs to be protected (I say this as someone who is probably going to step down from my tenured position soon).
The problems facing grad students and postdocs are just the tip of the iceberg. They don't end, even when you get a tenure-track position, and even when you get tenure. The system is broken, it's a mess that favors hype and popularity over rigor. What you end up with is the current reproducibility crisis, academic fraud, people manipulating the system to overclaim credit, and many more problems that society is unaware of because they have a stereotyped idea of what happens in science--the sort of "great man in the lab with a eureka moment" which is a false model of scientific process.