I think the whole "you need too many subjects and too much money" objection to doing science is very weak. There are other fields of research where they find inventive ways of experimenting on a shoestring budget (I think some social sciences belong in this category.)
You can even do proper causal analysis when all you have is an observational study, given that you can get an expert to produce a decent guess at the causal network. (And this guess can to some extent be verified by plain correlations.)[1]
Sure, you might not get the extremely high external validity of e.g. particle physics, but in many cases something is better than nothing.[2]
Heck, screw external validity. I'd be ecstatic to see more experiments just within an organisation, focusing on internal validity. Every engineering organisation can afford to experiment with different tools and practises. There are huge productivity gains to be unlocked (is my hypothesis, anyway), and the cost is small.
If we get together enough small studies lacking external validity, but which point in the same direction, we might even be able to infer some external validity between them.
[1]: For more on this approach, see Judea Pearl.
[2]: To use an example close to my heart: pregnancy is a condition where you rarely see randomised trials, and any research is sorely underfunded. Yet when people actually take the time to sit down and answer questions with data, we get something meaningful out of it.
I wouldn't be surprised if a few organizations had run major experiments over the years, with results never seeing the light of day.
It would also be great if there was more awareness of the kind of data the research community needs so that if someone saw a natural experiment shaping up in their company they knew who to call.
Regarding budgets, these researchers are a lot smarter than me and I bet they are doing their best with what they have. What might be good though is having several labs join forces and collaborate on one very good trial of a core issue.
Then you have a hope of settling a question and moving the field forward while creating some infrastructure that will help answer other big questions.
Maybe that's happening, I haven't seen a lot of it though.
It's worth pointing out that this blog post was published three years before Accelerate, by Forsgren, Humble, and Kim. That book demonstrates what you can do just with surveys. It's not about software engineering per se - it's got a devops focus - but the overlap is significant.
As far as I'm concerned that book significantly changed the game, not only because of the findings, but because it showed that hard results are possible in this arena. The downside is that you need enough data to make the maths work, and that precludes applying the techniques to internal studies in most organisations. There's just too much noise.
You can even do proper causal analysis when all you have is an observational study, given that you can get an expert to produce a decent guess at the causal network. (And this guess can to some extent be verified by plain correlations.)[1]
Sure, you might not get the extremely high external validity of e.g. particle physics, but in many cases something is better than nothing.[2]
Heck, screw external validity. I'd be ecstatic to see more experiments just within an organisation, focusing on internal validity. Every engineering organisation can afford to experiment with different tools and practises. There are huge productivity gains to be unlocked (is my hypothesis, anyway), and the cost is small.
If we get together enough small studies lacking external validity, but which point in the same direction, we might even be able to infer some external validity between them.
[1]: For more on this approach, see Judea Pearl.
[2]: To use an example close to my heart: pregnancy is a condition where you rarely see randomised trials, and any research is sorely underfunded. Yet when people actually take the time to sit down and answer questions with data, we get something meaningful out of it.