Delivering a quality product within time and budget constraints, consistently. Oh, but that's a team effort, you say? Well, that's the working environment, isn't it? Projects are an awfully coarse measurement, you say? Well, that's the only deliverable that matters.
I recall reading years ago (in Peopleware or one of its contemporaries) about a company's evaluation of one of its coders. She was definitely mediocre by every measure they had. But someone noticed that every project she was on succeeded, over many projects and many years. Though she wasn't a monster at the keyboard, something she brought to the team engendered success. How productive was she? Would you want to hire her and have her on your team?
You must measure what you actually care about. Measuring things that you think are factors is fine and noble, but if you're not measuring the actual "product" of "productivity" then you'll never know how well your factors correlate with the real goal.
But someone noticed that every project she was on succeeded, over many projects and many years. Though she wasn't a monster at the keyboard, something she brought to the team engendered success.
Correlation does not equal causation. Another possible explanation: She was a monster at predicting project success and worked her way onto projects that were going to succeed with or without her.
Sure, but if causation is effectively impossible to rigorously determine, it can end up being all you've got. If you've got the choice between going with someone whose presence correlates with success on the project and one who does not, my inability to be rigorously sure about causation isn't going to make me lose much sleep at night when I chose the correlative one.
I'm coming to dislike the citing of "correlation does not equal causation" when there's no way to determine causation at all, and when scientific certainty isn't the question at hand. At that point it's an excessively-powerful criticism, one that can't be discharged, so is it really a useful criticism at all if so?
I'm coming to dislike the citing of "correlation does not equal causation" when there's no way to determine causation at all
This is perfectly understandable, however this particular discussion is one where the difference between correlation and causation is appropriate. We are talking specifically about paying programmers by their "productivity." If you want to say that "productivity" is defined as the correlation between a person and project success regardless of whether there is a causal relationship or not, and regardless of whether they engage in programming activities, project management activities, picking good project activities, discussion activities, or even just making everyone else espresso so they can produce working code, that's fine.
But what we're saying in that case is that we can't measure the productivity of a programmer, we can't establish a relationship between programming activities on the scale of a single person. I agree that the correlations you can observe are perfectly useful for management and that one can deliver great (or working, or valuable) software without an objective metric of programmer productivity. I agree that this elusive metric may not be necessary. It may not even be useful, as I tried to demonstrate elsewhere when I discussed Ned, Fred, Ed, and Jed.
But that really underscores my point: We can't tie compensation to programmer productivity because we can't measure it. Your point seems to be that we don't have to tie compensation to programmer productivity, that we can tie it to correlation with project success, for example.
Fine with me, I'd say we're in violent agreement and that our stances are compatible.
Sure, in general. In this case managers were fairly familiar with her and the teams and posited that the success was due to what she added to team discussions. I think they were probably correct.
OTOH, having a project success divining rod could have its own value.
Even if she was just a monster at predicting project success, I'd still want her on my team. Can you imagine how useful it would be to have someone able to consistently predict project success working with you?
(Interestingly, there's a less flattering interpretation of the data as well. Given a large enough organization that promiscuously shuffles people onto new projects, some of which randomly succeed, someone is going to have randomly ended up on all the successful projects. We'd then look at them and say "Look how awesome they are!", when really they just got lucky over and over again and we're looking because they got lucky. This requires that the organization size be large relative to the number of possible combinations of projects, though, which becomes increasingly unlikely over time. It's like the people who look at Warren Buffett and proclaim "he just got lucky for 40 years in a row", then do out the math and realize that the chance of someone being that consistently lucky is several million to one.)
So ask her if she's willing to be on this project! Seriously though, ask your developer why there is a correlation between them and project success - if they can tell you, even if it shows they should be a business analyst, you're on to something very useful and valuable.
"Correlation does not equal causation. Another possible explanation: She was a monster at predicting project success and worked her way onto projects that were going to succeed with or without her."
Fair point, but what would you pay for knowing in advance what projects are going to succeed and by inference which may fail...
And how would you measure Intuition then? Maybe she had terrible intuition but is just a fantastically productive programmer.
Correlation not implying causation is a big deal because it's possible to draw (probably exponentially) many alternative causal chains than the one that you're discovering correlation along.
If the above isn't the case, and it's at least theoretically possible to design experiments like that, then correlation does[1] equal causation.
[1] Sort of. See pretty much anything written by Judea Pearl.
"I recall reading years ago (in Peopleware or one of its contemporaries) about a company's evaluation of one of its coders. She was definitely mediocre by every measure they had. But someone noticed that every project she was on succeeded, over many projects and many years."
She was the Shane Battier of software.
(I happen to be reading a book on basketball right now.)
Interesting you should mention it. Shane Battier gets traded to Memphis, and for the first time in their history, they have a playoff win and are well within the sight of a series win over the top ranked Spurs.
I recall reading years ago (in Peopleware or one of its contemporaries) about a company's evaluation of one of its coders. She was definitely mediocre by every measure they had. But someone noticed that every project she was on succeeded, over many projects and many years. Though she wasn't a monster at the keyboard, something she brought to the team engendered success. How productive was she? Would you want to hire her and have her on your team?
You must measure what you actually care about. Measuring things that you think are factors is fine and noble, but if you're not measuring the actual "product" of "productivity" then you'll never know how well your factors correlate with the real goal.