Perhaps I'm naive, but it seems like a model used for decisionmaking should be one that can show predictive performance - one that can predict, based on historical data about a set of students and a specific teacher, how well a teacher would do teaching that set of students. If it can't be accurate in that, how is it possible to know that it's capturing enough of the variables? And it seems that VAMs are decidedly not such a model.
It's hard to know what the system really does because the article really doesn't explain it.
I think what you're describing is Cross Validation[1]. It would work if they are predicting performance, but it sounds like the VAM system might be trying to figure out what a hypothetical "average" teacher would have achieved with the same students and comparing that the actual teacher's performance. This is basically trying to predict how the students will do independent of the teacher, but without such a teacher there is no real way to validate the model. Perhaps if they examined students across all teachers.
The system may ultimately be more about comparing teachers to each other and not about actually determining the value provided by an individual teacher.
