There is no way to scale this with tech, that's why it doesn't scale :P
The core problem with multiple choice and other trivial tests is that a person understanding the concept will find the correct answer, but people not understanding the concept (and just memorizing, guessing, cheating, ...) will also be able to find the correct answer. You have many false positives (and depending on test quality also false negatives).
The only real solution is a combination of complex problem solving settings and individual evaluation by a teacher or similar role. You have to find out what is going on inside the person's brain, why they chose this or that answer - essentially by talking to the person.
You'd need AGI to tech that.
A step in the right direction would be to abandon single-number scores and introduce more differentiated numbers for different properties of learning, knowledge and so on.
Even better would be assessment of individual learning progress instead of objective result, though this is a hard sell as long as there is a labor market. Employers will want to have an easy metric for comparison of different candidates. Objective scores produce perverse incentives: Say I'm bad at math (school grade E level) , but I am motivated to improve. I work my ass off trying to learn after school etc., and after 3 months I'll write another test. Now I get a D. Objectively still a bad result, almost guaranteed to demotivate me after months of hard work, despite the relative progress being substantial. A grade reflecting the relative progress would reward effort, which usually is what educational settings want to foster.
I appreciate your sentiment and agree that formative assessment can be a lot better. But I would argue that there must be something in between the current situation and AGI.
I've been trying to find literature about this, but am not sure where to look.
The core problem with multiple choice and other trivial tests is that a person understanding the concept will find the correct answer, but people not understanding the concept (and just memorizing, guessing, cheating, ...) will also be able to find the correct answer. You have many false positives (and depending on test quality also false negatives).
The only real solution is a combination of complex problem solving settings and individual evaluation by a teacher or similar role. You have to find out what is going on inside the person's brain, why they chose this or that answer - essentially by talking to the person.
You'd need AGI to tech that.
A step in the right direction would be to abandon single-number scores and introduce more differentiated numbers for different properties of learning, knowledge and so on.
Even better would be assessment of individual learning progress instead of objective result, though this is a hard sell as long as there is a labor market. Employers will want to have an easy metric for comparison of different candidates. Objective scores produce perverse incentives: Say I'm bad at math (school grade E level) , but I am motivated to improve. I work my ass off trying to learn after school etc., and after 3 months I'll write another test. Now I get a D. Objectively still a bad result, almost guaranteed to demotivate me after months of hard work, despite the relative progress being substantial. A grade reflecting the relative progress would reward effort, which usually is what educational settings want to foster.