Testing, Education and AI
Tyler Cowen asks How AI will change student evaluation?
Maybe we should ask, what really is the point of evaluation in the first place? Hopefully someone setting standards will eventually do that, rather than just mindlessly continuing with tradition. If tests came to mean what Cowen predicts, “a kind of conscientiousness”, wouldn’t it be a mistake to continue using them in the same way we do today?
Test scores and overall evaluations shouldn't have any long term value even today. Every individual should be building upon themselves and to think the only place you could have learned a topic is prior to a test is at the very least archaic, and also somewhat arrogant of educators to assume. Educators provide an opportunity to learn in a world full of opportunities. They don't have a monopoly on it and never have.
I think the impact of AI on testing techniques will make that even more obvious than it is, which hopefully will cause more to evaluate, what do they continue to mean, what purpose should they be used for, and then we can think on how to achieve that purpose by a different structure.
In my mind, testing shouldn't be a signal to outsiders, it should be a service to students. If you can remove the first part, it makes the second easier to achieve, since the student now has less motivation to cheat. Tests should help a student and an education system know when they are ready to progress, i.e. they have the mastery of ideas and concepts that has typically accelerated other students ability to master a further set of ideas and concepts.
The value of an education system is not as a monopoly on learning, but as a optimization of learning based on it's experiences in educating other students. I think a great deal of our education systems have lost sight of that, and it will do us a favor to force them to confront it again, and realize that the Internet, Wikipedia, books and now AI exist with or without them, and that they aren't and shouldn't be looked at as repositories of information that are the only path to learning, but as a service to accelerate learning.
If they don't understand this, they are unlikely to realize that the next frontier here is understanding the differences among individuals. If that isn't confronted, I suspect the entire education system will disappear and be replaced by AI learning software that will actively monitor and understand the differences among individuals and what helps them get to mastery of topics the quickest.
My main prediction though is that despite a lot of conflict and resistance, the education system will make that realization and support that goal in symbiosis with the capabilities that AI learning software could provide.
In terms of evaluations, here’s another idea, they will be like oral exams, but AI delivered. Instead of trying to use an AI to filter out cheating on a fixed test, that goes into a permanent history, engage AI to ask questions of students, and allow the students to ask questions back. The test here isn’t knowing the answer directly, but knowing the right questions to ask to get to the answer. Any educator has experience with this when performing 1 on 1 tutoring.
What is the educators role here? Appeals is one. If a student feels they are stuck unfairly an educator can override, which will also function as a RLHF for the AI trainer. But every attempt of a student to appeal serves two purposes. If it is the case that the AI trainer isn’t evaluating them well, yes, that’s one, but realistically a student might not know this and be overconfident in whether they do. The real outcome might be that they need personal attention, and their appeal serves as an opportunity to engage it, even if not explicitly stated as such.
