AI grading AI
Lesson 22 of 23 in the course “AI basics”. Free, with animations and a quiz.
AI hands out grades
An AI judge is a model that grades the answers of another model, or your own writing. Example: you wrote two versions of a cover letter, a short note about yourself for an employer, for a summer job at a café. You can ask the judge for three things: give a score from 1 to 5, pick the better of two (A or B), or compare the text with a model answer. Companies check thousands of answers this way because it is faster and cheaper than paying people.
The judge's three weak spots
The judge is not neutral. It has biases. First, position bias: it more often picks whichever answer comes first. Second, length bias: a longer answer looks better to it, even when it contains mistakes. Third, self bias: a model rates text written in its own style higher. The picture sums it up: the judge likes the first and the long answer. There is no bad intent here, it is a side effect of how the model learned.
Swap A and B
A simple trick against position bias: ask twice. First put letter A first, then letter B, ideally in a new chat. Did the judge pick the same letter both times? Then you can trust that verdict more. Did the choice flip when the order flipped? Then the judge doesn't really see a difference, so count it as a tie. Against length bias, say outright “length doesn't matter” or compare texts of similar size.
Judge = model + prompt
The grade depends not only on the text but on who judges and how. Change the model or a couple of words in the prompt and you have a different judge. Even the same judge can give the same text a different grade, because AI answers are partly random. Example: a “7 out of 10” in one chat and a “9 out of 10” in another can't be compared. For steadier grades, give the judge the task, the criteria (what to judge by) and a 1–5 scale that describes each grade.
A ranking built from duels
Models themselves are compared the same way. People see two answers to one question, often without knowing which model wrote which, and pick the better one. Each pick is like a match. From lots of matches you get a rating, like in chess or online games: beating a strong opponent earns more points. The catch: voting works for taste and style. Whether a fact is true isn't decided by votes, it's decided by checking.
You make the final call
How to use a judge for yourself. Ask it to review your essay against a few clear criteria, for example: a clear main point, examples, grammar. Three or four sharp points beat fifteen vague ones. Ask it to explain first and give the grade after. Comparing two versions? Run them in both orders. Bottom line: an AI judge is a second opinion, not a teacher with a grade book. Check facts yourself, and the decision is yours.
Check yourself
- The judge picked letter A. You swapped the order and it picked B. What does that mean?
Correct answer: The judge sees no real difference, so count it as a tie. If the choice flips with the order, the judge is reacting to position, not to quality. It's fairer to call it a tie. - Why can't you compare a “7 out of 10” from one chat with a “9 out of 10” from another?
Correct answer: A different prompt or model makes a different judge. A judge is a model plus a prompt. Change either and the scale changes, and even the same judge can vary from run to run. - Which question is voting on “which answer is better” worst for?
Correct answer: What year a historical event happened. Votes measure taste and style well. A date is either right or wrong, and a source settles that, not the majority.