Ask three times, pick the best
Lesson 23 of 23 in the course “AI basics”. Free, with animations and a quiz.
One question, different answers
Remember temperature? The model draws words by their chances, so the same question can get different answers. This is called inconsistency. It can be worse: slightly different wording, say a different word order, and the answer changes a lot. For ideas that's a plus. But when you need one exact result, like the answer to tomorrow's algebra homework, it's a problem.
Spend more effort on the answer
You can get a better answer without a bigger model. Spend more computing at the moment of answering: get several versions instead of one. This is called test-time compute. The simplest method is best of N: make N attempts and keep the best. You can do the picking, or a simple check, or an AI judge. The downside: 3 attempts cost about 3 times the time and money.
Voting: where answers agree
For tasks with one exact answer there's an even simpler trick: majority vote. In the picture: one problem, three attempts, answers 42, 42 and 38 (example numbers). Two of the three match, so we take 42. The logic: random mistakes are usually different each time, while a correct line of reasoning tends to land on the same number. Good for math, multiple choice tests, pulling a number out of a text. An essay has nothing to vote on, so there you pick the best.
“Thinking” mode does something similar
The reasoning models from the lesson on steps also spend more effort before answering. They write a draft, try different ways to solve the problem, check themselves, and only then reply. It's the same idea: more work at answer time instead of a bigger model. That's why they're slower and cost more. Turn that mode on for a hard problem or a plan, not for “how do you say thank you in Japanese”.
Where made-up answers come from
Why does AI hallucinate, meaning confidently write things that aren't true? The model continues text in a plausible way and can't always tell what it knows from what it guessed. And everything it has already written, it reads as fact. Example: in the first sentence the model names the wrong author of a book, then invents that person's life story so it all hangs together. This is called snowballing: one wrong guess drags more after it.
Agreement isn't proof
A recipe for important numbers and facts. Ask 2–3 times: regenerate the answer or open a new chat, where an old mistake in the history can't drag a new one along. Answers disagree? That's a red flag: the model isn't sure, so check a reliable source. Answers match? The chance of a mistake is lower but not zero: a model can repeat the same popular mistake three times. Bottom line: agreement is a hint, a source is the proof.
Check yourself
- Three attempts gave 42, 42 and 38. What does a majority vote pick?
Correct answer: 42, because two of the three attempts said so. A majority vote takes the answer most attempts agree on: 2 of 3 gave 42. Averaging makes no sense when there's one right answer. - Why is a new chat better for double-checking a fact?
Correct answer: An old mistake in the history won't sway the new answer. The model reads everything already in the chat as fact. A new chat doesn't contain its earlier mistake, so the check is fairer. - The model gave the same answer to an important question three times in a row. What's true?
Correct answer: A mistake is less likely, but still check the fact. Agreement raises the chance the answer is right, but a model can steadily repeat a popular mistake. A source is the proof.