How to tell if an AI is good
Lesson 9 of 23 in the course “AI basics”. Free, with animations and a quiz.
Test it on something new
A fair test of a model uses examples it never saw during training. It's like an exam, not the textbook problems you already solved. In the picture, the model scores 100% on training examples and 70% on new ones. The percentage is the share of correct answers, and the numbers are just for illustration. Trust the second number: it shows how the model will do in real life.
Overfitting: memorized, not understood
Overfitting means the model memorized its training examples so well that it gets lost on new ones. Like a classmate who crammed the answers from the workbook but can't solve the same problem with different numbers on the exam. So a big gap between the textbook score and the exam score is a warning sign. A good model picks up the general idea instead of memorizing specific answers.
Accuracy can lie
Accuracy is the share of correct answers. Sounds reliable, but there's a trap. For example, you work part-time at an online sneaker shop, and an AI sorts customer emails. Out of 100 emails, 95 are regular questions and 5 are complaints. An AI that labels everything a “question” is right 95 times out of 100, so its accuracy is 95%. But it caught zero complaints. Check how an AI handles rare but important cases.
Leaderboards aren't about you
Companies brag about benchmark results. A benchmark is a shared set of test tasks, say in math or coding, used to compare models. That's useful, but your task isn't in there. A model that aces competition math can still mix up the sizes and return rules of your shop. Spoiler: first place on a leaderboard doesn't mean best for you.
Your own mini exam
Collect 5–10 real examples from your own task; that's enough to start. For instance, real customer emails: a regular question, a complaint, one written in slang, one full of typos. For each, write down what you think the right answer is. Run them through the AI and compare. That's your exam: it shows right away where the model goes wrong, and it's more honest than any leaderboard.
Compare side by side
Want to know which prompt or which model is better? Give both options the same examples from your mini exam and put the answers side by side. Score them with a simple checklist: is it correct, is it on point, is the tone right. Change one thing at a time, or you won't know what worked. And remember: answers vary a bit every time, so run each example a couple of times.
Check yourself
- A model does great on training examples but poorly on new ones. What is this called?
Correct answer: Overfitting. Overfitting: the model memorized the training examples instead of the general idea, so it fails on new ones. - An AI has 95% accuracy but found none of the 5 complaints in 100 emails. How?
Correct answer: Complaints are rare, and it calls everything a question. It labeled the 95 regular emails correctly, which is already 95 out of 100. Accuracy hides the failure on rare but important cases. - What's the best way to choose an AI for your task?
Correct answer: Test it on 5–10 of your own real examples. Leaderboards test general tasks, not yours. Your own mini exam shows right away where the model fails on your work.