Why the answers differ every time
Lesson 5 of 23 in the course “AI basics”. Free, with animations and a quiz.
Every word gets a chance
The model doesn't pick one word right away. First it gives each option a probability, meaning a chance. Example from the picture, the phrase “The sky today is…”: “blue” 62%, “grey” 21%, “clear” 9%, “purple” 2%. The remaining 6% is shared by all other words: 100 − 62 − 21 − 9 − 2 = 6. The numbers are made up for the example, but the mechanism is real.
The model rolls the dice
Next comes sampling: the model draws a word at random, but weighted by the chances. Out of 100 runs, “blue” comes up about 62 times and “grey” about 21 times, because those are their chances. That's why the same question gets different answers: one different word early on and the whole text goes its own way. It's not a bug, it's a feature.
Temperature: the risk slider
Temperature is a setting that changes those chances. Low temperature makes the tall bars even taller: the model almost always picks “blue”, and answers are predictable. High temperature flattens the bars: even “purple” gets a shot, and the text becomes bolder and more random. A similar setting is top-p: the model picks only from the most likely words until their combined chance reaches a set share.
When to go cold, when to go hot
Low temperature suits tasks with one right answer: working out a trip budget, formatting a table, translating instructions. High temperature suits ideas: channel names, a video plot, a gift for a friend. Regular chat apps often have no slider. Then ask for “5 very different options” or hit the regenerate button: the model draws its words all over again.
One answer isn't the truth
Since an answer is a random draw weighted by chances, a single answer isn't “the” answer. Regenerate it 2–3 times. If the facts differ between answers, the model isn't sure: check with a reliable source. The trap is that even at low temperature the model can confidently repeat the same mistake every time. Predictable doesn't mean correct.
Check yourself
- Why does the model answer the same question differently?
Correct answer: It picks words at random, weighted by chances. That's sampling: words are drawn at random but by probability. A different word early on changes the whole rest of the text. - Which temperature is better for coming up with a channel name?
Correct answer: Higher: you'll get more varied options. Ideas need variety. High temperature evens out the chances, so less obvious words make it into the answer too. - You regenerated an answer 3 times and the year of an event is different each time. What does that mean?
Correct answer: The model isn't sure, so check the year. Varying facts mean the model has no confident answer. Neither the latest nor the most frequent option guarantees the truth: check a source.
← Embeddings: meaning as coordinates Data decides everything →