How a chatbot is trained: 3 stages
Lesson 7 of 23 in the course “AI basics”. Free, with animations and a quiz.
Three stages to a chatbot
A chatbot isn't born a polite assistant. It's built in three stages. First, pretraining: the model reads mountains of text. Second, fine-tuning on examples: it's taught to carry out requests. Third, human ratings: it's taught to answer the way people like. Sounds scary, but it's actually simple. Let's go through each stage in turn.
Stage 1: pretraining
Pretraining: the model reads a huge pile of text (books, websites, code) and learns to guess the next token, meaning the next piece of text. It's the longest and most expensive stage. The result is a base model: it continues text brilliantly but can't answer questions. For example, given “How do I prepare for an exam?” it might add five more questions, like a list on a forum.
Stage 2: learning from examples
Next, the model is fine-tuned, meaning it keeps learning on a smaller but carefully made set. These are pairs of “request → good answer” written or picked by people. For example: “Make me a one-month exam study plan” next to a solid week-by-week plan. After seeing lots of these pairs, the model gets it: a question needs an answer, and a task needs doing.
Stage 3: human ratings
Then people are shown two answers from the model to the same question, and they pick the better one: a like or a dislike. From lots of these choices, the model learns which answers people like: clear, polite, safe. This stage is called RLHF, reinforcement learning from human feedback. Put simply, the model gets a “plus” for the answers people would choose.
Why a bot can be a people-pleaser
People tend to reward polite, agreeable answers. So the model learns to please. Here's the downside: it can be a people-pleaser, meaning it agrees too easily. For example, you confidently argue that the Great Wall of China is visible from space with the naked eye. The bot may give in, even though that's a myth. It's not a bug, it's a side effect of training.
How not to fall for it
Don't hint at the answer you want to hear. Instead of “my essay is great, right?” ask “find three weak spots in my essay”. Ask for honesty directly: “don't agree if I'm wrong”. If the bot suddenly flips after you say “you're wrong”, ask it to explain why and check the fact yourself. In short: a bot agreeing with you isn't proof.
Check yourself
- What does the model do during pretraining?
Correct answer: Learns to guess the next token from huge amounts of text. Pretraining is reading mountains of text and guessing the next piece. Good-answer pairs and human choices come in the later stages. - Why are people shown two model answers and asked to pick the better one?
Correct answer: So the model learns to answer the way people like. That's the third stage, RLHF: from people's choices, the model learns which answers they like. - You told the bot “you're wrong” and it agreed right away. What's the smart move?
Correct answer: Ask it to explain and check the fact yourself. After training on human ratings, bots tend to agree. Neither the first nor the second answer is proof: you need to check.