Tokens and the context window
Lesson 3 of 23 in the course “AI basics”. Free, with animations and a quiz.
How text gets cut into pieces
Before the model reads your message, a tokenizer cuts it up: a program that splits text into tokens. A token is a piece of text: a whole word or part of one, sometimes a space or a punctuation mark. Each token gets its own number (id) from the model's vocabulary. The model doesn't see letters, it sees a chain of those numbers. The counter in the picture shows how many tokens the phrase turned into.
One word, several tokens
Common words usually fit in one token, while rare and long ones get cut into parts: “unbelievably” can easily become three pieces. The same meaning in Russian, Arabic and many other languages often takes more tokens than in English, because model vocabularies were built mostly from English text. Bonus: a model struggles to count the letters in a word, since it never sees the letters.
The context window
The context window is how much text the model can see at once. It's measured in tokens. Everything goes into it together: your messages, its replies, pasted files and the chat's hidden settings. Whatever doesn't fit simply doesn't exist for the model. It can't scroll back like you can: it only sees what's inside the window right now, like the bright messages in the picture.
A long chat loses its start
Imagine a week-long chat about prepping for an exam. Once the text grows bigger than the window, the oldest messages drop out or get compressed, like the faded messages outside the frame in the picture. Even inside the window, the model sometimes misses details buried in the middle of a long text. What to do: ask for a short summary, start a new chat, paste the summary there and repeat key conditions.
Why limits are counted in tokens
Every token, read or written, costs the model time and computing power. That's why message limits, answer length and prices in AI services are counted in tokens. Rule of thumb: in English one token is about 3/4 of a word on average; in other languages you usually get fewer words per token. Bottom line: fluff in your prompt eats both the window and your limits and gives nothing back.
Check yourself
- What is the context window?
Correct answer: How many tokens the model sees at once. The context window is the amount of text, in tokens, the model sees at once. Anything outside it doesn't exist for the model. - Why does the same meaning often take more tokens in Russian than in English?
Correct answer: The token vocabulary was built mostly from English. Common English chunks more often get their own vocabulary entry, so words in other languages are cut into more pieces. - A chat has run for a week and the model forgot a condition from the first message. What's sensible?
Correct answer: Get a summary and continue in a new chat. The old messages fell out of the window, and asking it to “remember” won't bring them back. A summary in a new chat puts the key facts back inside the window.
← How a neural network thinks Embeddings: meaning as coordinates →