How a neural network thinks
Lesson 2 of 23 in the course “AI basics”. Free, with animations and a quiz.
It guesses the next word
A neural network is a program that learned from examples instead of following hand-written rules. ChatGPT and Claude are language models, meaning neural networks built for text. They write an answer piece by piece, each time picking the best-fitting continuation. For example, after “Bitcoin is often called digital…” the model is far more likely to pick “gold” than “pizza”. Then the next word, and so on.
A token is a piece of text
The model doesn't work with letters, and not always with whole words. It works with tokens. A token is a piece of text: a short word, part of a longer word or a punctuation mark. A long word is often split into several tokens. The trap is the name: this is not a crypto token and not money, just a unit of text. Paid AI services count chat length and price in tokens.
It learned from mountains of text
Before launch the model went through training: it read a huge amount of books, websites and code. It does not store whole pages. It learned patterns: how people usually continue a thought, how emails are written, how code is put together. That's why its writing is smooth. But it is not a library holding exact quotes, so it can get the wording of a source wrong.
Knowledge with a date
Training ends at some point, and after that the model learns nothing new on its own. That line is called the knowledge cutoff date. About anything later the model may know nothing or make up an answer. Some chatbots can search the web, but they don't always do it and don't always find the right thing. Bottom line: check fresh news, exchange rates and prices yourself.
Not a search engine, not a person
The model builds a plausible answer. It doesn't pull it from a database and doesn't check a source by itself. So it can sound confident and still be wrong. Example: ask how many moons Mars has, and a confident answer may turn out wrong. The right answer is two, Phobos and Deimos. Rule one: confident does not mean correct. Always check what matters.
Check yourself
- What does the model do while writing an answer?
Correct answer: Picks the best-fitting next piece of text. The answer is built piece by piece, token by token. The model has no ready answers, only learned patterns. - What is a token in a language model?
Correct answer: A word or a piece of a word. Here a token is a unit of text: a word, part of a word or a symbol. Don't mix it up with crypto tokens. - The answer sounds very confident. What does that tell you?
Correct answer: You should still check it. A confident tone says nothing about accuracy. The model picks what sounds plausible, so check what matters.
← AI, ML, neural nets: what's what Tokens and the context window →