Embeddings: meaning as coordinates
Lesson 4 of 23 in the course “AI basics”. Free, with animations and a quiz.
A word turns into numbers
Computers don't understand words, they work with numbers. So the model turns each word or phrase into an embedding: a long list of numbers, usually hundreds or thousands of them. Think of it as the coordinates of a point on a map, except with hundreds of axes instead of two. The model picks these numbers during training by looking at which words a word tends to appear next to in texts.
Close meaning, close points
In the picture, this map is squeezed down to two axes. “Cat”, “dog” and “puppy” sit close together because people write about them in similar texts. “Banana” and “apple” form their own cluster, “bitcoin” and “dollar” another. The arrow shows the distance between points: the shorter it is, the closer the meaning. The model doesn't know what a cat is, but it knows what a cat is similar to.
Search by meaning
Regular search looks for matching words. Embedding search looks for nearby points. You type “what to do on a rainy day in London” and it finds an article called “museums and cafés for bad weather”, even though the words barely match. Recommendations often work in a similar way: your feed gets videos whose points sit close to the ones you watched to the end.
King − man + woman
A famous trick: take the point for “king”, subtract “man” and add “woman”, and you land near “queen”. So some directions on the map store relationships between words, like “gender” or “country and its capital”. But don't get carried away: it's a neat illustration that doesn't work for every word and isn't always exact. There's no human-style understanding of the world here.
Attention: a word looks at its neighbors
One word can have several meanings. “Bank” in “we sat on the river bank” and in “I opened a bank account” are different things. The attention mechanism handles this: while reading each word, the model looks at the other words in the sentence and refines its meaning. “River” pulls “bank” toward water. With that refined meaning, the model then guesses the next word, like in the picture.
Check yourself
- What is an embedding?
Correct answer: A list of numbers that describes meaning. An embedding is the coordinates of a word or phrase on a map of meaning. Texts with similar meaning get nearby coordinates. - Why can search by meaning find an article that doesn't contain your words?
Correct answer: It looks for points close in meaning. The query and the articles become embeddings, and search picks the nearest points. Matching words aren't required. - How does the model tell that “bank” in “sat on the river bank” isn't about money?
Correct answer: It looks at the other words in the sentence. That's the attention mechanism: neighboring words refine the meaning of each word, so the sense is chosen from context.
← Tokens and the context window Why the answers differ every time →