Why models are so huge
Lesson 21 of 23 in the course “AI basics”. Free, with animations and a quiz.
A parameter is one tuning knob
Inside a model there are billions of numbers. Each one is called a parameter. During training the model reads text, guesses the next word and, when it’s wrong, nudges its parameters a tiny bit, like knobs on a DJ mixer. One knob means nothing on its own, but together they store the language patterns it learned. More knobs means more room for subtle things like sarcasm or style.
Three ingredients of a big model
A model’s scale rests on three things. Parameters: how many knobs it has. Data: how much text it read, counted in tokens, those little pieces of text. Compute: how many calculations were done in training, meaning how many chips and for how long they ran. That’s thousands of powerful chips, tons of electricity and huge money. One ingredient isn’t enough: a big model trained on a tiny bit of data will be weak.
Each step gives less
If you grow all three ingredients, the model usually gets better. This pattern is called a scaling law. But the curve in the picture rises steeply first and then flattens out. It’s like leveling up in a game: the first levels come fast, and each next one needs more and more XP. Example: pushing a model from “meh” to “good” is easier than from “good” to “almost no mistakes”.
Good text is running out
Models have already read a huge part of the open internet, and new quality text written by people doesn’t appear that fast. Many sites now forbid using their text for training. On top of that, more and more text online is written by AI itself. Learning from it, a model can pick up someone else’s mistakes, like in the diagram. Takeaway for you: anything you post publicly can end up in training data.
English dominates the data
Most text on the internet is in English; other languages have noticeably less. So models are often strongest in English, while in other languages they catch fewer nuances and spend more tokens on the same meaning. Don’t panic: they work well in Spanish, Russian or Arabic too. But if you ask in another language and the answer to a hard question comes out weak, try asking in English and compare.
Why small models exist
A big model is smarter, but every answer needs more calculations. So it’s slower, and each of its answers costs more. Small models are fast, cheap and sometimes run right on your phone, even offline. For example, a translator on a trip. Remember the curve: for simple tasks a small one is often enough. Bottom line: the fast mode for small stuff, the big one for hard problems.
Check yourself
- What is a model parameter?
Correct answer: A number inside the model that gets tuned in training. Parameters are billions of numbers the model adjusts while it learns. Together they store the patterns it learned. - A model is already big. All three ingredients are increased again. What to expect?
Correct answer: It usually improves, but the gain is smaller. Scale usually helps, but the curve flattens: each next step gives less, and mistakes don’t disappear. - Why make small models if big ones are smarter?
Correct answer: They are faster, cheaper and can run on a phone. Each answer from a big model needs more calculations. A small one answers faster and cheaper, and it is often enough for simple tasks.