Images, voice, video
Lesson 16 of 23 in the course “AI basics”. Free, with animations and a quiz.
AI that sees and hears
Many models work with more than text. They're called multimodal: a “modality” is a type of data, like text, image or sound. The diagram shows four directions: image to text, text to image, voice to text, text to voice. Example: you're traveling, you snap a photo of a menu in a language you don't know and ask which dishes have no meat. For a model, video is usually lots of image frames plus sound.
How the model “looks” at a photo
Sounds like magic, but it's simple. The image is cut into small squares (called patches), each square is turned into a set of numbers, and the model handles them just like tokens, meaning pieces of text. That's why it can describe a photo, read a sign or explain a chart from a slide. In the menu example, the model sees the text in the photo, translates it and answers your question.
Text to image, voice both ways
An image generator doesn't look up a ready photo. It draws a new one from your description. Example: “a party poster in pixel-game style”. A voice assistant often works as a chain: speech recognition turns your voice into text, the model writes a reply, and speech synthesis reads it out loud. Some models handle sound directly, but the idea is the same: sound into numbers, numbers into an answer.
It doesn't see like a human
The model guesses what looks plausible instead of checking, so its mistakes are predictable. Typical generator fails: letters in the image turn into mush, people get extra or twisted fingers. When reading a photo: it miscounts objects (how many people are in the shot), mixes up tiny or blurry text, and confidently describes things that aren't there. Always check anything important in an image with your own eyes.
Deepfakes: how not to get fooled
A deepfake is a photo, video or voice made by AI to look or sound like a real person. A voice can be faked from clips posted on social media. Example: a voice message from your “brother”: “send me money right now, I'll explain later”. Rule: urgency plus money means stop. Call back on a number you already know, and ask something only he would know. For photos, find the original source with a reverse image search.
Check yourself
- How does a multimodal model take in a photo?
Correct answer: Cuts it into squares and turns them into numbers, like tokens. The image is split into patches, and each one becomes a set of numbers. Then the model works with them just like pieces of text. - Where do image models make mistakes most often?
Correct answer: Letters in images, fingers, counting objects. The model draws and describes what looks plausible and doesn't check details. That's why text, fingers and exact counts are its weak spots. - A voice message from your “brother”: send money now. The voice sounds right. What do you do?
Correct answer: Call him back on a number you know and check. A voice can be faked, and so can a second voice message. Checking through another channel, like calling a number you know, breaks the scam.