Sound and speech
Lesson 7 of 22 in the course “AI Video, No Flops”. Free, with animations and a quiz.
Four layers of sound
Sound is half of realism. A silent AI video feels dead, even if the picture is perfect. In film, sound is built from four layers: speech, ambience (street hum, wind, room tone), effects and foley (steps, fabric, impacts) and music. These used to be made separately; now Veo 3.1, Kling 3.0 and Seedance 2.0 generate sound together with the picture. That’s handy, but sound has to be directed in words too, or the model adds something random. In this lesson we’ll go through each layer and decide what to generate and what to add in the edit.
Foley: sound that has weight
Foley is named after Jack Foley, a Universal sound artist who back in the early days of sound film began performing footsteps and clothing rustle in sync with the picture. The secret is sync: the sound happens exactly at the moment of contact. A foot hits concrete: squeak. An arm goes up: the hoodie rustles. The brain glues sound and motion together, and the body on screen instantly feels heavier and more real. So in the prompt, write the sound together with the action and the material: not “footsteps” but “sneakers squeak on wet concrete”. The material tells the model what the sound should be like.
A line in quotes
Models with sound understand a simple format: who speaks, in what tone, and the line itself in quotes. For example: Mira says: “Drop’s live. Grab it before I do.” The main rule is length. People speak about 2–3 words per second, so an 8-second shot fits up to 20 words, and 10–15 is comfortable. A long line either gets cut off or the model starts gabbling. One line and one speaker per shot; build a dialogue from several shots. If the model draws subtitles right into the frame, add the word “subtitles” to the negative prompt.
Music goes in the edit
A model can compose music for a shot, but for an ad that’s a trap. Every shot gets its own melody, and the music breaks at every cut. A generated track can’t be cut to the beat properly, and its rights are often unclear. Pros do this: in the prompt they describe only ambience and effects, like “Audio: street ambience, footsteps, distant traffic”, and lay one track on top in the edit. For brands, TikTok has the Commercial Music Library: music you can legally use in ads. Other options are your own track or licensed stock music.
Lip-sync and translation
If you need an exact line or your own voice, there’s a more reliable path than generating speech: record the voice (or synthesise it) and put it on the video with lip-sync, so the lips adjust to the sound. The same trick makes translations: Higgsfield Translate translates a video into 18 languages and moves the lips to the new text. That’s how one of Mira’s videos becomes versions for Brazil, Mexico and Turkey. Two rules. First: a native speaker checks the translation, because machine slang easily sounds cringe. Second: clone someone else’s voice only with the owner’s written consent. Your own voice is fine.
Sound at the end of the prompt
When light, camera and sounds are mixed in one paragraph, the model gets confused: a noise can turn into an image and an image into a sound. A reliable order: first the shot and action, then the camera, then light and colour, and as the last line, the sound marked “Audio:”. The official Veo prompt examples do the same: sound gets its own part. The order helps you too: the audio line is easy to find and change without touching the picture. Look at Mira’s prompt in the diagram: four blocks, each about its own thing.
Captions for people watching muted
Most of the feed gets scrolled on the subway, in class and at night, which means muted. Back in 2016 Digiday reported that up to 85% of Facebook videos were watched with the sound off. If the video’s meaning rests on a line, nobody gets it without captions. Make captions in the edit, not by generation: the model will draw scribbles. Make them big, 1–2 lines, in the middle of the screen. In TikTok and Reels the bottom and right edge are covered by the caption, buttons and the music name, so text gets lost there.
Check yourself
- Mira put a 40-word line into an 8-second shot. The model gabbles and cuts the line off. What should she do?
Correct answer: Cut it to 10–15 words or split it into two shots. People speak 2–3 words per second, so 10–15 words fit comfortably into 8 seconds. The rest goes into the next shot. - In a five-shot ad, the model generated different music in each shot. The melody breaks at every cut. What’s the right move?
Correct answer: Keep only ambience and effects in the prompts and lay one licensed track in the edit. One track on top ties the video together and lets you cut on the beat, and a licence settles the rights question. - Mira wants an English version voiced by her friend Lena. Lena is on holiday. What should she do?
Correct answer: Message Lena and get written consent; until then use her own voice or TTS. A voice is part of a person’s identity. Cloning it for an ad requires the owner’s explicit consent, ideally in writing. - The captions in Mira’s video sit at the bottom of the screen, and in Reels the post caption covers them. What should change?
Correct answer: Move big captions to the middle of the screen, away from the right edge. The middle of the screen is free of interface, and big text reads even on the go.