Transcripts to Video Prompts: LLM Workflow for Better AI Video

Most AI video failures begin before the generator opens. A creator has a transcript full of story, context, and spoken rhythm, then compresses it into a one-line prompt such as 'cinematic rooftop scene, emotional, high quality.' The generator receives mood, not direction. The missing layer is an LLM that reads the transcript, extracts visual beats, and writes structured shot prompts. That workflow does not replace creative judgment. It makes the transcript usable.

A transcript is not a shot list. It records what was said, not what the camera should see. An LLM can bridge that gap if you give it a clear job: preserve narrative intent, convert abstract language into visible evidence, and keep continuity across shots. The result is a set of prompts that a video model can interpret with fewer contradictions and fewer wasted generations.

Start With the Transcript, Not the Model

The first step is to stop treating the transcript as a prompt. A raw transcript contains filler words, interruptions, repeated ideas, and references that only make sense in context. It may say, 'I was alone on the roof for the first time in years.' That line carries emotion, backstory, and a location, but it does not specify framing, light, motion, or duration. A video model cannot infer all of that reliably.

A useful LLM pass separates three things: what the audience needs to understand, what the character is doing, and what the camera is doing. The audience might need to understand isolation and return. The character might be opening a rooftop door and pausing. The camera might start behind the character, then push slowly toward the city beyond. Those are different layers of direction.

This is why transcript-to-video work benefits from a structured intermediate format. Instead of asking an LLM to 'make prompts,' ask it to produce shot cards. A shot card contains the story beat, visible action, setting, camera plan, lighting plan, and continuity notes. The final prompt can then be written from that card in a style that suits the target model.

Use the transcript as evidence, not scripture. If the speaker says 'it felt like the whole city went quiet,' the LLM should not literally silence the city unless that is the creative choice. It might translate the line into a wide shot with muted traffic, a distant siren, and a character standing still while crowds blur past. The visual metaphor should be deliberate, not automatic.

Build a Transcript-to-Prompt Pipeline

A reliable pipeline has repeatable stages. The stages keep the LLM from jumping straight from spoken language to cinematic language, which is where most vague prompts come from.

Start by normalizing the transcript. Add speaker labels, remove accidental duplicates, and mark timecodes if you have them. Keep pauses that reveal emotion, but cut filler that adds nothing. If the transcript came from an interview, separate the subject's words from the interviewer's questions. If it came from a script read, mark scene changes and intended beats.

Next, segment the transcript into story beats. A beat is a small unit of change: a decision, a reveal, a reversal, a question, a reaction. For short video, a beat often lasts five to twelve seconds. If a beat contains two changes, split it. 'She opens the door and remembers her mother and decides to leave' is not one shot. It is at least three visual moments.

Then identify visual anchors. These are concrete details the camera can capture: a hand on a railing, a cracked phone screen, wet pavement, a specific jacket, a clock, a letter, a doorway. Visual anchors make prompts specific without making them crowded. An LLM should extract anchors from the transcript and mark which ones are essential, optional, or invented. Invented details are allowed, but they should be flagged so you can approve them.

After that, generate shot cards. Each card should describe one camera setup and one primary action. The card can include a short line of dialogue or voiceover, but it should not contain a paragraph of speech. Long speech belongs in audio post or separate voiceover work, not in a video prompt that expects visible lip sync.

Finally, compress each shot card into a model-ready prompt. This is where you decide how much cinematic language the target model needs. Some models respond well to lens and lighting detail. Others need simpler subject-action-setting phrasing. Keep the full card as documentation, and use a shorter prompt for generation.

A good pipeline also includes a review gate. Before generating ten variants, review the shot cards as a sequence. Does the story read without the transcript? Is any shot doing too much? Are character details consistent? Catching those issues in text is faster than catching them in motion.

Write a Prompt Schema the LLM Can Reuse

A schema turns prompt writing from a creative guessing game into a repeatable operation. It also makes it easier to compare generations. If a shot fails, you can see whether the problem came from the action, the camera, the lighting, or a missing continuity note.

A practical shot-card schema looks like this:

scene_id: s03
story_beat: Maya returns to the rooftop and decides not to call her sister.
subject: Maya, early 30s, dark green jacket, short curly hair, silver watch.
action: She pushes the roof door open, steps out, stops when she sees the skyline.
setting: City rooftop at dusk, wet concrete, vent pipes, distant towers.
camera: Medium wide shot, 35mm lens, slow push-in from behind.
lighting: Blue-hour ambience, warm stairwell light behind her, practical city glow.
motion: Door swings, jacket moves in wind, camera drifts forward.
continuity: Same jacket and watch as s01; skyline matches s02.
audio: Low wind, distant traffic, no dialogue.
negative: No text overlays, no extra people, no rapid camera shake.

This schema is not a rigid formula. It is a checklist that prevents omission. The story beat keeps the shot meaningful. The subject and continuity fields reduce drift. The camera and lighting fields give the model direction without turning the prompt into a poem. The negative field removes common failure modes.

Ask the LLM to fill the schema from the transcript, then ask it to critique its own shot cards. A useful critique prompt is: 'Identify any shot where the action, camera movement, and lighting describe three different moods. Rewrite only those shots.' This produces more focused revisions than a general request to improve everything.

Keep a library of approved schema blocks for recurring characters, locations, and visual styles. Reusing them is not laziness; it is continuity management. You can build a consistent look across a series while still varying framing and action. The prompt library is a useful place to study reusable patterns before writing your own.

Translate Direction into Camera, Light, and Motion

Abstract emotions are the enemy of usable prompts. 'She feels lonely' gives the model almost nothing. 'Wide shot, small figure at the edge of a lit rooftop, city lights out of focus, slow backward camera move' gives the model a visual argument for loneliness. The LLM's job is to propose that argument, not to label the emotion and stop.

Camera language should be specific but not overloaded. Pick one shot size, one angle, one movement, and one lens idea where it matters. A medium close-up with a slow push-in is understandable. A 'dynamic cinematic handheld dolly zoom with anamorphic flares and documentary realism' is a pile of competing instructions. If everything is cinematic, nothing is.

Lighting works the same way. Describe the dominant source, its direction, its color, and its contrast. 'Warm lamp from the left, deep shadows on the right, cool window light in the background' is more useful than 'moody lighting.' If the scene depends on a time of day, say it plainly. Blue hour, overcast noon, and sodium-vapor night produce different textures and different emotional reads.

Motion needs a budget. A shot can support a subject action, a camera move, and an atmospheric movement, but not five of each. If the character turns, the camera pushes in, curtains billow, papers fly, and rain streaks across the frame, the model may choose the wrong element to animate. Assign one primary motion and one secondary motion. Let the rest be texture.

A practical exercise is to rewrite one transcript line three ways: intimate, observational, and dramatic. For 'I should have called,' an intimate version might be a close-up on a thumb hovering over a contact name. An observational version might be a wide shot of a person alone on a bench while crowds pass. A dramatic version might be a slow push-in as the phone screen goes dark. None is universally correct. The choice depends on the scene's job in the story.

Use the AI video generator as a testing ground for these choices. Generate short variations, compare them against the shot card, and keep the version that communicates the beat most clearly. The goal is not to make every frame beautiful. The goal is to make every shot legible.

Handle Dialogue, Voiceover, and On-Screen Text

Transcripts are full of words, but video models are not reliable transcription machines. Long dialogue prompts often produce awkward lip movement, garbled audio, or text artifacts. Decide early whether the words will be spoken on camera, delivered as voiceover, shown as on-screen text, or removed entirely.

If dialogue must be on camera, keep lines short and physically actable. A character saying 'I'm not going back' can be performed in a few seconds. A character explaining a three-sentence backstory while walking through a crowd is a sync problem waiting to happen. Use reaction shots and cutaways to carry longer meaning. The transcript can remain rich; the prompt should stay performable.

Voiceover is more forgiving. You can generate visual shots that illustrate the narration without forcing lip sync. In that case, separate the voiceover script from the visual prompt. The LLM can mark which transcript lines are narration and which are visual cues. That prevents the model from trying to render a voice that belongs in post-production.

On-screen text needs its own discipline. Video models often distort small type, create fake logos, or invent captions. If a shot requires a sign, a phone message, or a title card, specify the exact short text and its placement. Keep it brief. For anything text-heavy, generate the background plate and add typography later in an editor. The LLM should flag text-heavy moments rather than forcing them into the video prompt.

A useful LLM instruction is: 'Classify each transcript line as spoken dialogue, voiceover, visual action, or removable filler. Do not put more than twelve spoken words into a single shot prompt.' That one constraint prevents many bad generations.

Keep Characters and Locations Consistent

Character drift is one of the most common failures in multi-shot AI video. The face changes, the jacket changes color, the hair length shifts, and the location stops matching. An LLM cannot guarantee visual consistency by itself, but it can enforce consistent descriptions and continuity notes.

Create a character sheet before writing prompts. Include age range, build, hair, wardrobe, accessories, posture, and one distinguishing feature. Avoid generic labels like 'beautiful woman' or 'businessman.' Use visual specifics that survive generation: 'silver hoop earrings, navy work jacket, close-cropped hair, habit of holding a phone with both hands.' Keep the sheet short enough to reuse without bloating every prompt.

Do the same for locations. A rooftop is not just a rooftop. It has an access door, a railing, a vent stack, a skyline direction, and a weather condition. If the story returns to that roof in three scenes, the location sheet should lock those details. The LLM can then write continuity lines such as 'same rooftop as s01, door on camera left, skyline behind subject.'

For difficult sequences, generate reference stills before video. A still image can establish a face, wardrobe, or set design that you then reference in the video prompt. The AI image generator can help create those anchors quickly. This is especially useful when a character appears in multiple angles or when a location needs a consistent layout.

Consistency also applies to tone. If the first shot is naturalistic and the fifth shot is surreal, the transition should be intentional. Ask the LLM to review the sequence for tonal jumps that are not supported by the transcript. Sometimes a jump is exactly right. Often it is an accident caused by inconsistent prompt language.

Iterate in Passes Instead of Rewriting Everything

A common mistake is to rewrite the entire prompt after a bad generation. That makes it impossible to know what changed. Use passes instead. Each pass tests one layer of the shot card.

Start with the narrative pass. Does the shot communicate the story beat? If not, change the action or the framing, not the lighting. A beautiful shot that does not advance the beat is still a failed shot. Ask the LLM to describe what a viewer would understand from the current prompt without seeing the transcript. If the answer is vague, the prompt needs a clearer action.

Move to the staging pass. Is the subject positioned clearly? Is the action physically possible in the shot duration? Is there a clear before-and-after state? If the model keeps adding extra people, simplify the background. If the action is unreadable, move the camera closer or split the shot.

Then run the cinematic pass. Adjust lens, light, color, and atmosphere. This is where you can make the shot feel like the same film as its neighbors. Compare the shot card with adjacent shots. If the previous scene is warm and handheld, a sudden cold static wide shot may need a transition or a story reason.

Finish with the technical pass. Check aspect ratio, duration, frame rate, reference images, and negative constraints. Keep a log of what you changed. A simple log with columns for shot, prompt version, change, result, and next action saves hours. The Orelon blog is a good place to find workflow ideas that support this kind of disciplined iteration.

When a generation fails, ask a diagnostic question instead of a vague improvement request. 'Why did the camera move become a zoom?' 'Which words may have caused the extra character?' 'Is the action too complex for a five-second shot?' LLMs are better at diagnosing specific conflicts than at guessing your taste.

Quality Control Before You Generate More

Before you spend more time generating, review the sequence as a whole. Story legibility comes first. Can someone follow the sequence without reading the transcript? If not, add an establishing shot, a reaction shot, or a clearer action. The transcript may contain context that the visuals have not earned yet.

Check continuity next. Do wardrobes, props, locations, and time of day match? Are there contradictory light directions? Does a character enter from the same side of the frame in consecutive shots? These details are easy to miss in individual prompts and obvious in sequence.

Look for prompt contradictions. 'Wide close-up' is not a shot size. 'Static camera push-in' is not a camera move. 'Bright night interior lit by a single candle' may be physically possible, but the model needs a clear exposure strategy. Contradictions do not make a prompt sophisticated; they make it unstable.

Test temporal feasibility. If a shot is four seconds long, the character cannot walk across a room, answer a phone, and sit down. Split the action or extend the duration. Video models respond to time pressure, even when the prompt does not mention seconds. Too many actions produce morphing, skipped beats, or unnatural speed.

Check text and audio expectations. If the shot depends on readable text, plan to add it in post. If the shot depends on precise lip sync, plan for a shorter line or a different angle. If the shot depends on a specific sound, note it for the edit rather than assuming the video model will create it.

Finally, check rights and safety. Do not prompt for a real person's likeness without permission. Do not use trademarks as set dressing unless you have the right to do so. Avoid generating private information, medical claims, or sensitive content without appropriate review. A clean transcript-to-prompt workflow should include these checks before generation, not after publication.

Common Mistakes That Weaken LLM-to-Video Prompts

The first mistake is treating the transcript as a shot list. A transcript explains meaning through words. A shot list explains meaning through visible changes. The LLM must convert one into the other, not paste paragraphs into a prompt.

The second mistake is overloading style words. 'Cinematic, epic, masterpiece, hyperrealistic, 8K, film grain, volumetric light' can cancel each other out. Choose the few style cues that matter for the shot. Let the story carry the emotion.

The third mistake is vague emotion. Replace adjectives with visible behavior, framing, and light. 'Anxious' becomes a bouncing knee, a tight close-up, a flickering fluorescent tube. 'Relieved' becomes a loosened grip, a wider frame, warmer light.

The fourth mistake is inconsistent naming. If the character is 'Maya' in one prompt and 'the woman in the green jacket' in another, the model may treat them as different people. Use the same character label across all shot cards.

The fifth mistake is long dialogue in visual prompts. Keep spoken lines short or move them to voiceover. Let the camera capture reaction and subtext.

The sixth mistake is ignoring duration. A prompt that describes a mini-scene will struggle in a short clip. Match the action to the time available. If the idea needs more time, use more shots.

The seventh mistake is letting the LLM invent important details without review. An LLM may add a wedding ring, a scar, or a location change that conflicts with the story. Mark invented details clearly and approve them before they enter the shot cards.

The eighth mistake is building one mega-prompt for the whole video. A video model needs shot-level direction. A sequence needs continuity notes. Those are related but not the same artifact. Keep a sequence outline and separate shot prompts.

FAQ: Transcripts, LLMs, and AI Video Prompts

How long should each shot prompt be?

Long enough to cover the story beat, subject, action, setting, camera, light, and continuity, but short enough to avoid contradictions. For many models, a focused paragraph plus a short negative list works better than a page of adjectives. If the prompt feels crowded, move style and continuity notes into a reusable block.

Can I feed a raw transcript directly to a video generator?

You can, but the results are unpredictable. A raw transcript lacks shot boundaries, visual anchors, and camera direction. Use an LLM to segment it into beats and shot cards first. That extra step usually saves more time than it costs.

Do I need an LLM to make AI video prompts?

No, but an LLM is useful for volume and consistency. It can extract beats, maintain character sheets, rewrite abstract language into visual evidence, and diagnose failed prompts. You still need to review the output. The LLM is a translator and assistant, not the director.

How do I stop character drift across shots?

Use a compact character sheet, repeat the same labels, and include continuity notes in every relevant shot card. Generate reference stills for difficult faces or wardrobes. Keep camera angles varied but description stable. If a shot still drifts, simplify the background and reduce competing actions.

Should dialogue go inside the video prompt?

Only when the line is short and the shot depends on visible performance. Otherwise, use voiceover or add dialogue in post. Long spoken lines often create lip-sync artifacts and pull the model away from the visual action.

How many shots do I need per minute?

There is no fixed number. A dialogue scene may use fewer, longer shots. A montage may use many short ones. Plan by story beats, not by a quota. If a beat needs two visual changes, give it two shots. If one shot communicates the beat clearly, do not cut it just to increase pace.

Will the same prompt work across different video models?

Not exactly. Models interpret camera language, motion, and style differently. Keep the shot card as the stable source of truth, then adapt the final prompt to the model you are using. Test a short batch before committing to a full sequence.

What is the best first step if a generation fails?

Compare the result to the shot card and identify the first layer that broke: story, staging, camera, light, or technical settings. Change one variable and regenerate. If you change everything at once, you learn nothing from the next result.

Turn the transcript into motion with Orelon. Start with a clean transcript, let an LLM extract beats and shot cards, then bring those cinematic ideas to life in the AI video generator. Keep your prompts structured, your continuity deliberate, and your review process honest. That is how a transcript stops being raw text and starts becoming a sequence with direction.