Video Prompt Writing Guide (T2VA / I2VA / FL2VA / L2VA)
1. Task Overview
- T2VA: Builds a complete audiovisual timeline from text.
- I2VA: T2VA body + first-frame instruction + a visual path that develops forward from the first frame.
- FL2VA: T2VA body + first-and-last-frame instruction + a continuous path from the first frame to the last frame.
- L2VA: T2VA body + last-frame instruction + a path that converges from a plausible preceding state to the last frame.
2. Final Prompt Structure
2.1 Part One Is the Instruction
T2VA has no image-alignment instruction and begins directly with the three core fields.
I2VA always uses:
For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced.
FL2VA always uses:
How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot N) aligns with the S.SS-second mark of the target video.
L2VA always uses:
How the reference pictures align with the target video — <Picture 1> (from [Shot N]) aligns with the S.SS-second mark of the target video.
Here, N is the index of the actual final shot, and S.SS is the effective video duration formatted to exactly two decimal places. The instruction must be the first line of the final prompt, followed by one blank line before the core fields.
2.2 Part Two Contains the Three Core Fields
integrated_multimodal_description: [Shot 1] ...
overall_soundscape: ...
non_diegetic_music: ...
- integrated_multimodal_description: Describes visuals, actions, shots, speakers, dialogue, singing, and diegetic audio along the timeline.
- overall_soundscape: Summarizes ambient sound, physical action sounds, and non-verbal human sounds across the entire video.
- non_diegetic_music: Describes background music that the characters cannot hear and only the audience can hear.
3. How to Incorporate Keyframes into the Multimodal Description
3.1 I2VA: Begin from the Image and Develop Forward
<Picture 1> is the actual first frame of the video at 0.00 seconds and belongs to [Shot 1]. The description should first establish the style, subjects, composition, and scene anchors in the image, then describe the next action. Character identity, clothing, colors, key objects, and spatial relationships should remain consistent.
Recommended structure: first-frame anchor → action onset → continuous development → result or reaction.
3.2 FL2VA: Describe the Path Between the First and Last Frames
Picture 1 is the opening, and Picture 2 is the ending. Focus on how the subject moves, how poses change, how objects are manipulated, how the composition evolves, and how the scene or lighting transitions.
FL2VA generally favors a single shot so the model can interpolate continuously from the first frame to the last frame. Use multiple shots only when they are explicitly specified. The last frame must be reached by the final [Shot N] at the end of the video.
Recommended structure: first-frame state → observable intermediate changes → progressively narrowing differences → last-frame state.
3.3 L2VA: Infer the Opening and Land on the Image at the End
<Picture 1> is the final frame of the video and belongs to the last [Shot N]; it does not inherently belong to Shot 1. Infer a plausible earlier state from the user's intent and the last frame, then describe how the characters, objects, camera, and scene gradually approach the reference image.
Recommended structure: plausible preceding state → explicit action and transition path → gradual convergence in the final shot → last-frame landing.
4. How to Write the Three Shared Core Sections
4.1 Develop the Multimodal Description Along the Timeline
integrated_multimodal_description is the main body of the rewritten prompt. Every detail should correspond to something visible or audible: visual style, initial composition, subject appearance and position, scene and key props, actions and reactions, shot changes, spoken language, and synchronized diegetic sound.
At the beginning of [Shot 1], state the overall style and initial composition. Common styles include Cinematic, live-action, 2D-animated, 3D CG, claymation, watercolor, and vintage film. For keyframe tasks, derive the style from the reference image; for T2VA, select it from the user's text.
[Shot 1] Live-action, cinematic, a medium-wide shot frames...
4.2 Shots and Cuts
Do not add a timestamp to the first shot. Use sequential shot numbers for later shots, and begin each one with a strictly increasing cut time that falls within the video duration:
[Shot 2] At 00:03.500, the camera cuts to...
For ordinary cuts, use the camera cuts to, the shot cuts to, the shot transitions to, the shot changes to, or the shot switches to. When explicitly requested by the user, cross-dissolve, fade, or wipe may also be used. A cut should introduce new information about the subject, space, state, viewpoint, or time. If only the distance or a slight angle needs to change, prefer camera motion.
4.3 Camera Motion: Motion Type + Amplitude + Speed
A complete camera-motion expression has three dimensions: the motion type defines how the camera moves, amplitude defines the range of compositional change, and speed defines the pacing of that change. Add amplitude and speed only when they are meaningful; medium amplitude and normal speed are usually omitted.
| Dimension | Available Expression | Description |
|---|---|---|
| Motion type | Zoom In / Zoom Out |
The focal length changes while the camera body remains stationary |
| Motion type | Push In / Pull Out |
The camera moves forward / backward |
| Motion type | Pan Left / Pan Right |
The camera remains in place while the lens pivots horizontally |
| Motion type | Truck Left / Truck Right |
The camera translates horizontally |
| Motion type | Tilt Up / Tilt Down |
The camera remains in place while the lens pivots vertically |
| Motion type | Pedestal Up / Pedestal Down |
The entire camera moves upward / downward |
| Motion type | Arc Shot |
The camera moves in an arc around the subject |
| Motion type | Tracking Shot |
The camera follows a moving subject |
| Motion type | Static Shot |
The camera position and lens remain still |
| Motion type | Shake Slightly / Shake Strongly |
Slight / strong camera shake |
| Motion type | POV |
The subject's point of view |
| Motion type | Roll Clockwise / Roll Counterclockwise |
The camera rolls clockwise / counterclockwise around the lens axis |
| Amplitude | with small amplitude |
Small-range change |
| Amplitude | with large amplitude |
Large-range change |
| Speed | at slow speed |
Slow movement |
| Speed | at fast speed |
Fast movement |
Camera motion should be written as a natural English action within the shot, rather than stacked as separate labels at the end of a sentence:
The camera pushes in with small amplitude at slow speed toward the folded letter in her hands.
The camera pans right with large amplitude at fast speed, revealing the open doorway.
The camera holds a static shot as the runner exits the frame.
4.4 Speakers, Dialogue, and Singing
Subjects who speak, sing, or produce an off-screen human voice use stable IDs such as (S1) and (S2). When multiple already-numbered speakers speak or sing together, use a compound ID such as (S1,S2). A speaker keeps the same ID across shots; characters who never vocalize receive no speaker ID.
When a speaker first appears, provide enough information from the visual and audio context to establish a stable identity, such as character type, age, gender, whether the person is on-screen, pitch, timbre, speaking rate, or accent. Place the speaker's identifying phrase, ID, action, and delivery outside <d>. Inside <d>, include only the language tag and the actual user-provided spoken content. Preserve every original word and punctuation mark verbatim; do not translate or rewrite them.
The young woman with a quiet, breathy voice (S1) says: <d>[English] I get off at the next station.</d>
The two children (S1,S2) shout together, <d>[English] Wait for us!</d>
For voiceover, use the exact phrase says in an off-screen voiceover. Immediately after every voiceover <d> block, state that the corresponding on-screen character's lips remain closed:
The man (S1) says in an off-screen voiceover: <d>[English] I still remember that road.</d> while his lips remain completely closed.
When the same line of dialogue or lyrics crosses a cut, use <scenetrans> at the connecting points in both parts and explicitly state that the audio continues across the cut. Use <cutoff> when speech is truncated by the end of the video. Continuity may be expressed with continues seamlessly across the cut, continues uninterrupted into the next shot, carries over from the previous shot, or remains audible across the transition.
4.5 On-Screen Text
Place any banner, sign, label, subtitle, or neon text that is actually visible on screen in English double quotation marks. Preserve the original text and punctuation verbatim, without translation.
A red neon sign reading "营业中" glows above the doorway.
4.6 overall_soundscape
Use 1–4 English sentences in one continuous paragraph to summarize the ambient sound, physical action sounds, and non-verbal human sounds across the full video, such as wind, rain, traffic, footsteps, fabric movement, impacts, breathing, laughter, or panting. Dialogue, singing, and diegetic music already belong in the multimodal description and should not be repeated here. Use N/A only when the user explicitly requests complete silence throughout the video.
overall_soundscape: Steady rain taps against the café windows while low room ambience continues underneath. The entrance bell rings once, followed by wet footsteps and the soft scrape of a chair.
4.7 non_diegetic_music
Use 1–3 English sentences to describe background music that the characters cannot hear and only the audience can hear. Focus on instrumentation, speed, rhythm, and dynamic changes; do not use abstract mood words or explain the emotional function of the score. Singing, instruments, radio, television, or phone music audible to the characters are diegetic events and should appear in the multimodal description. Use N/A when there is no non-diegetic music.
non_diegetic_music: Sparse piano notes at a slow tempo, joined by sustained low strings that gradually increase in volume before fading out.
5. Cases
Case 1: T2VA
With no reference image, construct the complete timeline directly from the text. You may add scene, character, action, and sound details that remain consistent with the user's intent.
integrated_multimodal_description: [Shot 1] Live-action, cinematic, a medium-wide shot frames a baker opening the shutters of a small street bakery before sunrise. The camera pushes in with small amplitude at slow speed as the middle-aged baker with a calm, slightly raspy voice (S1) places a fresh loaf on the wooden counter and says: <d>[English] First batch of the morning.</d> [Shot 2] At 00:05.000, the camera cuts to a close-up of steam rising from the sliced bread while the baker's final words carry over from the previous shot.
overall_soundscape: Wooden shutters scrape open over a quiet street as trays clink softly inside the bakery. The doorbell rings once, followed by light footsteps and the crisp sound of bread being sliced.
non_diegetic_music: A soft acoustic-guitar pattern at a moderate tempo, joined by sparse upright-bass notes and a gentle fade at the end.
Case 2: I2VA
Write the first-frame instruction first, then use the subject, composition, and scene in Picture 1 as the starting point of Shot 1 before describing how the scene continues to develop.
For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced.
integrated_multimodal_description: [Shot 1] Live-action, cinematic, the young woman shown in <Picture 1> remains beside the rain-covered train window, preserving her appearance, clothing, seat position, and the carriage layout. The camera trucks right with small amplitude at slow speed as she lifts her gaze from the folded letter toward the passing city lights. Her reflection moves across the glass while the quiet, breathy young woman (S1) says: <d>[English] I get off at the next station.</d> She folds the letter along its existing crease.
overall_soundscape: The train wheels produce a steady metallic rhythm beneath a low ventilation hum. Rain ticks against the window while paper rustles softly in her hands.
non_diegetic_music: Sustained cello notes at a slow tempo with widely spaced piano tones, gradually decreasing in volume.
Case 3: FL2VA
The two images anchor the opening and ending respectively. The body should not repeat two static image descriptions; instead, it should supply the motion path that connects them. The following example is an eight-second single shot.
How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 8.00-second mark of the target video.
integrated_multimodal_description: [Shot 1] Live-action, cinematic, a rain-soaked cyclist begins in the position and framing established by Picture 1, holding a closed black umbrella beside a silver bicycle. The camera pulls out with small amplitude at slow speed as she releases the bicycle handle, raises the umbrella above her shoulder, and presses the runner upward until the canopy opens. Water rolls from the expanding fabric while she steps beneath it, rotates the handle into the final angle, and settles into the pose, spacing, and composition established by Picture 2 at the end of the shot.
overall_soundscape: Rain falls steadily on the pavement, followed by the metallic click of the umbrella runner and the soft snap of the canopy opening. Water drips from the bicycle frame as distant traffic passes.
non_diegetic_music: N/A
Case 4: L2VA
The image anchors only the final moment. First establish a compatible earlier state, then let the actions, object states, and composition gradually land on Picture 1 in the final shot. The following example is a six-second single shot.
How the reference pictures align with the target video — <Picture 1> (from [Shot 1]) aligns with the 6.00-second mark of the target video.
integrated_multimodal_description: [Shot 1] Live-action, cinematic, a close shot begins with an intact drinking glass near the edge of a dark wooden table, while the same hand and sleeve visible in <Picture 1> approach from the right. The camera pushes in with small amplitude at slow speed as the fingertips strike the rim. The glass tips, falls, and hits the floor with a sharp impact; cracks spread through it as fragments slide outward. Toward the end, the moving pieces lose momentum and settle into the exact broken arrangement, hand position, camera angle, lighting, and final composition established by <Picture 1>.
overall_soundscape: Fingertips tap the glass before it scrapes across the tabletop, falls, and breaks with a sharp crash. Small fragments scatter and gradually stop sliding across the floor.
non_diegetic_music: A low electronic pulse at a slow tempo, ending immediately after the glass breaks.