Naming something that is not in the frame makes the model bring it into the frame
Title: I2V: naming a person or object that is not in the frame makes the model bring it in (with numbers and a workaround)
Setup: LTX-2.5 distilled bf16, ComfyUI two-stage I2V, 720p, 121 frames @ 24 fps, one seed per clip; a few hundred clips across three episodes of a film series.
An off-screen listener walks into the frame. In 10 prompts that mentioned someone "out of frame" / "outside the frame" (e.g. "speaks to a man who kneels out of frame"), that person entered the shot in 6 clips: a head at the edge, a shoulder, or a whole person, often in modern clothing that has nothing to do with the start frame. "Someone just below and in front of the camera" did the same. The line "He is alone in the frame and nobody else enters it." was in the same prompts and did not prevent it. A different seed does not fix it: on seed 11 only 1 of 7 such shots came out clean.
A light source named in the setting pulls the camera away from the face. Close-ups of a woman indoors had the setting line "soft daylight from a smoke-hole in the roof". In 4 of 7 close-ups the camera left her face at second 2–3 and showed an empty room with a hole in the roof. A part of the same scene whose setting line named a lamp that was in the frame had no such drift.
Workaround that held for us: never name what is not in the frame. Describe gaze by direction ("looks to the left of the frame"), not by addressee; in close-ups give light only ("soft dim daylight indoors, warm brown shadows"), with no objects or sources. After this change, none of the next 188 clips drifted away from the subject.
- Smaller sample, for what it's worth: in a two-person shot, a gesture written for one character (holding up a torch and looking at the sky) was performed by the other one in 2 of 2 clips (same start frame and seed), while the line of dialogue was spoken by the right character.
Our reading is that the text encoder treats every noun as something to show. If that is expected behaviour, a line about it in the prompting guide would help; if not, we hope these numbers are useful. Happy to share frame strips if the team wants them.
Thanks for documenting this with examples and numbers. Describing gaze direction and visible lighting instead of naming off-screen people or objects sounds like a useful approach.
I’m working on a film-restoration project where preserving composition, actor identity, expressions, and movement matters. A few questions that could help explore this further:
Controlled comparisons: Have you rerun the original failing shots using the revised wording while keeping the starting image, seed, and settings identical? A before-and-after prompt pair with frame strips would be especially helpful.
Prompt enhancement: Was any automatic prompt enhancement enabled? If so, could you share the final text passed to the model? It would be interesting to see whether the enhancement introduced or emphasized off-screen details.
Character actions: For the gesture assigned to the wrong person, have you tried identifying each character by screen position and visible clothing—for example, “the person on the left wearing a brown robe raises the torch”? Does that improve action assignment?
Performance fidelity: After removing off-screen references, did you also see better preservation of gaze, expressions, and intended movement, or was the improvement mainly in keeping the camera on the subject?
Your results seem worth including in the prompting guide. I’d also be interested in the team’s interpretation of whether this comes from text encoding, the model’s learned response to those descriptions, or another part of the pipeline.
Thanks for documenting this with examples and numbers. Describing gaze direction and visible lighting instead of naming off-screen people or objects sounds like a useful approach.
I’m working on a film-restoration project where preserving composition, actor identity, expressions, and movement matters. A few questions that could help explore this further:
Controlled comparisons: Have you rerun the original failing shots using the revised wording while keeping the starting image, seed, and settings identical? A before-and-after prompt pair with frame strips would be especially helpful.
Prompt enhancement: Was any automatic prompt enhancement enabled? If so, could you share the final text passed to the model? It would be interesting to see whether the enhancement introduced or emphasized off-screen details.
Character actions: For the gesture assigned to the wrong person, have you tried identifying each character by screen position and visible clothing—for example, “the person on the left wearing a brown robe raises the torch”? Does that improve action assignment?
Performance fidelity: After removing off-screen references, did you also see better preservation of gaze, expressions, and intended movement, or was the improvement mainly in keeping the camera on the subject?
Your results seem worth including in the prompting guide. I’d also be interested in the team’s interpretation of whether this comes from text encoding, the model’s learned response to those descriptions, or another part of the pipeline.
Thanks for the careful questions. Here is what we measured, and also what we did not.
- Controlled comparison. We have one clean pair. The four close-ups that drifted to the roof were reshot with the same start frame, the same seed (7), the same workflow and the same model files. Only the setting line changed:
"a small mud-brick room hung with felt, soft daylight from a smoke-hole in the roof" → "soft dim daylight indoors, warm brown shadows".
(It was a different A100 pod, so the runs are not bit-identical.)
What the frames show: in the original clips the shot starts on the woman's face as in the start image. Around second 2–3 the camera slides off her and up, and the rest of the clip is an empty felt-hung room with a hole in the roof; she is out of frame or at the edge. In the retakes the camera stays on her head and shoulders for the whole clip; she says her line and holds still, looking where the prompt says. All 4 retakes stayed on the face.
The "188 clips with no drift" in my post were new shots written with the rule, not pairs.
For the off-screen listener the evidence is weaker. With the same prompt on a new seed (11) the person still walked in, in all 6 shots, so the seed did not help. But we only tested the reworded prompt (gaze by direction, no addressee) locally on LTX-2.3 (1 intruder in 12 clips), not as a same-seed A/B on 2.5. So for that case the claim is only "changing the prompt helps more than changing the seed".
Prompt enhancement. None. ComfyUI, with the prompt going straight into CLIPTextEncode (Gemma text encoder from the repo), no enhancer node and no LLM rewrite. The strings I quoted are exactly what the model got; the off-screen nouns were ours.
Character actions. A correction first: it was not a torch. The prompt said the old man "points the split stick at the young man's collar, then at the coals, then lifts his eyes to the sky", and the young man did it instead. Both men were identified by visible appearance (white beard, bronze neck-ring, grey wolf-skin mantle vs. long snow-white hair and a dark wool coat), but not by screen position. We have not tried "the man on the left…". We worked around it instead: a gesture that carries meaning goes on a single-person shot, and two-person wide shots get only simple actions. So your question is still open, and our sample was only 2 clips.
Performance fidelity. The improvement was mainly the camera staying on the subject. In the drifted clips the face left the frame by second 2–3, so there was no performance left to compare. We accepted all four retakes by eye (lines spoken as written), but we did not score expression or movement separately. One related note: with the first + last frame workflow the camera was right in 6 of 6 clips, but the acting looked unnatural in 2 of 4 shots. Getting the camera right did not give us better acting.
Interpretation. Honestly, we can't tell. "The encoder treats every noun as something to show" was our guess from the outputs; we did not look inside the encoder, so we can't separate text encoding from learned behaviour. What we can say: the drift came when a phrase named something absent from the start frame. A similar setting line naming a lamp that was in the frame caused no drift. A later episode hints at the same thing with "tower" in wide shots, but we haven't tested that.