license: other
license_name: ltx-2-community
license_link: https://github.com/Lightricks/LTX-2/blob/main/LICENSE
base_model: Lightricks/LTX-2.3
tags:
- video
- lora
- camera-control
- interactive
helloworld-interactor-lora-r32-2000
⚠️ This LoRA is a fine-tuned Derivative of LTX-2 and is distributed under the LTX-2 Community License (including its use-based restrictions, Attachment A). A complete copy of the license is included in this repository.
Rank-32 LoRA for Lightricks LTX-2.3 that adds camera-trajectory following from warp-video history conditioning — the self-distillation recipe of "HelloWorld: Enabling Socially Interactive Characters in Video World Models" (Ouyang, Liu, Chu, Zhang, Sato — Alaya Lab / The University of Tokyo), reimplemented independently before the authors' official code release. Paired with the training-free F-press cross-attention mask from the same paper, you get characters that perform at the camera inside a chosen time window while the shot follows a camera path you draw.
- Code, warp pipeline, Eq. 4 mask, training config, reproduction guide: https://github.com/scrappylabsai/helloworld-interactor
- What it is: LoRA rank 32, alpha 32, on all self-attention projection matrices of the
video branch (
attn1.*), trained 2000 steps @ LR 1e-4, flow matching, bf16, on 156 self-generated 1280×704×121 clips with recovered-trajectory warp videos as reference conditioning; camera text excluded from captions (paper §3.2, §5.2). - Training cost: ~12 h on one rented RTX PRO 6000 (96 GB), peak 72.3 GB; whole project ~$43.50.
Results (mini-bench, 40 generated characters, base vs LoRA)
| Metric | Judge | Base LTX-2.3 | + this LoRA | Paper |
|---|---|---|---|---|
| Performs in window (n=9 pairs) | Gemini, true-video | 1/9 (11%) | 7/9 (78%) | TimeAcc 81.7% |
| TimeAcc (n=40) | Qwen3.6-35B-A3B, frame grids | 21% | 44% | 81.7% |
| Performs at all (n=40) | Qwen3.6-35B-A3B, frame grids | 24/40 | 34/40 | — |
Qwen3.6-35B-A3B is the paper's judge model; our serving feeds it frame grids rather than raw video, which attenuates absolute scores for both columns (motion between sampled frames is invisible) — the true-video Gemini calibration on the same clips lands at the paper's number within small-n error. Caveats: n=40/n=9 (paper: 400); trained at 121 frames vs the paper's 241; our geometric CamCtrl metric is pending a pose-surveyor bugfix (qualitative camera control — dolly-in, orbit — confirmed side-by-side at fixed seed). Where the trained model missed the directed interaction (2/9 true-video pairs, both subtle low-amplitude gestures — a wink, a sliding object), the untrained baseline missed the same clips: misses degrade to baseline behavior rather than below it, and trained clips showed fewer artifacts (1/9 vs 3/9).
Example pairs (same seed, same prompt, same warp input)
Full-quality mp4s in examples/ — <id>__base.mp4 is stock LTX-2.3,
<id>__lora.mp4 is +this LoRA. All characters fully generated.
| Character | What was directed (interaction · window · camera) | Base | +LoRA |
|---|---|---|---|
| Brewer (taproom) | Raises a full glass toward the viewer in a toast, saying "Cheers!" · 1.5–2.5 s · dolly-in | base | LoRA |
| Baker (window) | Holds up a steaming loaf toward the viewer and winks · 1.5–2.5 s · static | base | LoRA |
| Clay snowman | Tips his tiny top hat toward the camera · 1.5–2.5 s · dolly-in | base | LoRA |
| Tin robot (workbench) | Winds its own key, then salutes the viewer stiffly · 1.5–2.5 s · dolly-in | base | LoRA |
| Plush octopus (bathtub) | Wiggles two tentacles toward the viewer in greeting · 1.5–2.5 s · orbit-left | base | LoRA |
| Mechanic (garage) | Wipes his hands on a rag and gives the viewer a thumbs-up · 3.0–4.0 s · pan-right | base | LoRA |
Usage
With the repo's inference driver (LTX-2 packages installed + the repo's vendor patch applied —
see the repo's docs/REPRODUCE.md):
python -m helloworld_ltx.infer_warp \
--distilled-checkpoint-path models/ltx-2.3/ltx-2.3-22b-distilled-1.1.safetensors \
--gemma-root models/gemma-3-12b \
--spatial-upsampler-path models/ltx-2.3/ltx-2.3-spatial-upscaler-x2-1.1.safetensors \
--image inputs/shot/first_frame.png 0 1.0 \
--warp-npz inputs/shot/warp.npz \
--scene "A lighthouse keeper on the gallery rail at dusk." \
--interaction "He turns and waves at the viewer." \
--quality "Photoreal, natural lighting." \
--window 24 48 \
--height 704 --width 1280 --num-frames 121 --frame-rate 24 \
--lora helloworld-interactor-lora-r32-2000.safetensors 1.0 \
--seed 42 --output-path out.mp4
--warp-npz comes from the repo's warp/warp.py (choose a trajectory) or warp/annotate.py
(recover one from a video). --window is the F-press window in pixel frames at the output rate.
Timing control is training-free; this LoRA adds the camera-following.
Dependency license note (Pi3X)
Generating the warp inputs uses the Pi3X / π³ geometry model (arXiv:2507.13347): its weights are CC BY-NC 4.0 (non-commercial) and are NOT included here or in the code repo. The LoRA weights themselves have no Pi3X dependency, but the as-shipped warp pipeline does — commercial users must substitute a permissively-licensed geometry backbone for warp generation.
Credit
All credit for the method to the HelloWorld authors (Alaya Lab / The University of Tokyo) — this is an unofficial reproduction from the paper text, published with full attribution. Watch https://github.com/AlayaLab/HelloWorld for their official release.
Built by ScrappyLabs.
Try it on your own productions — if you direct your own characters with this, we'd love to see the results: open a Discussion on the GitHub repo with your clips and what you directed.