README.md

---
license: other
license_name: ltx-2-community
license_link: https://github.com/Lightricks/LTX-2/blob/main/LICENSE
base_model: Lightricks/LTX-2.3
tags:
- video
- lora
- camera-control
- interactive
---

# helloworld-interactor-lora-r32-2000

> ⚠️ **This LoRA is a fine-tuned Derivative of LTX-2 and is distributed under the
> [LTX-2 Community License](LICENSE-LTX2)** (including its use-based restrictions,
> Attachment A). A complete copy of the license is included in this repository.

Rank-32 LoRA for **Lightricks LTX-2.3** that adds **camera-trajectory following from warp-video
history conditioning** — the self-distillation recipe of
["HelloWorld: Enabling Socially Interactive Characters in Video World Models"](https://arxiv.org/abs/2608.05070)
(Ouyang, Liu, Chu, Zhang, Sato — Alaya Lab / The University of Tokyo), reimplemented
independently before the authors' official code release. Paired with the training-free F-press
cross-attention mask from the same paper, you get characters that perform at the camera inside
a chosen time window while the shot follows a camera path you draw.

- **Code, warp pipeline, Eq. 4 mask, training config, reproduction guide:**
  https://github.com/scrappylabsai/helloworld-interactor
- **What it is:** LoRA rank 32, alpha 32, on all self-attention projection matrices of the
  video branch (`attn1.*`), trained 2000 steps @ LR 1e-4, flow matching, bf16, on 156
  self-generated 1280×704×121 clips with recovered-trajectory warp videos as reference
  conditioning; camera text excluded from captions (paper §3.2, §5.2).
- **Training cost:** ~12 h on one rented RTX PRO 6000 (96 GB), peak 72.3 GB; whole project
  ~$43.50.

## Results (mini-bench, 40 generated characters, base vs LoRA)

| Metric | Judge | Base LTX-2.3 | + this LoRA | Paper |
|---|---|---|---|---|
| Performs in window (n=9 pairs) | Gemini, true-video | 1/9 (11%) | **7/9 (78%)** | TimeAcc 81.7% |
| TimeAcc (n=40) | Qwen3.6-35B-A3B, frame grids | 21% | **44%** | 81.7% |
| Performs at all (n=40) | Qwen3.6-35B-A3B, frame grids | 24/40 | **34/40** | — |

Qwen3.6-35B-A3B is the paper's judge model; our serving feeds it frame grids rather than raw
video, which attenuates absolute scores for both columns (motion between sampled frames is
invisible) — the true-video Gemini calibration on the same clips lands at the paper's number
within small-n error. Caveats: n=40/n=9 (paper: 400); trained at 121 frames vs the paper's 241;
our geometric CamCtrl metric is pending a pose-surveyor bugfix (qualitative camera control —
dolly-in, orbit — confirmed side-by-side at fixed seed). Where the trained model missed the
directed interaction (2/9 true-video pairs, both subtle low-amplitude gestures — a wink, a
sliding object), the untrained baseline missed the same clips: misses degrade to baseline
behavior rather than below it, and trained clips showed fewer artifacts (1/9 vs 3/9).

## Example pairs (same seed, same prompt, same warp input)

Full-quality mp4s in [`examples/`](examples/) — `<id>__base.mp4` is stock LTX-2.3,
`<id>__lora.mp4` is +this LoRA. All characters fully generated.

| Character | What was directed (interaction · window · camera) | Base | +LoRA |
|---|---|---|---|
| Brewer (taproom) | Raises a full glass toward the viewer in a toast, saying "Cheers!" · 1.5–2.5 s · dolly-in | [base](examples/brewer-taproom__base.mp4) | [LoRA](examples/brewer-taproom__lora.mp4) |
| Baker (window) | Holds up a steaming loaf toward the viewer and winks · 1.5–2.5 s · static | [base](examples/baker-window__base.mp4) | [LoRA](examples/baker-window__lora.mp4) |
| Clay snowman | Tips his tiny top hat toward the camera · 1.5–2.5 s · dolly-in | [base](examples/clay-snowman__base.mp4) | [LoRA](examples/clay-snowman__lora.mp4) |
| Tin robot (workbench) | Winds its own key, then salutes the viewer stiffly · 1.5–2.5 s · dolly-in | [base](examples/robot-workbench__base.mp4) | [LoRA](examples/robot-workbench__lora.mp4) |
| Plush octopus (bathtub) | Wiggles two tentacles toward the viewer in greeting · 1.5–2.5 s · orbit-left | [base](examples/plush-octopus__base.mp4) | [LoRA](examples/plush-octopus__lora.mp4) |
| Mechanic (garage) | Wipes his hands on a rag and gives the viewer a thumbs-up · 3.0–4.0 s · pan-right | [base](examples/mechanic-garage__base.mp4) | [LoRA](examples/mechanic-garage__lora.mp4) |

## Usage

With the repo's inference driver (LTX-2 packages installed + the repo's vendor patch applied —
see the repo's `docs/REPRODUCE.md`):

```bash
python -m helloworld_ltx.infer_warp \
    --distilled-checkpoint-path models/ltx-2.3/ltx-2.3-22b-distilled-1.1.safetensors \
    --gemma-root models/gemma-3-12b \
    --spatial-upsampler-path models/ltx-2.3/ltx-2.3-spatial-upscaler-x2-1.1.safetensors \
    --image inputs/shot/first_frame.png 0 1.0 \
    --warp-npz inputs/shot/warp.npz \
    --scene "A lighthouse keeper on the gallery rail at dusk." \
    --interaction "He turns and waves at the viewer." \
    --quality "Photoreal, natural lighting." \
    --window 24 48 \
    --height 704 --width 1280 --num-frames 121 --frame-rate 24 \
    --lora helloworld-interactor-lora-r32-2000.safetensors 1.0 \
    --seed 42 --output-path out.mp4
```

`--warp-npz` comes from the repo's `warp/warp.py` (choose a trajectory) or `warp/annotate.py`
(recover one from a video). `--window` is the F-press window in pixel frames at the output rate.
Timing control is training-free; this LoRA adds the camera-following.

## Dependency license note (Pi3X)

Generating the warp inputs uses the **Pi3X / π³** geometry model
([arXiv:2507.13347](https://arxiv.org/abs/2507.13347)): its weights are **CC BY-NC 4.0
(non-commercial)** and are NOT included here or in the code repo. The LoRA weights themselves
have no Pi3X dependency, but the as-shipped warp pipeline does — commercial users must
substitute a permissively-licensed geometry backbone for warp generation.

## Credit

All credit for the method to the HelloWorld authors (Alaya Lab / The University of Tokyo) —
this is an unofficial reproduction from the paper text, published with full attribution.
Watch https://github.com/AlayaLab/HelloWorld for their official release.

Built by [ScrappyLabs](https://scrappylabs.ai).

*Try it on your own productions — if you direct your own characters with this, we'd love to see
the results: open a Discussion on the
[GitHub repo](https://github.com/scrappylabsai/helloworld-interactor) with your clips and what
you directed.*