README.md

# LTX-V2V Reference-Conditioned Video-to-Video

Video-to-video generation on top of [LTX-2](https://github.com/Lightricks/LTX-2)
(Lightricks' DiT-based video+audio diffusion model): given a **source video**
(structure/motion signal) and a **first-frame reference image** (target
identity/appearance), generate an **output video** that follows the source
video's motion while adopting the reference image's look.

This repo is a trimmed-down, sharable fork containing:
- `packages/ltx-core`, `packages/ltx-pipelines` — the LTX-2 inference stack,
  with a small local patch to the reference-conditioning position logic (see
  below) that is **not** in upstream `Lightricks/LTX-2`.
- `packages/ltx-trainer` — LTX-2's LoRA/IC-LoRA training stack, with a matching
  patch on the training side, plus a `reference_image` dataset column so you
  can train on (video, reference_image, caption) triples.
- `scripts/infer_v2v.py` — new inference script for the video+ref-image →
  video workflow described above.
- `configs/*.yaml`, `scripts/train_ic_lora.sh` — training configs and launcher
  for the two custom IC-LoRA variants this workflow needs.
- `data/` — dataset format documentation and example JSONL rows.

**Not included:** pretrained IC-LoRA checkpoints. The ones we trained
internally were trained on licensed movie footage, so we're not redistributing
the weights — train your own with `scripts/train_ic_lora.sh` on your own data
(see `data/README.md`).

## Quick start

```bash
./setup.sh                     # uv sync + download ~90GB of base-model weights
source .venv/bin/activate

# 1. Prepare your data (see data/README.md for the JSONL schema)
scripts/preprocess_dataset.sh data/dataset.jsonl "1920x1024x233"

# 2. Train an IC-LoRA (defaults to configs/v2v_reference_ic_lora.yaml)
scripts/train_ic_lora.sh --config configs/v2v_reference_ic_lora.yaml

# 3. Run inference with your trained checkpoint
python scripts/infer_v2v.py \
  --input-video my_source_video.mp4 \
  --ref-image my_reference.png \
  --prompt "A woman in a red dress walks through a garden, camera tracking steadily." \
  --structure-lora packages/ltx-trainer/outputs/v2v_reference_ic_lora/checkpoints/lora_weights_step_08000.safetensors \
  --output out.mp4
```

`setup.sh` downloads gated/large weights from HuggingFace — if a download
401s, run `huggingface-cli login` (or set `HF_TOKEN`) first.

## How it works

### Input video → output video

The source video is never fed to the transformer as raw pixels through a
special "V2V" code path — there isn't one. It goes through the exact same
`video_conditioning` mechanism IC-LoRA uses for any reference signal
(`ICLoraPipeline.__call__`'s `video_conditioning: list[tuple[path, strength]]`
parameter, in `packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py`): it's
VAE-encoded, patchified into tokens exactly like the noisy video latent, and
**appended to the same self-attention token sequence** the denoiser attends
over (`VideoConditionByReferenceLatent`,
`packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py`).
A LoRA (`--structure-lora`) is what actually teaches the model to *use* those
extra tokens meaningfully — the base model has no opinion on what an arbitrary
appended video means.

Important: `reference_video`/`--input-video` is not necessarily raw source
footage — it's whatever structure/motion signal your LoRA was trained to
condition on (pose render, edge map, or the raw video itself, if that's what
you trained with). See `data/README.md` → "About `reference_video`".

### Two ways to position-encode the reference image

This is the part that's easy to get subtly wrong, so it's exposed as an
explicit `--pos-mode` flag in `scripts/infer_v2v.py` rather than being baked in
silently:

**`--pos-mode first-frame` (default)** — the reference image **reuses** the
denoised video's own frame-0 RoPE position. Implemented via `ICLoraPipeline`'s
native `images: list[(path, frame_idx, strength)]` parameter → this is a
**base-model capability, no LoRA required**. Code path:
`combined_image_conditionings()` in
`packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py` →
`VideoConditionByLatentIndex` / `VideoConditionByKeyframeIndex` in
`packages/ltx-core/src/ltx_core/conditioning/types/`. Because the reference
image is placed at the exact same absolute time position as frame 0, RoPE's
relative-rotation attention naturally treats it as "this is what frame 0 looks
like" rather than as a separate signal.

**`--pos-mode reference`** — the reference image gets its **own, disjoint**
RoPE position range instead of sharing frame 0's. Implemented via
`video_conditioning` → `VideoConditionByReferenceLatent`
(`packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py`).
This is the path our custom IC-LoRAs (`configs/ref_image_ic_lora.yaml`,
`configs/v2v_reference_ic_lora.yaml`) are actually trained against, and is
generally the better-performing option **once you have a matching LoRA** — the
base model alone doesn't know what to do with a disjoint-position reference
block; it has to be taught.

Why "disjoint" instead of just also using t=0: giving the reference image the
same absolute position as the target's frame 0 means RoPE's relative rotation
between them is **zero** — the attention mechanism can't tell the reference
token and the target's own frame-0 token apart by position alone. The fix
(originally a local patch on top of upstream LTX-2, in
`reference_video_cond.py`):

```python
# packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py
REF_POSITION_TICK_SECONDS = 1e-3
current_min_time = latent_state.positions[:, 0, ...].min()
ref_max_time = positions[:, 0, ...].max()
positions[:, 0, ...] = positions[:, 0, ...] - ref_max_time + current_min_time - REF_POSITION_TICK_SECONDS
```

This shifts the whole reference block to end `1ms` **before** whatever's
already the earliest time position in the sequence — a strictly ordered,
non-overlapping range. The training side
(`packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py`,
`_apply_reference_condition`) applies the **identical** tick-offset logic, so a
LoRA trained with this code sees the same position layout at train and
inference time. If you ever modify one side, modify the other — they must stay
in parity.

When two `reference`-type conditions are stacked (as in
`configs/v2v_reference_ic_lora.yaml`, which uses both `reference_video` and
`reference_image`), each additional block is slotted immediately before
whatever range is already occupied, so every block — target frames, structure
reference, appearance reference — ends up in its own strictly-ordered slice of
time, none of them aliasing.

### Conditioning strength (separate from position mode)

Two independent knobs control how strongly a reference item affects the
output, orthogonal to which position mode you use:

- **`--ref-strength` / `--structure-strength`** (denoise strength, `strength`
  in `ConditioningItem`): controls how much noise the conditioning latent
  itself gets mixed with during denoising (`GaussianNoiser.torch.lerp`, in
  `packages/ltx-core/src/ltx_core/components/noisers.py`). `1.0` = the
  reference stays exactly at its clean VAE-encoded value; lower values let the
  model deviate from it.
- **`--conditioning-attention-strength`**: scales how strongly conditioning
  tokens attend to/from the noisy tokens in self-attention, independent of the
  noise blending above.

## Repository layout

```
setup.sh                        one-click env + weights setup
pyproject.toml, uv.lock         workspace definition (uv)
packages/
  ltx-core/                     LTX-2 model + conditioning + RoPE (patched)
  ltx-pipelines/                ICLoraPipeline and friends
  ltx-trainer/                  LoRA / IC-LoRA training stack (patched)
configs/
  v2v_reference_ic_lora.yaml    train on (video, reference_video, reference_image, caption)
  ref_image_ic_lora.yaml        simpler: train on (video, reference_image, caption) only
  accelerate/                   accelerate configs referenced by the trainer docs
scripts/
  train_ic_lora.sh              resolves __REPO_ROOT__ in a config, launches training
  preprocess_dataset.sh         wraps packages/ltx-trainer/scripts/process_dataset.py
  infer_v2v.py                  video + reference image -> video, --pos-mode switch
data/
  README.md                     dataset JSONL schema, resolution buckets, etc.
  dataset.jsonl                 example rows: video + reference_video + reference_image + caption
  dataset_image_only.jsonl      example rows: video + reference_image + caption
weights/                        populated by setup.sh (gitignored — large binaries)
```

## Which config to train

- Have paired (source video, structure signal, reference image) data? Use
  `configs/v2v_reference_ic_lora.yaml` — trains one LoRA that uses both
  `--input-video` and `--ref-image` at inference.
- Only have (video, reference image) pairs, no structure video? Use
  `configs/ref_image_ic_lora.yaml` — simpler, doesn't need `--input-video` at
  all (pass an empty/no-op structure conditioning, or adapt
  `scripts/infer_v2v.py` to drop `--structure-lora`/`--input-video` for this
  case).

See `data/README.md` for the exact JSONL schema, column-to-encoder mapping, and
resolution-bucket rules.

## Known limitations / things we couldn't fully verify

- We could not confirm from the LTX-2 repo alone whether the public
  `LTX-2.3-22b-IC-LoRA-Union-Control` LoRA expects raw source video or a
  preprocessed control map (depth/pose/edge) for its `video_conditioning`
  input — its model card (not vendored here) would be the source of truth if
  you want to use it instead of training your own structure-control LoRA.
- `reference_video`/`--input-video` preprocessing (pose/edge/depth extraction)
  is out of scope for this repo — bring your own, or use the basic Canny-edge
  `compute_reference()` in `packages/ltx-trainer/scripts/process_videos.py` as
  a starting point.