data/README.md

Dataset format

Training data is a JSONL file: one JSON object per row, each row is one training clip. scripts/process_dataset.py (in packages/ltx-trainer/scripts/) auto-detects columns by name — no per-role CLI flags needed. Recognized columns:

Column	Encoded by	Output dir	Meaning
video	Video VAE	latents/	Ground-truth target video (what the model learns to generate)
reference_video	Video VAE	reference_latents/	Structure/motion control signal (see note below)
reference_image	Video VAE	reference_image_latents/	Single PNG/JPG, encoded as a 1-frame latent — identity/appearance anchor
reference_audio	Audio VAE	reference_audio_latents/	Not used by the configs in this repo, listed for completeness
caption	Text encoder	conditions/	Text prompt

Two example files are provided:

• dataset.jsonl — full schema (video + reference_video + reference_image + caption), used by configs/v2v_reference_ic_lora.yaml. Trains a LoRA that learns to use both an input/structure video AND a reference image at once.
• dataset_image_only.jsonl — simplified schema (video + reference_image + caption, no reference_video), used by configs/ref_image_ic_lora.yaml. Use this if you don't have a paired structure-control video for your clips.

Row example (dataset.jsonl):

{
  "video": "data/videos/sample_001/target.mp4",
  "reference_video": "data/videos/sample_001/structure_control.mp4",
  "reference_image": "data/videos/sample_001/ref_keyframe.png",
  "caption": "A woman with dark hair in a white dress walks through a garden..."
}

Paths can be relative (resolved from wherever you invoke process_dataset.py — we always invoke it with an absolute path via scripts/train_ic_lora.sh / scripts/preprocess_dataset.sh, see the repo README) or absolute.

About reference_video ("structure control")

reference_video is not the raw source footage — it's whatever structure/motion signal you want the model to condition on and preserve (e.g. a pose-skeleton render, an edge/Canny map, a depth map, or a custom "composite" render). The pipeline mechanically just VAE-encodes whatever video file you give it under this column; it does not know or care what's actually in the pixels. What matters is that the SAME kind of preprocessing is applied consistently at both training time (this column) and inference time (--input-video in scripts/infer_v2v.py).

packages/ltx-trainer/scripts/process_videos.py ships a basic Canny-edge compute_reference() you can use out of the box if you have no better option. For anything more specific (pose skeletons, depth, a custom composite), you need to generate that reference video yourself and point reference_video at it — this repo does not include a pose/depth extractor.

Resolution / frame-count buckets

process_dataset.py --resolution-buckets WxHxF must match your actual clip dimensions (or clips get filtered/skipped). The example configs use 1920x1024x233 (233 = 8×29+1, a valid VAE-causal frame count). If your source clips are a different size, adjust --resolution-buckets accordingly — see packages/ltx-trainer/docs/dataset-preparation.md for the full bucketing rules.

Preprocessing output

process_dataset.py --output-dir <dir> produces <dir>/latents/, <dir>/reference_latents/, <dir>/reference_image_latents/, <dir>/conditions/ — this <dir> is what you point data.preprocessed_data_root at in the training config (already wired up in configs/*.yaml via __REPO_ROOT__/data/preprocessed).