| # Dataset format |
|
|
| Training data is a JSONL file: one JSON object per row, each row is one training |
| clip. `scripts/process_dataset.py` (in `packages/ltx-trainer/scripts/`) auto-detects |
| columns by name — no per-role CLI flags needed. Recognized columns: |
|
|
| | Column | Encoded by | Output dir | Meaning | |
| |--------------------|---------------|---------------------------|-------------------------------------------------------| |
| | `video` | Video VAE | `latents/` | Ground-truth target video (what the model learns to generate) | |
| | `reference_video` | Video VAE | `reference_latents/` | Structure/motion control signal (see note below) | |
| | `reference_image` | Video VAE | `reference_image_latents/` | Single PNG/JPG, encoded as a 1-frame latent — identity/appearance anchor | |
| | `reference_audio` | Audio VAE | `reference_audio_latents/` | Not used by the configs in this repo, listed for completeness | |
| | `caption` | Text encoder | `conditions/` | Text prompt | |
|
|
| Two example files are provided: |
|
|
| - **`dataset.jsonl`** — full schema (`video` + `reference_video` + `reference_image` + `caption`), |
| used by `configs/v2v_reference_ic_lora.yaml`. Trains a LoRA that learns to use both |
| an input/structure video AND a reference image at once. |
| - **`dataset_image_only.jsonl`** — simplified schema (`video` + `reference_image` + `caption`, |
| no `reference_video`), used by `configs/ref_image_ic_lora.yaml`. Use this if you |
| don't have a paired structure-control video for your clips. |
|
|
| Row example (`dataset.jsonl`): |
|
|
| ```json |
| { |
| "video": "data/videos/sample_001/target.mp4", |
| "reference_video": "data/videos/sample_001/structure_control.mp4", |
| "reference_image": "data/videos/sample_001/ref_keyframe.png", |
| "caption": "A woman with dark hair in a white dress walks through a garden..." |
| } |
| ``` |
|
|
| Paths can be relative (resolved from wherever you invoke `process_dataset.py` — |
| we always invoke it with an absolute path via `scripts/train_ic_lora.sh` / |
| `scripts/preprocess_dataset.sh`, see the repo README) or absolute. |
|
|
| ## About `reference_video` ("structure control") |
| |
| `reference_video` is **not** the raw source footage — it's whatever |
| structure/motion signal you want the model to condition on and preserve (e.g. a |
| pose-skeleton render, an edge/Canny map, a depth map, or a custom "composite" |
| render). The pipeline mechanically just VAE-encodes whatever video file you give |
| it under this column; it does not know or care what's actually in the pixels. |
| What matters is that the SAME kind of preprocessing is applied consistently at |
| both training time (this column) and inference time (`--input-video` in |
| `scripts/infer_v2v.py`). |
|
|
| `packages/ltx-trainer/scripts/process_videos.py` ships a basic Canny-edge |
| `compute_reference()` you can use out of the box if you have no better option. |
| For anything more specific (pose skeletons, depth, a custom composite), you need |
| to generate that reference video yourself and point `reference_video` at it — |
| this repo does not include a pose/depth extractor. |
|
|
| ## Resolution / frame-count buckets |
|
|
| `process_dataset.py --resolution-buckets WxHxF` must match your actual clip |
| dimensions (or clips get filtered/skipped). The example configs use |
| `1920x1024x233` (233 = 8×29+1, a valid VAE-causal frame count). If your source |
| clips are a different size, adjust `--resolution-buckets` accordingly — see |
| `packages/ltx-trainer/docs/dataset-preparation.md` for the full bucketing rules. |
|
|
| ## Preprocessing output |
|
|
| `process_dataset.py --output-dir <dir>` produces `<dir>/latents/`, |
| `<dir>/reference_latents/`, `<dir>/reference_image_latents/`, |
| `<dir>/conditions/` — this `<dir>` is what you point `data.preprocessed_data_root` |
| at in the training config (already wired up in `configs/*.yaml` via |
| `__REPO_ROOT__/data/preprocessed`). |
|
|
