File size: 3,918 Bytes
be7e4b7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | # Dataset format
Training data is a JSONL file: one JSON object per row, each row is one training
clip. `scripts/process_dataset.py` (in `packages/ltx-trainer/scripts/`) auto-detects
columns by name — no per-role CLI flags needed. Recognized columns:
| Column | Encoded by | Output dir | Meaning |
|--------------------|---------------|---------------------------|-------------------------------------------------------|
| `video` | Video VAE | `latents/` | Ground-truth target video (what the model learns to generate) |
| `reference_video` | Video VAE | `reference_latents/` | Structure/motion control signal (see note below) |
| `reference_image` | Video VAE | `reference_image_latents/` | Single PNG/JPG, encoded as a 1-frame latent — identity/appearance anchor |
| `reference_audio` | Audio VAE | `reference_audio_latents/` | Not used by the configs in this repo, listed for completeness |
| `caption` | Text encoder | `conditions/` | Text prompt |
Two example files are provided:
- **`dataset.jsonl`** — full schema (`video` + `reference_video` + `reference_image` + `caption`),
used by `configs/v2v_reference_ic_lora.yaml`. Trains a LoRA that learns to use both
an input/structure video AND a reference image at once.
- **`dataset_image_only.jsonl`** — simplified schema (`video` + `reference_image` + `caption`,
no `reference_video`), used by `configs/ref_image_ic_lora.yaml`. Use this if you
don't have a paired structure-control video for your clips.
Row example (`dataset.jsonl`):
```json
{
"video": "data/videos/sample_001/target.mp4",
"reference_video": "data/videos/sample_001/structure_control.mp4",
"reference_image": "data/videos/sample_001/ref_keyframe.png",
"caption": "A woman with dark hair in a white dress walks through a garden..."
}
```
Paths can be relative (resolved from wherever you invoke `process_dataset.py` —
we always invoke it with an absolute path via `scripts/train_ic_lora.sh` /
`scripts/preprocess_dataset.sh`, see the repo README) or absolute.
## About `reference_video` ("structure control")
`reference_video` is **not** the raw source footage — it's whatever
structure/motion signal you want the model to condition on and preserve (e.g. a
pose-skeleton render, an edge/Canny map, a depth map, or a custom "composite"
render). The pipeline mechanically just VAE-encodes whatever video file you give
it under this column; it does not know or care what's actually in the pixels.
What matters is that the SAME kind of preprocessing is applied consistently at
both training time (this column) and inference time (`--input-video` in
`scripts/infer_v2v.py`).
`packages/ltx-trainer/scripts/process_videos.py` ships a basic Canny-edge
`compute_reference()` you can use out of the box if you have no better option.
For anything more specific (pose skeletons, depth, a custom composite), you need
to generate that reference video yourself and point `reference_video` at it —
this repo does not include a pose/depth extractor.
## Resolution / frame-count buckets
`process_dataset.py --resolution-buckets WxHxF` must match your actual clip
dimensions (or clips get filtered/skipped). The example configs use
`1920x1024x233` (233 = 8×29+1, a valid VAE-causal frame count). If your source
clips are a different size, adjust `--resolution-buckets` accordingly — see
`packages/ltx-trainer/docs/dataset-preparation.md` for the full bucketing rules.
## Preprocessing output
`process_dataset.py --output-dir <dir>` produces `<dir>/latents/`,
`<dir>/reference_latents/`, `<dir>/reference_image_latents/`,
`<dir>/conditions/` — this `<dir>` is what you point `data.preprocessed_data_root`
at in the training config (already wired up in `configs/*.yaml` via
`__REPO_ROOT__/data/preprocessed`).
|