File size: 9,399 Bytes
be7e4b7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 | # Utility Scripts Reference
This guide covers the various utility scripts available for preprocessing, conversion, and debugging tasks.
## 🎬 Dataset Processing Scripts
### Video Scene Splitting
The `scripts/split_scenes.py` script automatically splits long videos into shorter, coherent scenes.
```bash
# Basic scene splitting
uv run python scripts/split_scenes.py input.mp4 output_dir/ --filter-shorter-than 5s
```
**Key features:**
- **Automatic scene detection**: Uses PySceneDetect for intelligent splitting
- **Multiple algorithms**: Content-based, adaptive, threshold, and histogram detection
- **Filtering options**: Remove scenes shorter than specified duration
- **Customizable parameters**: Thresholds, window sizes, and detection modes
**Common options:**
```bash
# See all available options
uv run python scripts/split_scenes.py --help
# Use adaptive detection with custom threshold
uv run python scripts/split_scenes.py video.mp4 scenes/ --detector adaptive --threshold 30.0
# Limit to maximum number of scenes
uv run python scripts/split_scenes.py video.mp4 scenes/ --max-scenes 50
```
### Automatic Video Captioning
The `scripts/caption_videos.py` script generates a single, detailed combined audio-visual
caption per video as a continuous paragraph of prose. Two backends are available:
- **`qwen_omni` (default)** — Qwen3-Omni-30B-A3B-Thinking served via a local
[vLLM](https://docs.vllm.ai/) HTTP server (~1-3 s/video on H100). Highest quality, runs
fully offline once the model is downloaded.
- **`gemini_flash`** — Google Gemini (cloud, `gemini-3.5-flash`). No GPU required. Auth is
automatic: set `GEMINI_API_KEY` (or `GOOGLE_API_KEY`) for the Developer API, or just have
Google Cloud credentials available (`gcloud auth` / an attached service account) and it
uses Vertex AI with no extra setup.
**Step 1 — launch the captioner server** (`qwen_omni` only, one-time).
`scripts/serve_captioner.py` runs vLLM in an isolated environment via `uvx`, so vLLM's heavy
CUDA dependencies never touch the trainer's venv. It defaults to dynamic FP8 quantization
(~31 GiB weights, fits on 40 GB GPUs, same speed as BF16 on H100):
```bash
# Terminal 1 - stays running
uv run python packages/ltx-trainer/scripts/serve_captioner.py
# Useful variants:
# --print-cmd show the vLLM command without running it
# --quantization bf16 use BF16 instead (needs ~66 GiB free VRAM)
# --hf-home /mnt/disk override where the ~65 GB model is downloaded
```
**Step 2 — caption your videos.**
```bash
# Terminal 2 - default backend talks to the server above
uv run python packages/ltx-trainer/scripts/caption_videos.py videos_dir/ --output dataset.json
# Remote server: --vllm-url http://other-host:8001/v1
# Gemini (gemini-3.5-flash): --captioner-type gemini_flash (uses GEMINI_API_KEY, else gcloud/Vertex)
# Gemini, parallel calls: --captioner-type gemini_flash --num-workers 5
# Re-caption everything: --override
```
Captioning is incremental (already-captioned files are skipped, progress saves every 5 videos)
and writes JSON, JSONL, CSV, or TXT based on the output extension.
Qwen3-Omni-Thinking can optionally emit a `<think>...</think>` chain-of-thought before the
caption (`--enable-thinking`). It is off by default, which is recommended for bulk captioning
(thinking is slower as it generates the reasoning trace first).
For Gemini, keep `--num-workers` at 3-5 (higher values may hit API rate limits).
### Dataset Preprocessing
The `scripts/process_dataset.py` script processes videos and caches latents for training.
```bash
# Basic preprocessing
uv run python scripts/process_dataset.py dataset.json \
--resolution-buckets "960x544x49" \
--model-path /path/to/ltx-2-model.safetensors \
--text-encoder-path /path/to/gemma-model
# With video decoding for verification
uv run python scripts/process_dataset.py dataset.json \
--resolution-buckets "960x544x49" \
--model-path /path/to/ltx-2-model.safetensors \
--text-encoder-path /path/to/gemma-model \
--decode
```
Multiple resolution buckets can be specified, separated by `;`:
```bash
uv run python scripts/process_dataset.py dataset.json \
--resolution-buckets "960x544x49;512x512x81" \
--model-path /path/to/ltx-2-model.safetensors \
--text-encoder-path /path/to/gemma-model
```
> [!NOTE]
> When training with multiple resolution buckets, set `optimization.batch_size: 1`.
**Multi-GPU preprocessing.** Launch with `accelerate launch` to shard the dataset across processes. Reruns resume
by default (existing `.pt` outputs are skipped); writes are atomic so interrupted runs are safe. Pass `--overwrite`
when rerunning with changed parameters (different model, resolution buckets, text encoder, `--lora-trigger`, etc.)
so stale outputs are replaced. Use the same `accelerate launch` pattern (and `--overwrite` when needed) with
`process_videos.py` or `process_captions.py` when you run those scripts standalone.
```bash
# Multi-GPU preprocessing
uv run accelerate launch --num_processes 4 scripts/process_dataset.py dataset.json \
--resolution-buckets "960x544x49" \
--model-path /path/to/ltx-2-model.safetensors \
--text-encoder-path /path/to/gemma-model
# Force re-encoding of all items (e.g. after switching model or resolution)
uv run accelerate launch --num_processes 4 scripts/process_dataset.py dataset.json \
--resolution-buckets "960x544x49" \
--model-path /path/to/ltx-2.3-model.safetensors \
--text-encoder-path /path/to/gemma-model \
--overwrite
```
For detailed usage, see the [Dataset Preparation Guide](dataset-preparation.md).
### Reference Video Generation
The `scripts/compute_reference.py` script provides a template for creating reference videos needed for IC-LoRA training.
The default implementation generates Canny edge reference videos.
```bash
# Generate Canny edge reference videos
uv run python scripts/compute_reference.py videos_dir/ --output dataset.json
```
**Key features:**
- **Canny edge detection**: Creates edge-based reference videos
- **In-place editing**: Updates existing dataset JSON files
- **Customizable**: Modify the `compute_reference()` function for different conditions (depth, pose, etc.)
> [!TIP]
> You can edit this script to generate other types of reference videos for IC-LoRA training,
> such as depth maps, segmentation masks, or any custom video transformation.
> [!NOTE]
> `compute_reference.py` writes generated references to the `reference_video` column, which
> `process_dataset.py` detects automatically.
## 🔍 Debugging and Verification Scripts
### Latents Decoding
The `scripts/decode_latents.py` script decodes precomputed video latents back into video files for visual inspection.
```bash
# Basic usage
uv run python scripts/decode_latents.py /path/to/latents/dir \
--output-dir /path/to/output \
--model-path /path/to/ltx-2-model.safetensors
# With VAE tiling for large videos
uv run python scripts/decode_latents.py /path/to/latents/dir \
--output-dir /path/to/output \
--model-path /path/to/ltx-2-model.safetensors \
--vae-tiling
# Decode both video and audio latents
uv run python scripts/decode_latents.py /path/to/latents/dir \
--output-dir /path/to/output \
--model-path /path/to/ltx-2-model.safetensors \
--with-audio
```
**The script will:**
1. **Load the VAE model** from the specified path
2. **Process all `.pt` latent files** in the input directory
3. **Decode each latent** back into a video using the VAE
4. **Save resulting videos** as MP4 files in the output directory
**When to use:**
- **Verify preprocessing quality**: Check that your videos were encoded correctly
- **Debug training data**: Visualize what the model actually sees during training
- **Quality assessment**: Ensure latent encoding preserves important visual details
### Inference with Trained Models
For inference with trained LoRAs, use the [`ltx-pipelines`](../../ltx-pipelines/) package which provides
production-ready pipelines:
- **Text/Image-to-Video**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`
- **Distilled (fast) inference**: `DistilledPipeline`
- **IC-LoRA video-to-video**: `ICLoraPipeline`
- **Keyframe interpolation**: `KeyframeInterpolationPipeline`
All pipelines support loading custom LoRAs trained with this trainer.
## 🚀 Training Scripts
### Basic and Distributed Training
Use `scripts/train.py` for both single GPU and multi-GPU runs:
```bash
# Single-GPU training
uv run python scripts/train.py configs/t2v_lora.yaml
# Multi-GPU (uses your accelerate config)
uv run accelerate launch scripts/train.py configs/t2v_lora.yaml
# Override number of processes
uv run accelerate launch --num_processes 4 scripts/train.py configs/t2v_lora.yaml
```
For detailed usage, see the [Training Guide](training-guide.md).
## 💡 Tips for Using Utility Scripts
- **Start with `--help`**: Always check available options for each script
- **Test on small datasets**: Verify workflows with a few files before processing large datasets
- **Use decode verification**: Always decode a few samples to verify preprocessing quality
- **Monitor VRAM usage**: Reach for quantization or lower-memory settings (e.g. FP8 for the captioner server) when running into memory issues
- **Keep backups**: Make copies of important dataset files before running conversion scripts
|