packages/ltx-trainer/docs/configuration-reference.md

# Configuration Reference

The trainer uses structured Pydantic models for configuration, making it easy to customize training parameters.
This guide covers all available configuration options and their usage.

## 📋 Overview

The main configuration class is [`LtxTrainerConfig`](../src/ltx_trainer/config.py), which includes the following
sub-configurations:

- **ModelConfig**: Base model and training mode settings
- **LoraConfig**: LoRA training parameters
- **TrainingStrategyConfig**: Training strategy settings (flexible conditioning framework)
- **OptimizationConfig**: Learning rate, batch sizes, and scheduler settings
- **AccelerationConfig**: Mixed precision and quantization settings
- **DataConfig**: Data loading parameters
- **ValidationConfig**: Validation and inference settings
- **CheckpointsConfig**: Checkpoint saving frequency and retention settings
- **HubConfig**: Hugging Face Hub integration settings
- **WandbConfig**: Weights & Biases logging settings
- **FlowMatchingConfig**: Timestep sampling parameters

## 📄 Example Configuration Files

Check out our example configurations in the `configs` directory:

- 📄 [Text-to-Video LoRA](../configs/t2v_lora.yaml) - Text-to-video LoRA training
- 📄 [Image-to-Video LoRA](../configs/i2v_lora.yaml) - Image-to-video LoRA training
- 📄 [IC-LoRA Video-to-Video](../configs/v2v_ic_lora.yaml) - IC-LoRA video-to-video training
- 📄 [Audio-to-Video LoRA](../configs/a2v_lora.yaml) - Audio-to-video LoRA training
- 📄 [Video-to-Audio LoRA](../configs/v2a_lora.yaml) - Video-to-audio (Foley) LoRA training
- 📄 [Video Extension LoRA](../configs/video_extend_lora.yaml) - Video extension (forward) LoRA training
- 📄 [Video Suffix LoRA](../configs/video_suffix_lora.yaml) - Video extension (backward) LoRA training
- 📄 [Video Inpainting LoRA](../configs/video_inpainting_lora.yaml) - Video inpainting LoRA training
- 📄 [Video Outpainting LoRA](../configs/video_outpainting_lora.yaml) - Video outpainting (spatial crop) LoRA training
- 📄 [Text-to-Audio LoRA](../configs/t2a_lora.yaml) - Text-to-audio LoRA training
- 📄 [Audio Extension LoRA](../configs/audio_extend_lora.yaml) - Audio extension (forward) LoRA training
- 📄 [Audio Suffix LoRA](../configs/audio_suffix_lora.yaml) - Audio extension (backward) LoRA training
- 📄 [Audio Inpainting LoRA](../configs/audio_inpainting_lora.yaml) - Audio inpainting LoRA training
- 📄 [Audio-to-Audio IC-LoRA](../configs/a2a_ic_lora.yaml) - Audio IC-LoRA transformation training
- 📄 [AV2AV IC-LoRA](../configs/av2av_ic_lora.yaml) - Audio+video IC-LoRA transformation training
- 📄 [T2V LoRA (Low VRAM)](../configs/t2v_lora_low_vram.yaml) - Memory-optimized config for 32GB GPUs

## ⚙️ Configuration Sections

> [!NOTE]
> The YAML snippets below show **recommended starting values**, not necessarily the code defaults.
> Fields you omit from your config file will use the code defaults from [`config.py`](../src/ltx_trainer/config.py).

### ModelConfig

Controls the base model and training mode settings.

```yaml
model:
  model_path: "/path/to/ltx-2-model.safetensors"  # Local path to model checkpoint
  text_encoder_path: "/path/to/gemma-model"       # Path to Gemma text encoder directory
  training_mode: "lora"                           # "lora" or "full"
  load_checkpoint: null                           # Path to checkpoint to resume from
```

**Key parameters:**

| Parameter           | Description                                                                                                                                                    |
|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `model_path`        | **Required.** Local path to the LTX-2 model checkpoint (`.safetensors` file). URLs are not supported.                                                          |
| `text_encoder_path` | **Required.** Path to the Gemma text encoder model directory. Download from [HuggingFace](https://huggingface.co/google/gemma-3-12b-it-qat-q4_0-unquantized/). |
| `training_mode`     | Training approach - `"lora"` for LoRA training or `"full"` for full-rank fine-tuning.                                                                          |
| `load_checkpoint`   | Optional path to resume training from a checkpoint file or directory.                                                                                          |

> [!NOTE]
> LTX-2 requires both a model checkpoint and a Gemma text encoder. Both must be local paths.

### LoraConfig

LoRA-specific fine-tuning parameters (only used when `training_mode: "lora"`).

```yaml
lora:
  rank: 32         # LoRA rank (higher = more parameters)
  alpha: 32        # LoRA alpha scaling factor
  dropout: 0.0     # Dropout probability (0.0-1.0)
  target_modules: # Modules to apply LoRA to
    - "to_k"
    - "to_q"
    - "to_v"
    - "to_out.0"
```

**Key parameters:**

| Parameter        | Description                                                                     |
|------------------|---------------------------------------------------------------------------------|
| `rank`           | LoRA rank - higher values mean more trainable parameters (typical range: 8-128) |
| `alpha`          | Alpha scaling factor - typically set equal to rank                              |
| `dropout`        | Dropout probability for regularization                                          |
| `target_modules` | List of transformer modules to apply LoRA adapters to (see below)               |

#### Understanding Target Modules

The LTX-2 transformer has separate attention and feed-forward blocks for video and audio, as well as cross-attention
modules that enable the two modalities to exchange information. Choosing the right `target_modules` is critical for
achieving good results, especially when training with audio.

**Video-only modules:**

| Module Pattern                                             | Description                     |
|------------------------------------------------------------|---------------------------------|
| `attn1.to_k`, `attn1.to_q`, `attn1.to_v`, `attn1.to_out.0` | Video self-attention            |
| `attn2.to_k`, `attn2.to_q`, `attn2.to_v`, `attn2.to_out.0` | Video cross-attention (to text) |
| `ff.net.0.proj`, `ff.net.2`                                | Video feed-forward network      |

**Audio-only modules:**

| Module Pattern                                                                     | Description                     |
|------------------------------------------------------------------------------------|---------------------------------|
| `audio_attn1.to_k`, `audio_attn1.to_q`, `audio_attn1.to_v`, `audio_attn1.to_out.0` | Audio self-attention            |
| `audio_attn2.to_k`, `audio_attn2.to_q`, `audio_attn2.to_v`, `audio_attn2.to_out.0` | Audio cross-attention (to text) |
| `audio_ff.net.0.proj`, `audio_ff.net.2`                                            | Audio feed-forward network      |

**Audio-video cross-attention modules:**

These modules enable bidirectional information flow between the audio and video modalities:

| Module Pattern                                                                                                     | Description                                           |
|--------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|
| `audio_to_video_attn.to_k`, `audio_to_video_attn.to_q`, `audio_to_video_attn.to_v`, `audio_to_video_attn.to_out.0` | Video attends to audio (Q from video, K/V from audio) |
| `video_to_audio_attn.to_k`, `video_to_audio_attn.to_q`, `video_to_audio_attn.to_v`, `video_to_audio_attn.to_out.0` | Audio attends to video (Q from audio, K/V from video) |

**Recommended configurations:**

For **video-only training**, target the video attention layers:

```yaml
target_modules:
  - "attn1.to_k"
  - "attn1.to_q"
  - "attn1.to_v"
  - "attn1.to_out.0"
  - "attn2.to_k"
  - "attn2.to_q"
  - "attn2.to_v"
  - "attn2.to_out.0"
```

For **audio-video training**, use patterns that match both branches:

```yaml
target_modules:
  - "to_k"
  - "to_q"
  - "to_v"
  - "to_out.0"
```

> [!NOTE]
> Using shorter patterns like `"to_k"` will match all attention modules including `attn1.to_k`, `audio_attn1.to_k`,
> `audio_to_video_attn.to_k`, and `video_to_audio_attn.to_k`, effectively training video, audio, and cross-modal
> attention branches together.

> [!TIP]
> You can also target the feed-forward (FFN) modules (`ff.net.0.proj`, `ff.net.2` for video,
> `audio_ff.net.0.proj`, `audio_ff.net.2` for audio) to increase the LoRA's capacity and potentially
> help it capture the target distribution better.

### TrainingStrategyConfig

Configures the training strategy. The recommended strategy is `"flexible"`, which supports all conditioning scenarios through configuration.

#### Flexible Strategy

The flexible strategy provides a unified conditioning framework. Each modality (video, audio) is configured
independently with its own latents directory, generation flag, and list of conditions.

```yaml
training_strategy:
  name: "flexible"
  video:
    is_generated: true                 # Video is denoised during training
    latents_dir: "latents"             # Directory containing precomputed video latents
    conditions:
      - type: first_frame              # Use first frame as conditioning
        probability: 0.5               # Apply this condition 50% of the time
  audio:
    is_generated: true                 # Audio is denoised during training
    latents_dir: "audio_latents"       # Directory containing precomputed audio latents
    conditions: []                     # No additional audio conditions (text-only)
```

**ModalityConfig parameters:**

| Parameter      | Description                                                                                                      |
|----------------|------------------------------------------------------------------------------------------------------------------|
| `is_generated` | `true` = modality is denoised (contributes to loss). `false` = frozen conditioning (sigma=0, no loss).           |
| `latents_dir`  | Directory name within `preprocessed_data_root` containing precomputed latents for this modality.                 |
| `conditions`   | List of conditioning configs applied during training (see condition types below). Text conditioning is implicit. |

**Condition types:**

| Type           | Parameters                                          | Description                                                                           |
|----------------|-----------------------------------------------------|---------------------------------------------------------------------------------------|
| `first_frame`  | `probability`                                       | First latent frame is clean, excluded from loss. **Video only.**                      |
| `prefix`       | `temporal_boundary`, `probability`                  | First N latent temporal units are clean. For extension forward.                       |
| `suffix`       | `temporal_boundary`, `probability`                  | Last N latent temporal units are clean. For extension backward.                       |
| `spatial_crop` | `spatial_region` (y1, x1, y2, x2 in px), `probability` | Rectangular region is clean, excluded from loss. For outpainting. **Video only.** |
| `mask`         | `mask_dir`, `probability`                           | Per-sample float mask [0,1] from directory. For inpainting. Supports soft conditioning. |
| `reference`    | `latents_dir`, `probability`                        | IC-LoRA style concatenation. Reference tokens are prepended, clean (timestep=0), no loss. |

> [!NOTE]
> The `prefix`, `suffix`, `mask`, and `reference` condition types work on both video and audio modalities —
> place them in the `video.conditions` or `audio.conditions` list as appropriate.
> `first_frame` and `spatial_crop` are video-only conditions.

> [!NOTE]
> Training conditions reference **directories** of precomputed data (within `preprocessed_data_root`),
> while validation conditions reference **individual files** (images, videos, masks) that are encoded
> on-the-fly during validation. The condition `type` names are the same, but the fields differ.

> [!NOTE]
> The legacy `text_to_video` and `video_to_video` strategies are deprecated but remain forward-compatible.
> New configs should use `name: "flexible"`.

### OptimizationConfig

Training optimization parameters including learning rates, batch sizes, and schedulers.

```yaml
optimization:
  learning_rate: 1e-4                  # Learning rate
  steps: 2000                          # Total training steps
  batch_size: 1                        # Batch size per GPU
  gradient_accumulation_steps: 1       # Steps to accumulate gradients
  max_grad_norm: 1.0                   # Gradient clipping threshold
  optimizer_type: "adamw"              # "adamw" or "adamw8bit"
  scheduler_type: "linear"             # Scheduler type
  scheduler_params: { }                # Additional scheduler parameters
  enable_gradient_checkpointing: true  # Memory optimization
```

**Key parameters:**

| Parameter                       | Description                                                                                  |
|---------------------------------|----------------------------------------------------------------------------------------------|
| `learning_rate`                 | Learning rate for optimization (typical range: 1e-5 to 1e-3)                                 |
| `steps`                         | Total number of training steps                                                               |
| `batch_size`                    | Batch size per GPU (reduce if running out of memory)                                         |
| `gradient_accumulation_steps`   | Accumulate gradients over multiple steps                                                     |
| `scheduler_type`                | LR scheduler: `"constant"`, `"linear"`, `"cosine"`, `"cosine_with_restarts"`, `"polynomial"`, `"step"` |
| `enable_gradient_checkpointing` | Trade training speed for GPU memory savings (recommended for large models)                   |

### AccelerationConfig

Hardware acceleration and compute optimization settings.

```yaml
acceleration:
  mixed_precision_mode: "bf16"                  # "no", "fp16", or "bf16"
  quantization: null                            # Quantization options
  load_text_encoder_in_8bit: false              # Load text encoder in 8-bit
  offload_optimizer_during_validation: false    # Offload optimizer state to CPU during validation
```

**Key parameters:**

| Parameter                             | Description                                                                                                                                                                              |
|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `mixed_precision_mode`                | Precision mode - `"bf16"` recommended for modern GPUs                                                                                                                                    |
| `quantization`                        | Model quantization: `null`, `"int8-quanto"`, `"int4-quanto"`, `"int2-quanto"`, `"fp8-quanto"`, or `"fp8uz-quanto"`                                                                       |
| `load_text_encoder_in_8bit`           | Load the Gemma text encoder in 8-bit to save GPU memory                                                                                                                                  |
| `offload_optimizer_during_validation` | Move optimizer state to CPU before validation video sampling and back afterwards. Useful when validation OOMs because VAE decoder + transformer + optimizer state can't coexist on the GPU (full fine-tune, high-rank LoRA). No effect for FSDP. |

### DataConfig

Data loading and processing configuration.

```yaml
data:
  preprocessed_data_root: "/path/to/preprocessed/data"  # Path to precomputed dataset
  num_dataloader_workers: 2                             # Background data loading workers
```

**Key parameters:**

| Parameter                | Description                                                                                |
|--------------------------|--------------------------------------------------------------------------------------------|
| `preprocessed_data_root` | Path to your preprocessed dataset (contains `latents/`, `conditions/`, etc.)               |
| `num_dataloader_workers` | Number of parallel data loading processes (0 = synchronous loading, useful when debugging) |

### ValidationConfig

Validation and inference settings for monitoring training progress. Validation samples use a self-describing
format where each sample specifies its own prompt and conditions.

```yaml
validation:
  samples:
    - prompt: "A cat playing with a ball"
      conditions:
        - type: first_frame
          image_or_video: "/path/to/image.png"
    - prompt: "A dog running in a field"
  video_dims: [576, 576, 89]                # Output dimensions: [width, height, frames]
  negative_prompt: "worst quality, inconsistent motion, blurry, jittery, distorted"  # Negative prompt for all samples
  frame_rate: 25.0                          # Output video frame rate (fps)
  seed: 42                                  # Random seed for reproducibility
  inference_steps: 30                       # Number of denoising steps
  interval: 100                             # Run validation every N steps (null to disable)
  guidance_scale: 4.0                       # CFG scale (higher = stronger prompt adherence)
  stg_scale: 1.0                            # STG scale (0.0 to disable)
  stg_blocks: [29]                          # Transformer blocks to apply STG perturbation
  stg_mode: "stg_av"                        # STG mode: "stg_av" (audio+video) or "stg_v" (video only)
  generate_audio: true                      # Whether to generate audio during validation
  generate_video: true                      # Whether to generate video during validation
  skip_initial_validation: false            # Skip validation at step 0
```

**Key parameters:**

| Parameter                | Description                                                                                                              |
|--------------------------|--------------------------------------------------------------------------------------------------------------------------|
| `samples`                | List of `ValidationSample` objects (see below). Replaces the legacy `prompts`/`images`/`reference_videos` fields.        |
| `video_dims`             | Output dimensions `[width, height, frames]`. Width/height must be divisible by 32, frames must satisfy `frames % 8 == 1` |
| `interval`               | Steps between validation runs (set to `null` to disable)                                                                 |
| `guidance_scale`         | CFG (Classifier-Free Guidance) scale. Recommended: 4.0                                                                   |
| `stg_scale`              | STG (Spatio-Temporal Guidance) scale. 0.0 disables STG. Recommended: 1.0                                                 |
| `stg_blocks`             | Transformer blocks to perturb for STG. Recommended: `[29]` (single block)                                                |
| `stg_mode`               | STG mode: `"stg_av"` perturbs both audio and video, `"stg_v"` perturbs video only                                        |
| `generate_audio`         | Whether to generate audio in validation samples                                                                          |
| `generate_video`         | Whether to generate video in validation samples. Set to `false` for V2A (video-to-audio) validation. Default: `true`     |
| `skip_initial_validation`| Skip validation video sampling at step 0 (beginning of training)                                                         |

#### ValidationSample

Each sample in the `samples` list has:

| Field        | Description                                                                                     |
|--------------|-------------------------------------------------------------------------------------------------|
| `prompt`     | Text prompt for this validation sample.                                                         |
| `conditions` | List of validation conditions (see types below). Empty list = text-only generation.             |
| `video_dims` | Optional per-sample override for `(width, height, frames)`. Inherits from `ValidationConfig` if not set. |
| `seed`       | Optional per-sample override for random seed. Inherits from `ValidationConfig` if not set.      |

#### Validation Condition Types

| Type             | Parameters                                                 | Description                                                             |
|------------------|------------------------------------------------------------|-------------------------------------------------------------------------|
| `first_frame`    | `image_or_video` (path)                                    | Use the first frame of the image/video as conditioning.                 |
| `prefix`         | `video` or `audio` (path), optional `num_frames`/`duration`| Use a video/audio clip as temporal prefix (for extension forward).      |
| `suffix`         | `video` or `audio` (path), optional `num_frames`/`duration`| Use a video/audio clip as temporal suffix (for extension backward).     |
| `spatial_crop`   | `video` (path), `spatial_region` (y1, x1, y2, x2)         | Provide spatial context for outpainting. Video only.                    |
| `mask`           | `video` or `audio` (path), `mask` (path)                   | Mask-based inpainting with a binary mask file.                          |
| `reference`      | `video` or `audio` (path), optional `downscale_factor`, `include_in_output` | IC-LoRA style reference conditioning.            |
| `video_to_audio` | `video` (path)                                             | Freeze video, generate audio. For Foley/V2A tasks.                      |
| `audio_to_video` | `audio` (path)                                             | Freeze audio, generate video. For audio-driven generation.              |

> [!NOTE]
> The legacy fields `prompts`, `images`, and `reference_videos` are deprecated but auto-converted to `samples`
> internally. New configs should use the `samples` format.

### CheckpointsConfig

Model checkpointing configuration.

```yaml
checkpoints:
  interval: 250       # Steps between checkpoint saves (null = disabled)
  keep_last_n: 3      # Number of recent checkpoints to retain
  precision: bfloat16 # Precision for saved weights (bfloat16 or float32)
  no_resume: false            # Ignore saved state, start from step 0
  save_training_state: "minimal"  # "full", "minimal", or "off"
```

**Key parameters:**

| Parameter     | Description                                                                   |
|---------------|-------------------------------------------------------------------------------|
| `interval`    | Steps between intermediate checkpoint saves (set to `null` to disable)        |
| `keep_last_n` | Number of most recent checkpoints to keep (-1 = keep all)                     |
| `precision`   | Precision for saved checkpoint weights: `"bfloat16"` (default) or `"float32"` |
| `no_resume` | When `true`, ignore saved training state and start from step 0. Model weights from `load_checkpoint` are still loaded. |
| `save_training_state` | Save training state for resume: `"full"` (optimizer + scheduler + RNG), `"minimal"` (scheduler + RNG only, sufficient for LoRA), `"off"` (no resume). |

### HubConfig

Hugging Face Hub integration for automatic model uploads.

```yaml
hub:
  push_to_hub: false                   # Enable Hub uploading
  hub_model_id: "username/model-name"  # Hub repository ID
```

**Key parameters:**

| Parameter      | Description                                                      |
|----------------|------------------------------------------------------------------|
| `push_to_hub`  | Whether to automatically push trained models to Hugging Face Hub |
| `hub_model_id` | Repository ID in format `"username/repository-name"`             |

### WandbConfig

Weights & Biases logging configuration.

```yaml
wandb:
  enabled: false               # Enable W&B logging
  project: "ltx-2-trainer"     # W&B project name
  entity: null                 # W&B username or team
  tags: [ ]                    # Tags for the run
  log_validation_videos: true  # Log validation videos to W&B
```

**Key parameters:**

| Parameter               | Description                                      |
|-------------------------|--------------------------------------------------|
| `enabled`               | Whether to enable W&B logging                    |
| `project`               | W&B project name                                 |
| `entity`                | W&B username or team (null uses default account) |
| `log_validation_videos` | Whether to log validation videos to W&B          |

### FlowMatchingConfig

Flow matching training configuration for timestep sampling.

```yaml
flow_matching:
  timestep_sampling_mode: "shifted_logit_normal"  # Timestep sampling strategy
  timestep_sampling_params: { }                   # Additional sampling parameters
```

**Key parameters:**

| Parameter                  | Description                                                |
|----------------------------|------------------------------------------------------------|
| `timestep_sampling_mode`   | Sampling strategy: `"uniform"` or `"shifted_logit_normal"` |
| `timestep_sampling_params` | Additional parameters for the sampling strategy            |

### General Configuration

Top-level settings for the training run.

```yaml
seed: 42                                  # Random seed for reproducibility
output_dir: "outputs/my_training_run"     # Directory for outputs (checkpoints, validation videos, logs)
```

| Parameter    | Description                                              |
|--------------|----------------------------------------------------------|
| `seed`       | Random seed for reproducibility (default: `42`)          |
| `output_dir` | Directory to save outputs (default: `"outputs"`)         |

## 🚀 Next Steps

Once you've configured your training parameters:

- Set up your dataset using [Dataset Preparation](dataset-preparation.md)
- Choose your training approach in [Training Modes](training-modes.md)
- Start training with the [Training Guide](training-guide.md)