Upload LTX25ModularPipeline

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,144 @@
+---
+library_name: diffusers
+tags:
+- modular-diffusers
+- diffusers
+- ltx2.5
+---
+This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.
+
+**Pipeline Type**: LTX25AutoBlocks
+
+**Description**: Auto blocks for LTX-2.5 supporting text-to-video, image-to-video, condition-to-video and in-context (IC-LoRA) generation (joint video + audio). Identical to `LTX2AutoBlocks` except that the video decoder is `LTX2DiffusionVaeDecoderStep`, since the diffusion decoder is the native default from LTX-2.5 on. To decode with the convolutional VAE instead, swap the decode block: `blocks.sub_blocks["decode"] = LTX2AutoDecoderStep()`.
+
+This pipeline uses a 8-block architecture that can be customized and extended.
+
+## Example Usage
+
+[TODO]
+
+## Pipeline Architecture
+
+This modular pipeline is composed of the following blocks:
+
+1. **prompt_enhancer** (`LTX2AutoPromptEnhancerStep`)
+   - Conditional prompt-enhancer step, run only when `enable_prompt_enhancement` is truthy.
+2. **text_encoder** (`LTX2TextConditioningStep`)
+   - Text-conditioning stage for LTX-2.X: encodes the prompt(s), then runs the text connectors to produce the video/audio-branch connector embeddings the denoiser consumes. Outputs stay at one row per prompt -- the denoise stage expands them by `num_videos_per_prompt` -- so they can be reused across denoise runs.
+3. **duration** (`LTX2AutoDurationStep`)
+   - Conditional duration-prediction step, run only when `num_frames` is omitted.
+4. **vae_encoder** (`LTX2AutoVaeEncoderStep`)
+   - VAE encoder step that encodes the reference `image` into latents for image-to-video.
+5. **condition_encoder** (`LTX2AutoConditionEncoderStep`)
+   - Conditional condition-encoder step, run only for the condition and in-context workflows.
+6. **reference_encoder** (`LTX2AutoReferenceEncoderStep`)
+   - Conditional reference-encoder step, run only when `reference_conditions` are supplied.
+7. **denoise** (`LTX2AutoCoreDenoiseStep`)
+   - Auto denoise block that selects the workflow based on inputs.
+8. **decode** (`LTX25AutoDecoderStep`)
+   - Auto decode block for LTX-2.5 that selects the decoder based on inputs. 
+
+## Model Components
+
+1. prompt_enhancer (`PreTrainedModel`)
+2. processor (`ProcessorMixin`)
+3. text_encoder (`PreTrainedModel`)
+4. tokenizer (`PreTrainedTokenizerBase`)
+5. connectors (`LTX2TextConnectors`)
+6. duration_head (`LTX2DurationHead`)
+7. vae (`AutoencoderKLLTX2Video`)
+8. video_processor (`VideoProcessor`)
+9. transformer (`LTX2VideoTransformer3DModel`)
+10. scheduler (`FlowMatchEulerDiscreteScheduler`)
+11. audio_vae (`AutoencoderKLLTX2Audio`)
+12. guider (`LTX2Guidance`)
+13. audio_guider (`LTX2Guidance`)
+14. diffusion_decoder (`LTX2VideoDiffusionDecoderModel`)
+15. vocoder (`LTX2Vocoder`) 
+
+## Workflow Input Specification
+
+<details>
+<summary><strong>text2video</strong></summary>
+
+- `prompt` (`str`): The prompt or prompts to guide image generation.
+
+</details>
+
+<details>
+<summary><strong>image2video</strong></summary>
+
+- `prompt` (`str`): The prompt or prompts to guide image generation.
+- `image` (`Image | list`): Reference image(s) for denoising. Can be a single image or list of images.
+
+</details>
+
+<details>
+<summary><strong>condition</strong></summary>
+
+- `prompt` (`str`): The prompt or prompts to guide image generation.
+- `conditions` (`list`, *optional*): `LTX2VideoCondition` (or list of them) placing image/video conditions at latent frame indices of the generated video.
+
+</details>
+
+<details>
+<summary><strong>in_context</strong></summary>
+
+- `prompt` (`str`): The prompt or prompts to guide image generation.
+- `num_frames` (`int`, *optional*): The number of frames in the generated video. Omit to auto-predict via the `duration_head` (see `LTX2AutoDurationStep`).
+- `reference_conditions` (`list`): `LTX2ReferenceCondition` (or list of them) whose videos are encoded into extra latent tokens the IC-LoRA adapter attends to.
+
+</details>
+
+
+## Input/Output Specification
+
+**Inputs:**
+
+- `prompt` (`str`, *optional*): The prompt or prompts to guide image generation.
+- `conditions` (`list`, *optional*): `LTX2VideoCondition` (or list of them) placing image/video conditions at latent frame indices of the generated video.
+- `enable_prompt_enhancement` (`bool`, *optional*, defaults to `False`): Whether to run the prompt enhancer. Opt-in, matching the Lightricks reference pipelines.
+- `system_prompt` (`str`, *optional*): System prompt for enhancement. Defaults to `LTX2_5_I2V_DEFAULT_SYSTEM_PROMPT` when a `PIL.Image.Image` condition frame is available, else `LTX2_5_T2V_DEFAULT_SYSTEM_PROMPT`.
+- `prompt_max_new_tokens` (`int`, *optional*): Maximum number of new tokens to generate during prompt enhancement. Defaults to 600, the LTX-2.5 Gemma-4 enhancer's budget.
+- `prompt_enhancement_kwargs` (`dict`, *optional*): Keyword arguments for the enhancer's `.generate` call. Defaults to greedy decoding.
+- `prompt_enhancement_seed` (`int`, *optional*, defaults to `10`): Random seed for prompt enhancement (inert under LTX-2.5's greedy decoding).
+- `generator` (`Generator`, *optional*): Torch generator for deterministic generation.
+- `image` (`Image | list`, *optional*): Reference image(s) for denoising. Can be a single image or list of images.
+- `negative_prompt` (`str`, *optional*): The prompt or prompts not to guide the image generation.
+- `max_sequence_length` (`int`, *optional*, defaults to `1024`): Maximum sequence length for prompt encoding.
+- `min_seconds` (`float`, *optional*, defaults to `1.0`): Lower bound on the auto-predicted duration.
+- `max_seconds` (`float`, *optional*, defaults to `20.0`): Upper bound on the auto-predicted duration. Must be strictly greater than `min_seconds`.
+- `frame_rate` (`float`, *optional*, defaults to `24.0`): Frames per second of the generated video.
+- `height` (`int`, *optional*, defaults to `512`): The height in pixels of the generated image.
+- `width` (`int`, *optional*, defaults to `704`): The width in pixels of the generated image.
+- `image_crf` (`int`, *optional*): H.264 CRF used to re-compress the conditioning `image` before VAE encode, matching the compression the model was trained against. `None` (default) resolves from the text-encoder generation (33 through LTX-2.3, 18 for LTX-2.5). Pass `0` to skip re-compression. Requires a `PIL.Image.Image` when re-compression runs.
+- `num_frames` (`int`, *optional*): The number of frames in the generated video. Omit to auto-predict via the `duration_head` (see `LTX2AutoDurationStep`).
+- `reference_conditions` (`list`, *optional*): `LTX2ReferenceCondition` (or list of them) whose videos are encoded into extra latent tokens the IC-LoRA adapter attends to.
+- `reference_downscale_factor` (`int`, *optional*, defaults to `1`): Ratio between the target and reference resolutions; 2 means the reference is preprocessed at half the target resolution. Spatial coordinates are scaled by this factor so the reference tokens land in the target coordinate space. Must match the factor the IC-LoRA was trained with.
+- `conditioning_attention_strength` (`float`, *optional*, defaults to `1.0`): Scalar in [0, 1] controlling how strongly the noisy tokens and reference tokens attend to each other. 1.0 (default) leaves attention unmasked.
+- `conditioning_attention_mask` (`Tensor`, *optional*): Optional pixel-space mask of shape (1, 1, F, H, W) with values in [0, 1] giving spatially varying attention strength. Downsampled to the reference's latent grid and multiplied by `conditioning_attention_strength`.
+- `num_videos_per_prompt` (`int`, *optional*, defaults to `1`): The number of images to generate per prompt.
+- `condition_latents` (`list`, *optional*): Per-condition normalized VAE latents of shape [1, C, F, H, W].
+- `condition_strengths` (`list`, *optional*): Per-condition conditioning strengths.
+- `condition_indices` (`list`, *optional*): Per-condition latent frame index at which the condition is applied.
+- `condition_pixel_frames` (`list`, *optional*): Per-condition trimmed pixel frame count, used to clamp single-frame keyframe coords.
+- `reference_latents` (`Tensor`, *optional*): Packed reference tokens of shape [1, total_reference_tokens, C], or `None` when no reference conditions were supplied (`LTX2AutoReferenceEncoderStep` is skipped).
+- `reference_coords` (`Tensor`, *optional*): RoPE coordinates for the reference tokens.
+- `reference_token_counts` (`list`, *optional*): Per-reference token counts, in `reference_conditions` order.
+- `latents` (`Tensor`): Pre-generated noisy latents for image generation.
+- `noise_scale` (`float`, *optional*): Initial noise level for the un-conditioned tokens. `None` (default) resolves to `sigmas[0]` when custom `sigmas` are supplied, else 1.0.
+- `sigmas` (`list`, *optional*): Custom sigmas for the denoising process.
+- `reference_cross_mask` (`Tensor`, *optional*): Per-reference-token noisy<->reference attention strengths of shape [1, num_ref_tokens].
+- `num_inference_steps` (`int`): The number of denoising steps.
+- `timesteps` (`Tensor`): Timesteps for the denoising process.
+- `audio_latents` (`Tensor`): Optional pre-encoded audio latents; random noise is used when not provided.
+- `**denoiser_input_fields` (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
+- `use_cross_timestep` (`bool`, *optional*, defaults to `True`): Whether to condition the transformer on a separate per-token cross timestep (LTX-2.3+).
+- `attention_kwargs` (`dict`, *optional*): Additional kwargs for attention processors.
+- `image_latents` (`Tensor`, *optional*): VAE-encoded reference-image latents used for image-to-video conditioning.
+- `output_type` (`str`, *optional*, defaults to `pil`): Output format: 'pil', 'np', 'pt'.
+
+**Outputs:**
+
+- `videos` (`list`): The generated videos.
+- `audio` (`Tensor`): The generated audio waveform.

audio_vae/config.json

@@ -0,0 +1,24 @@
+{
+  "_class_name": "AutoencoderKLLTX2Audio",
+  "_diffusers_version": "0.40.0.dev0",
+  "_name_or_path": "hf-internal-testing/tiny-ltx2-modular-pipe",
+  "attn_resolutions": null,
+  "base_channels": 4,
+  "causality_axis": "height",
+  "ch_mult": [
+    1
+  ],
+  "double_z": true,
+  "dropout": 0.0,
+  "in_channels": 2,
+  "is_causal": true,
+  "latent_channels": 2,
+  "mel_bins": 8,
+  "mel_hop_length": 160,
+  "mid_block_add_attention": false,
+  "norm_type": "pixel",
+  "num_res_blocks": 1,
+  "output_channels": 2,
+  "resolution": 32,
+  "sample_rate": 16000
+}

audio_vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a86c1d8d653163146e3e1c6e5f4365ba23edb41993640b9f4458d39950862610
+size 13416

connectors/config.json

@@ -0,0 +1,26 @@
+{
+  "_class_name": "LTX2TextConnectors",
+  "_diffusers_version": "0.40.0.dev0",
+  "_name_or_path": "hf-internal-testing/tiny-ltx2-modular-pipe",
+  "audio_connector_attention_head_dim": 8,
+  "audio_connector_num_attention_heads": 4,
+  "audio_connector_num_layers": 1,
+  "audio_connector_num_learnable_registers": null,
+  "audio_gated_attn": false,
+  "audio_hidden_dim": 2048,
+  "caption_channels": 32,
+  "causal_temporal_positioning": false,
+  "connector_rope_base_seq_len": 32,
+  "per_modality_projections": false,
+  "proj_bias": false,
+  "rope_double_precision": false,
+  "rope_theta": 10000.0,
+  "rope_type": "split",
+  "text_proj_in_factor": 3,
+  "video_connector_attention_head_dim": 8,
+  "video_connector_num_attention_heads": 4,
+  "video_connector_num_layers": 1,
+  "video_connector_num_learnable_registers": null,
+  "video_gated_attn": false,
+  "video_hidden_dim": 4096
+}

connectors/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf8b2f822740f1cc4cbad5462b9679dc7c1ff1a9138080d08ae432d17e51d01d
+size 116776

diffusion_decoder/config.json

@@ -0,0 +1,84 @@
+{
+  "_class_name": "LTX2VideoDiffusionDecoderModel",
+  "_diffusers_version": "0.40.0.dev0",
+  "decoder_head_dim": 8,
+  "decoder_model_output_type": "x0",
+  "decoder_num_inference_steps": 1,
+  "decoder_stage5_kernel": [
+    3,
+    3,
+    3
+  ],
+  "decoder_stage_channels": [
+    16,
+    16,
+    16,
+    16,
+    16
+  ],
+  "decoder_stage_depths": [
+    1,
+    1,
+    1,
+    1,
+    1
+  ],
+  "decoder_stage_kernels": [
+    [
+      3,
+      3,
+      3
+    ],
+    [
+      3,
+      3,
+      3
+    ],
+    [
+      3,
+      3,
+      3
+    ],
+    [
+      3,
+      3,
+      3
+    ]
+  ],
+  "decoder_t_emb_dim": 16,
+  "decoder_timestep_scale_multiplier": 1000.0,
+  "decoder_upsample_channel_reductions": [
+    1,
+    1,
+    1,
+    1
+  ],
+  "decoder_upsample_strides": [
+    [
+      1,
+      1,
+      1
+    ],
+    [
+      1,
+      1,
+      1
+    ],
+    [
+      1,
+      1,
+      1
+    ],
+    [
+      2,
+      2,
+      2
+    ]
+  ],
+  "latent_channels": 4,
+  "out_channels": 3,
+  "patch_size": 1,
+  "scaling_factor": 1.0,
+  "spatial_compression_ratio": 2,
+  "temporal_compression_ratio": 2
+}

diffusion_decoder/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2222de9e6ced96ba9b89b22327554a7cb0708d50bdb4b8d2eb8f24b3fdbdec51
+size 133692

duration_head/config.json

@@ -0,0 +1,11 @@
+{
+  "_class_name": "LTX2DurationHead",
+  "_diffusers_version": "0.40.0.dev0",
+  "_name_or_path": "hf-internal-testing/tiny-ltx2-modular-pipe",
+  "audio_cross_attention_dim": 32,
+  "mlp_hidden_dim": 8,
+  "num_pooler_heads": 2,
+  "num_queries": 1,
+  "pooler_hidden_dim": 8,
+  "video_cross_attention_dim": 32
+}

duration_head/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05f94cb25d576346d6ef09ac9f17eeb93656a816c0cfd9c59c0c629da2399aa7
+size 5260

modular_model_index.json

@@ -0,0 +1,173 @@
+{
+  "_blocks_class_name": "LTX25AutoBlocks",
+  "_class_name": "LTX25ModularPipeline",
+  "_diffusers_version": "0.40.0.dev0",
+  "audio_vae": [
+    "diffusers",
+    "AutoencoderKLLTX2Audio",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "audio_vae",
+      "type_hint": [
+        "diffusers",
+        "AutoencoderKLLTX2Audio"
+      ],
+      "variant": null
+    }
+  ],
+  "connectors": [
+    "ltx2",
+    "LTX2TextConnectors",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "connectors",
+      "type_hint": [
+        "ltx2",
+        "LTX2TextConnectors"
+      ],
+      "variant": null
+    }
+  ],
+  "diffusion_decoder": [
+    "diffusers",
+    "LTX2VideoDiffusionDecoderModel",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "diffusion_decoder",
+      "type_hint": [
+        "diffusers",
+        "LTX2VideoDiffusionDecoderModel"
+      ],
+      "variant": null
+    }
+  ],
+  "duration_head": [
+    "ltx2",
+    "LTX2DurationHead",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "duration_head",
+      "type_hint": [
+        "ltx2",
+        "LTX2DurationHead"
+      ],
+      "variant": null
+    }
+  ],
+  "processor": [
+    null,
+    null,
+    {
+      "pretrained_model_name_or_path": null,
+      "revision": null,
+      "subfolder": "",
+      "type_hint": [
+        "transformers",
+        "ProcessorMixin"
+      ],
+      "variant": null
+    }
+  ],
+  "prompt_enhancer": [
+    null,
+    null,
+    {
+      "pretrained_model_name_or_path": null,
+      "revision": null,
+      "subfolder": "",
+      "type_hint": [
+        "transformers",
+        "PreTrainedModel"
+      ],
+      "variant": null
+    }
+  ],
+  "scheduler": [
+    "diffusers",
+    "FlowMatchEulerDiscreteScheduler",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "scheduler",
+      "type_hint": [
+        "diffusers",
+        "FlowMatchEulerDiscreteScheduler"
+      ],
+      "variant": null
+    }
+  ],
+  "text_encoder": [
+    "transformers",
+    "Gemma3ForConditionalGeneration",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "text_encoder",
+      "type_hint": [
+        "transformers",
+        "Gemma3ForConditionalGeneration"
+      ],
+      "variant": null
+    }
+  ],
+  "tokenizer": [
+    "transformers",
+    "GemmaTokenizer",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "tokenizer",
+      "type_hint": [
+        "transformers",
+        "GemmaTokenizer"
+      ],
+      "variant": null
+    }
+  ],
+  "transformer": [
+    "diffusers",
+    "LTX2VideoTransformer3DModel",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "transformer",
+      "type_hint": [
+        "diffusers",
+        "LTX2VideoTransformer3DModel"
+      ],
+      "variant": null
+    }
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKLLTX2Video",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "vae",
+      "type_hint": [
+        "diffusers",
+        "AutoencoderKLLTX2Video"
+      ],
+      "variant": null
+    }
+  ],
+  "vocoder": [
+    "ltx2",
+    "LTX2Vocoder",
+    {
+      "pretrained_model_name_or_path": "hf-internal-testing/tiny-ltx2-5-modular-pipe",
+      "revision": null,
+      "subfolder": "vocoder",
+      "type_hint": [
+        "ltx2",
+        "LTX2Vocoder"
+      ],
+      "variant": null
+    }
+  ]
+}

scheduler/scheduler_config.json

@@ -0,0 +1,18 @@
+{
+  "_class_name": "FlowMatchEulerDiscreteScheduler",
+  "_diffusers_version": "0.40.0.dev0",
+  "base_image_seq_len": 256,
+  "base_shift": 0.5,
+  "invert_sigmas": false,
+  "max_image_seq_len": 4096,
+  "max_shift": 1.15,
+  "num_train_timesteps": 1000,
+  "shift": 1.0,
+  "shift_terminal": null,
+  "stochastic_sampling": false,
+  "time_shift_type": "exponential",
+  "use_beta_sigmas": false,
+  "use_dynamic_shifting": false,
+  "use_exponential_sigmas": false,
+  "use_karras_sigmas": false
+}

text_encoder/config.json

@@ -0,0 +1,78 @@
+{
+  "architectures": [
+    "Gemma3ForConditionalGeneration"
+  ],
+  "boi_token_index": 255999,
+  "dtype": "float32",
+  "eoi_token_index": 256000,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "image_token_index": 262144,
+  "initializer_range": 0.02,
+  "mm_tokens_per_image": 256,
+  "model_type": "gemma3",
+  "text_config": {
+    "_sliding_window_pattern": 2,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_logit_softcapping": null,
+    "bos_token_id": 2,
+    "cache_implementation": "hybrid",
+    "dtype": "float32",
+    "eos_token_id": 1,
+    "final_logit_softcapping": null,
+    "head_dim": 32,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 32,
+    "initializer_range": 0.02,
+    "intermediate_size": 128,
+    "layer_types": [
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 131072,
+    "model_type": "gemma3_text",
+    "num_attention_heads": 1,
+    "num_hidden_layers": 2,
+    "num_key_value_heads": 1,
+    "pad_token_id": 0,
+    "query_pre_attn_scalar": 168,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "factor": 8.0,
+        "rope_theta": 1000000.0,
+        "rope_type": "linear"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "sliding_window_pattern": 2,
+    "tie_word_embeddings": true,
+    "use_bidirectional_attention": false,
+    "use_cache": true,
+    "vocab_size": 262208
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.15.0",
+  "vision_config": {
+    "attention_dropout": 0.0,
+    "dtype": "float32",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 32,
+    "image_size": 896,
+    "intermediate_size": 128,
+    "layer_norm_eps": 1e-06,
+    "model_type": "siglip_vision_model",
+    "num_attention_heads": 1,
+    "num_channels": 3,
+    "num_hidden_layers": 2,
+    "patch_size": 14,
+    "vision_use_head": false
+  }
+}

text_encoder/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "cache_implementation": "hybrid",
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "5.15.0"
+}

text_encoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f2b2e38ee7aced4b332a26484feae78e3d809c0cec32b1bb2a50e62962d0614
+size 34409608

tokenizer/chat_template.jinja

@@ -0,0 +1,47 @@
+{{ bos_token }}
+{%- if messages[0]['role'] == 'system' -%}
+    {%- if messages[0]['content'] is string -%}
+        {%- set first_user_prefix = messages[0]['content'] + '
+
+' -%}
+    {%- else -%}
+        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '
+
+' -%}
+    {%- endif -%}
+    {%- set loop_messages = messages[1:] -%}
+{%- else -%}
+    {%- set first_user_prefix = "" -%}
+    {%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+    {%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
+        {{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
+    {%- endif -%}
+    {%- if (message['role'] == 'assistant') -%}
+        {%- set role = "model" -%}
+    {%- else -%}
+        {%- set role = message['role'] -%}
+    {%- endif -%}
+    {{ '<start_of_turn>' + role + '
+' + (first_user_prefix if loop.first else "") }}
+    {%- if message['content'] is string -%}
+        {{ message['content'] | trim }}
+    {%- elif message['content'] is iterable -%}
+        {%- for item in message['content'] -%}
+            {%- if item['type'] == 'image' -%}
+                {{ '<start_of_image>' }}
+            {%- elif item['type'] == 'text' -%}
+                {{ item['text'] | trim }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{ raise_exception("Invalid content type") }}
+    {%- endif -%}
+    {{ '<end_of_turn>
+' }}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{'<start_of_turn>model
+'}}
+{%- endif -%}

tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be966fdc945f0a849b2712f3e469802ee6997ab2df4e5a6f673956c41194619d
+size 33384829

tokenizer/tokenizer_config.json

@@ -0,0 +1,26 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "processor_class": "Gemma3Processor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false,
+  "variant": null
+}

transformer/config.json

@@ -0,0 +1,55 @@
+{
+  "_class_name": "LTX2VideoTransformer3DModel",
+  "_diffusers_version": "0.40.0.dev0",
+  "_name_or_path": "hf-internal-testing/tiny-ltx2-modular-pipe",
+  "activation_fn": "gelu-approximate",
+  "attention_bias": true,
+  "attention_head_dim": 8,
+  "attention_out_bias": true,
+  "audio_attention_head_dim": 4,
+  "audio_cross_attention_dim": 8,
+  "audio_cross_attn_mod": false,
+  "audio_ff_bias": true,
+  "audio_gated_attn": false,
+  "audio_hop_length": 160,
+  "audio_in_channels": 4,
+  "audio_num_attention_heads": 2,
+  "audio_out_channels": 4,
+  "audio_patch_size": 1,
+  "audio_patch_size_t": 1,
+  "audio_pos_embed_max_pos": 20,
+  "audio_sampling_rate": 16000,
+  "audio_scale_factor": 4,
+  "base_height": 2048,
+  "base_width": 2048,
+  "caption_channels": 32,
+  "causal_offset": 1,
+  "cross_attention_dim": 16,
+  "cross_attn_mod": false,
+  "cross_attn_timestep_scale_multiplier": 1000,
+  "ff_bias": true,
+  "gated_attn": false,
+  "in_channels": 4,
+  "norm_elementwise_affine": false,
+  "norm_eps": 1e-06,
+  "num_attention_heads": 2,
+  "num_layers": 2,
+  "out_channels": 4,
+  "patch_size": 1,
+  "patch_size_t": 1,
+  "perturbed_attn": false,
+  "pos_embed_max_pos": 20,
+  "qk_norm": "rms_norm_across_heads",
+  "rope_double_precision": false,
+  "rope_theta": 10000.0,
+  "rope_type": "split",
+  "timestep_scale_multiplier": 1000,
+  "use_keyframes_abs_pos_embedding": false,
+  "use_prompt_adaln_single": true,
+  "use_prompt_embeddings": true,
+  "vae_scale_factors": [
+    8,
+    32,
+    32
+  ]
+}

transformer/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:972d94425158a7c4b7b43594891c54d36e09dcaea9ddef2cec49d491ab0deb3a
+size 173544

vae/config.json

@@ -0,0 +1,62 @@
+{
+  "_class_name": "AutoencoderKLLTX2Video",
+  "_diffusers_version": "0.40.0.dev0",
+  "_name_or_path": "hf-internal-testing/tiny-ltx2-modular-pipe",
+  "block_out_channels": [
+    8
+  ],
+  "decoder_block_out_channels": [
+    8
+  ],
+  "decoder_causal": false,
+  "decoder_inject_noise": [
+    false,
+    false
+  ],
+  "decoder_layers_per_block": [
+    1,
+    1
+  ],
+  "decoder_spatial_padding_mode": "reflect",
+  "decoder_spatio_temporal_scaling": [
+    true
+  ],
+  "down_block_types": [
+    "LTX2VideoDownBlock3D",
+    "LTX2VideoDownBlock3D",
+    "LTX2VideoDownBlock3D",
+    "LTX2VideoDownBlock3D"
+  ],
+  "downsample_type": [
+    "spatial"
+  ],
+  "encoder_causal": true,
+  "encoder_spatial_padding_mode": "zeros",
+  "in_channels": 3,
+  "latent_channels": 4,
+  "layers_per_block": [
+    1
+  ],
+  "out_channels": 3,
+  "patch_size": 1,
+  "patch_size_t": 1,
+  "resnet_norm_eps": 1e-06,
+  "scaling_factor": 1.0,
+  "spatial_compression_ratio": null,
+  "spatio_temporal_scaling": [
+    true
+  ],
+  "temporal_compression_ratio": null,
+  "timestep_conditioning": false,
+  "upsample_factor": [
+    1
+  ],
+  "upsample_residual": [
+    false
+  ],
+  "upsample_type": [
+    "spatiotemporal",
+    "spatiotemporal",
+    "spatiotemporal"
+  ]
+}

vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:056241bb4b8a029713f5f71bbe3c5f904cdea378fab80ecbb2a8e7e9035140c8
+size 116464

vocoder/config.json

@@ -0,0 +1,34 @@
+{
+  "_class_name": "LTX2Vocoder",
+  "_diffusers_version": "0.40.0.dev0",
+  "_name_or_path": "hf-internal-testing/tiny-ltx2-modular-pipe",
+  "act_fn": "leaky_relu",
+  "antialias": false,
+  "antialias_kernel_size": 12,
+  "antialias_ratio": 2,
+  "final_act_fn": "tanh",
+  "final_bias": true,
+  "hidden_channels": 32,
+  "in_channels": 16,
+  "leaky_relu_negative_slope": 0.1,
+  "out_channels": 2,
+  "output_sampling_rate": 16000,
+  "resnet_dilations": [
+    [
+      1,
+      3,
+      5
+    ]
+  ],
+  "resnet_kernel_sizes": [
+    3
+  ],
+  "upsample_factors": [
+    2,
+    2
+  ],
+  "upsample_kernel_sizes": [
+    4,
+    4
+  ]
+}

vocoder/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be58882aaf71ba653bb607eb160f025c3526a85b589770836a1169b513cfbb94
+size 51616