README.md

---
language:
- en
license: other
license_name: minimax-h3-community-license-agreement
license_link: https://huggingface.co/OzzyGT/MiniMax_H3_sdnq_4bit_pruned/blob/main/LICENSE
base_model:
  - MiniMaxAI/MiniMax-H3
base_model_relation: quantized
tags:
- video-generation
- text-to-audio-video
- minimax-h3
- sdnq
pipeline_tag: text-to-video
library_name: diffusers
---
# MiniMax-H3 SDNQ Dynamic INT4 (pruned)

<table>
  <tr>
    <td width="50%">
      <video controls width="100%">
        <source src="https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/minimax_h3/h3_4bit_pruned_cart7s_tight2_2233586303.mp4" type="video/mp4">
      </video>
    </td>
    <td width="50%">
      <video controls width="100%">
        <source src="https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/minimax_h3/h3_8bit_pruned_cart7s_tight2_2233586303.mp4" type="video/mp4">
      </video>
    </td>
  </tr>
  <tr>
    <td align="center"><b>this model, SDNQ int4</b></td>
    <td align="center">SDNQ int8</td>
  </tr>
</table>

Left: this int4 model. Right: the [int8 build](https://huggingface.co/OzzyGT/MiniMax_H3_sdnq_8bit_pruned). Same prompt, same seed (2233586303), same 20 steps at 864x480 and 175 frames, generated with the script below. Turn the sound on, the audio is generated jointly with the video.

This is an int4 quantized version of [MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) using [SDNQ](https://github.com/Disty0/sdnq) (SD.Next Quantization) with the dynamic option and Hadamard Rotation, with the AdaLN branches pruned.

MiniMax point this out themselves in the [base model card](https://huggingface.co/MiniMaxAI/MiniMax-H3#h3-omni-transformer): of the transformer's 33B parameters, about 13B sit in AdaLN-related branches, and "because the AdaLN modulation outputs can be precomputed and cached, these parameters do not need to be loaded for inference-only deployment". They ship the full weights so the model can still be fine-tuned; this repository is the inference-only version with those parameters removed.

Only the two transformer weights are stored in this repository. The text encoder, both VAEs and the schedulers come from [OzzyGT/MiniMax_H3_sdnq_dynamic_4bit](https://huggingface.co/OzzyGT/MiniMax_H3_sdnq_dynamic_4bit); the modular index already points at it, so there is nothing extra to download by hand.

Notes:

- You need SDNQ v0.2.2 and above.
- `trust_remote_code=True` is required, on both `from_pretrained` and `load_components`. The pruned transformer ships its own model class.
- Pass `workflow="t2va"` (or `"fl2va"` / `"ref2va"`) so only that task's partition is loaded. Without it both partitions are pulled.

## Usage

You can find ready-to-use scripts in the [diffusers-recipes](https://github.com/asomoza/diffusers-recipes/blob/main/models/minimax_h3/README.md) repository, covering group offloading and automatic offload management.

## Sample code

```python
import torch
from diffusers import AutoencoderKLMiniMaxH3, ModularPipeline
from diffusers.hooks import apply_group_offloading
from diffusers.utils.export_utils import encode_video
from sdnq import SDNQConfig  # noqa: F401

# patch until fixed upstream, the fp32 pin covers the whole VAE and ignores dtype
AutoencoderKLMiniMaxH3._keep_in_fp32_modules = []

pipe = ModularPipeline.from_pretrained("OzzyGT/MiniMax_H3_sdnq_4bit_pruned", workflow="t2va", trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)

offload = {"onload_device": torch.device("cuda"), "offload_device": torch.device("cpu"), "use_stream": True}
pipe.transformer.enable_group_offload(offload_type="block_level", num_blocks_per_group=1, **offload)
apply_group_offloading(pipe.text_encoder.model, offload_type="leaf_level", **offload)
apply_group_offloading(pipe.vae, offload_type="leaf_level", **offload)
pipe.audio_vae.to("cuda")

prompt = """integrated_multimodal_description: Live-action, cinematic street food scene at night, warm string lights and soft neon bokeh in the background. A fluffy orange tabby cat wearing a tiny white chef hat works the griddle of a small wooden street cart, standing upright and holding a spatula. The front panel of the cart is plain bare wood, completely blank, with no writing, no signs and no lettering anywhere on it. On a plate beside the griddle sits a tall stack of finished golden pancakes, plain and dry, with no syrup on them yet. Filling the left third of the frame, close to camera and angled toward the lens, a hand-painted white price board carries two lines of very large bold black lettering: "PANCAKE $3.00" on the top line and "COFFEE $2.00" on the line below, thick brush strokes, high contrast, unmistakably legible. This board is the only text in the shot. [0s-1.1s] The camera holds on the price board and the plain stack, then pushes in slowly toward the griddle. [1.1s-1.7s] The cat flicks the spatula hard and the pancake launches straight up off the griddle, rising high above the cat's hat with nothing under it. [1.7s-3.2s] The shot ramps into smooth slow motion while the pancake is airborne, and it turns over once as it hangs there. While it is up there a dry male narrator (S1) says in an off-screen voiceover: <d>[English] He has never dropped one.</d> [3.2s-3.8s] The shot ramps back to normal speed as the pancake drops and lands flat on the griddle with a wet slap, the other side down now. The cat blinks once, looking extremely pleased with itself. Nobody speaks. [3.8s-4.3s] The cat lays the spatula flat on the cart and picks up a small glass syrup jug with both front paws. [4.3s-5.6s] It tips the jug over the dry stack and a thick ribbon of dark maple syrup pours out, spreading across the top pancake and running down the sides, and by 5.6s the pouring is completely finished and the jug is upright again. Nobody speaks. [5.6s-7.3s] All action is over. The cat sets the jug down on the cart, looks straight into the lens and holds completely still, only blinking once. Nothing else moves and nobody speaks for the rest of the shot.

overall_soundscape: A steady griddle sizzle runs underneath everything, with faint street ambience and distant traffic behind it. A soft whoosh as the pancake goes up, stretching and dropping in pitch while the shot is in slow motion. A clear wet slap exactly as it lands back on the griddle, alone in the mix with no voice over it. A light wooden clack as the spatula is set down. Then a thick glugging pour of syrup, loud and close, the only sound besides the sizzle while it runs, finishing by 5.6s. After that only the quiet sizzle and street ambience remain.

non_diegetic_music: Bright upbeat street jazz: brushed drums, a walking upright bass and a playful clarinet melody, thinning out to a held bass note through the slow motion section and picking back up after the pancake lands. Over the last second and a half the clarinet plays a short closing phrase and the whole band resolves together onto one clean final chord that lands on the last frame, so the music finishes with the shot instead of stopping mid-phrase."""

state = pipe(
    prompt=prompt,
    height=480,
    width=864,
    num_frames=175,
    num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(2233586303),
    output=["videos", "audio", "sampling_rate"],
)

encode_video(
    state.get("videos")[0],
    fps=24,
    audio=state.get("audio")[0],
    audio_sample_rate=state.get("sampling_rate"),
    output_path="sample.mp4",
)
```