InstantX/MiniMax-H3-Turbo-Lora-Diffusers

MiniMax-H3 Turbo LoRA — Diffusers

Diffusers / PEFT conversion of larryvrh/MiniMax-H3-Turbo-Lora: a LoRA that lets MiniMax-H3 render joint video + synchronized stereo audio in about 4 sampling steps instead of the usual ~20.

This repo ships:

• converted LoRA weights in Diffusers PEFT layout (transformer.*.lora_A/B.weight)
• convert.py to turn the original ComfyUI / generate.py safetensors into that layout

⚠️ Early prototype. Same caveat as the upstream release: under-trained preview weights, not production quality. They already beat the base model at 4 steps (sharper detail, cleaner / better-synced audio), but treat this as a work-in-progress taste, not a finished product. Prefer the non-EMA ckpt500 weights by default.

Weights

Converted from the upstream Turbo LoRA (bf16, W_eff = W + lora_B @ lora_A, alpha = rank so scale is 1). QKV is split into to_q / to_k / to_v, and SwiGLU fc1 halves are swapped to match Diffusers' [value; gate] layout.

file	source (ComfyUI layout)	notes
minimax_h3_turbo_4step_ckpt500_diffusers.safetensors	minimax_h3_turbo_4step_ckpt500.safetensors	recommended default — newest non-EMA @ ~500 steps, usually sharpest
minimax_h3_turbo_4step_ema_ckpt500_diffusers.safetensors	minimax_h3_turbo_4step_ema_ckpt500.safetensors	EMA @ ~500 steps — smoother, but early EMA can show ghosting / motion smear

Ranks: attention / MLP = 64, AdaLN = 16. Keys are prefixed with transformer. for MiniMaxH3Transformer3DModel.load_lora_adapter.

Requirements

MiniMax-H3 is not in a released Diffusers build yet. Install Diffusers from main, plus PEFT:

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
pip install -r requirements.txt
pip install git+https://github.com/huggingface/diffusers.git

Base weights: Diffusers-format MiniMaxAI/MiniMax-H3 (or your local conversion). Use a non-pruned DiT; pruned time-conditioning layouts are not compatible with this LoRA (same restriction as upstream).

Quick start (Diffusers)

import torch
from diffusers import ComponentsManager, ModularPipeline
from diffusers.utils.export_utils import encode_video
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

def network_alphas_alpha_eq_rank(state_dict):
    # Turbo LoRA: alpha == rank. Required when ranks differ (attn/mlp=64, adaln=16).
    alphas = {}
    for key, tensor in state_dict.items():
        if key.endswith(".lora_B.weight") and tensor.ndim > 1:
            base = key[: -len(".lora_B.weight")]
            alphas[f"{base}.alpha"] = float(tensor.shape[1])
    return alphas

lora_path = hf_hub_download(
    "InstantX/MiniMax-H3-Turbo-Lora-Diffusers",
    "minimax_h3_turbo_4step_ckpt500_diffusers.safetensors",
)

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", components_manager=manager)
pipe.load_components(dtype=torch.bfloat16)

lora_sd = load_file(lora_path, device="cpu")
pipe.transformer.load_lora_adapter(
    lora_sd,
    prefix="transformer",
    adapter_name="turbo_4step",
    network_alphas=network_alphas_alpha_eq_rank(lora_sd),
)

# Load LoRA *before* enabling offload so PEFT injects into resident modules.
manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="12GB")

# Optional: FlashAttention-3 on Hopper (kernels from the Hub).
try:
    pipe.transformer.set_attention_backend("_flash_3_hub")
except Exception:
    pipe.transformer.set_attention_backend("native")

# MiniMaxH3Scheduler: num_inference_steps is the sigma grid length *including* terminal 0,
# so it drives (num_inference_steps - 1) model evals.
#   5 -> 4 evals  (matches upstream generate.py --steps 4)
#   7–9 -> 6–8 evals  (upstream comfort zone for sharpness at this early checkpoint)
results = pipe(
    prompt="A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark.",
    num_frames=124,   # 17*k+5, ~5.17s @ 24fps
    height=768,
    width=1344,
    num_inference_steps=5,
    generator=torch.Generator().manual_seed(42),
    output=["videos", "audio", "sampling_rate"],
)

encode_video(
    results["videos"][0],
    fps=24,
    output_path="out.mp4",
    audio=results["audio"][0],
    audio_sample_rate=results["sampling_rate"],
)

Diffusers already runs dual video / audio schedules (scheduler shift 12, audio_scheduler shift 3). You do not need the ComfyUI Turbo custom sampler node; a wrong single-schedule sampler is what blows up audio at 4 steps in ComfyUI.

Convert from the original Turbo LoRA

Original weights live in larryvrh/MiniMax-H3-Turbo-Lora (ComfyUI module names, fused qkv_proj / mlp.fc1).

pip install safetensors torch

python convert.py \
  --input minimax_h3_turbo_4step_ckpt500.safetensors \
  --output minimax_h3_turbo_4step_ckpt500_diffusers.safetensors

What convert.py does:

1. Renames ComfyUI paths onto MiniMaxH3Transformer3DModel (blocks.* → transformer_blocks.*, mlp.fc* → ff.net.*, final_layer.adaln_proj → norm_out.linear, …).
2. Splits fused attn.qkv_proj LoRA into to_q / to_k / to_v (shared A, row-split B in [q_all; k_all; v_all] layout).
3. Swaps mlp.fc1 LoRA halves from [gate; value] to Diffusers SwiGLU [value; gate].
4. Writes keys with a transformer. prefix for load_lora_adapter.

Notes

• Steps: 4 model evals (num_inference_steps=5) works; at this early checkpoint 6–8 evals (num_inference_steps=7…9) are usually sharper. Any count ≥ 4 evals is valid; more steps look better.
• Resolution / duration: height / width multiples of 32 (short edge typically 768). num_frames at 24 fps snaps up to the video VAE’s 17·k+5 grid (124 ≈ 5 s). Validated roughly 5–15 s.
• VRAM: the base DiT is ~33B. An 80–96 GB GPU is comfortable with ComponentsManager.enable_auto_cpu_offload; smaller cards need quantization / group offload as in the MiniMax-H3 Diffusers docs.
• Audio: 32 kHz stereo aligned to the video; video and audio ride different flow schedules inside one transformer call.
• ComfyUI: for the original graph / custom Turbo sampler, use the upstream repo and Larryvrh/ComfyUI-MiniMax-H3-Turbo.

Credit

• Turbo LoRA training & original release: larryvrh/MiniMax-H3-Turbo-Lora
• Base model: MiniMaxAI/MiniMax-H3
• Diffusers MiniMax-H3 integration: Hugging Face Diffusers (modular pipeline)