Instructions to use akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0 with Diffusers:
pip install -U diffusers transformers accelerate
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline controlnet = ControlNetModel.from_pretrained("akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0") pipe = StableDiffusionControlNetPipeline.from_pretrained( "MiniMaxAI/MiniMax-H3-Image", controlnet=controlnet ) - VideoX Fun
How to use akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0 with VideoX Fun:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Download README.md from akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0: direct link, hf CLI and curl.
- Browser
- Download file 12.4 kB
-
https://huggingface.co/akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0/resolve/main/README.md
- Command line
-
hf download hf://akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0/README.md
-
curl -L -o README.md https://huggingface.co/akhaliq/MiniMax-H3-Image-Fun-Controlnet-Union-2.0/resolve/main/README.md
license: other
license_name: minimax-h3-community-license-agreement
license_link: LICENSE
library_name: diffusers
base_model:
- MiniMaxAI/MiniMax-H3-Image
- alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0
base_model_relation: merge
tags:
- minimax-h3
- videox_fun
- controlnet
- controlnet-union
- text-to-image
- image-to-image
- ml-intern
MiniMax-H3 + Fun ControlNet-Union 2.0 β video-model control, validated
TL;DR: this repo is the ControlNet-Union-2.0 branch working with the MiniMax-H3 video model β all 9 official demos reproduced, every condition validated, a ComfyUI workflow, and a repeated-frame recipe that turns the video stack into a still-image controlnet (canny/gray/HED).
What we tried that did NOT work: driving the MiniMax-H3-Image
model (the fast 6-step PDD-distilled image DiT) with this branch. The runtime wiring is verified
mathematically exact (scale-0 output is bit-identical to base), but the video-trained control
blocks collapse the spatial signal on the image model's distilled trajectory β it perturbs,
never steers (full evidence: assets/diag*, assets/image-dit-official/).
Future work: fine-tune the control branch on the H3-Image DiT (frozen DiT, zero-gated start β
all machinery in merge/).
Contents: MiniMaxAI/MiniMax-H3-Image packaged with the alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0 control branch (8 union conditions + inpainting), plus verified, reproducible scripts and samples.
Showcase: extreme poses β man in a suit
Each row is one extreme pose; the task is the same pose as a man in a tailored suit in a new
setting. Columns, left to right: the text-to-video source pose (woman in athletic wear); its
canny map; the H3 video model with the source image as a keyframe reference (no
ControlNet); the ControlNet-Union-2.0 video stack (22-frame repeated canny, 40 steps,
guidance_scale=1.0, seed 904231, control scale 1.0 on row 1, 0.6 on rows 2β4); and H3-Image
with the source image as a <Picture 1> reference. With a reference image both baselines
approximate the poses β H3-Image's multi-reference editing is genuinely strong β but only the
ControlNet column matches the exact geometry every row (compare row 2's one-arm handstand
arm/leg angles). Identity and clothing follow the prompt at control scale 0.6. Variants
(0.6/0.8/1.0, animated control) and per-column frames:
assets/showcase2/variants/, assets/showcase2/.
Skeleton (pose) control on stills: partial, honestly
Same task (single-person poses β man in a suit, generic prompt, same seed). Columns: dataset
person, pose skeleton input, H3 video + keyframe reference (no ControlNet),
ControlNet-Union-2.0 (22-frame repeated skeleton, scale 1.0), H3-Image + <Picture 1>
reference. Read it row by row: static skeletons steer only coarse posture classes β the
seated and kneeling rows produce sitting men while every baseline stands β but the lunge rows
are ignored, and on those rows H3-Image's reference editing actually lands the crouch better
than the sparse skeleton. The video keyframe path mostly ignores the reference or copies its
identity instead. For dense, reliable still-image control, canny remains the working
condition (grid above); real pose control wants motion (see the official dancer demo below)
or the v1-based P-H3 pack. Full-res cells: assets/showcase3/.
Official card demos, reproduced
All 9 official demos from the upstream card, reproduced end-to-end: the official control
videos (alibaba-pai/.../asset/)
at exact card settings β 40 steps, guidance_scale=1.0, control_context_scale=1.0, seed 43,
704Γ1280 budget, 24 fps β one H200, one model load
(merge/card_repro_job.py). The upstream card publishes no prompts, so
ours are scene-matched. Mid-frames shown; compare with the official result videos on the
upstream card.
First frames, extracted control maps and every sample: assets/card-repro/.
More samples (same-seed on/off proof, static-frame coverage, diagnostics): assets/.
Reproduce / use
- All 9 demos: run
merge/card_repro_job.pyon an H200 (~1 h incl. downloads). - Your own control:
merge/expC_video_pipeline.pyβ repeat a control image into a 22-frame video and run the VideoX-Fun control pipeline; any output frame is a controlled image (~8 min on an H200). - All 8 conditions from one photo:
merge/coverage_job.py.
ComfyUI
ComfyUI β₯ 0.35 ships the core Apply MiniMax H3 Fun ControlNet node β no custom nodes
needed. Drag comfyui/workflows/minimax-h3-fun-controlnet-union-2.0.json
onto the canvas (the official template, adapted to load the 2.0 union patch).
| File | Goes in |
|---|---|
minimax_h3_ref2va_pruned_int8_convrot.safetensors (19.5 GB) |
models/diffusion_models/ |
minimax_h3_fun_controlnet_union_2.0_pruned_int8_convrot.safetensors (4.5 GB) β the 2.0 patch (bf16 also available) |
models/model_patches/ |
qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (14.6 GB) |
models/text_encoders/ |
minimax_h3_video_vae_int8_convrot.safetensors + minimax_h3_audio_vae_fp32.safetensors |
models/vae/ |
Usage notes:
- Weights caveat: ComfyUI's pruned bases need the curve-form patches from Comfy-Org (table
above). The checkpoint in this repo's
controlnet/is the original full-width AdaLN release β use it with the VideoX-Fun pipeline (merge/scripts), not with ComfyUI's pruned models. strength=control_context_scale(1.0 strongest, 0 off); keep guidance at 1.0. Control is windowable viastart_percent/end_percent.- Frames snap to the 17n+5 grid at 24 fps. For single-image control, repeat the control image into a 22-frame clip β static frames steer for canny/gray/HED (see coverage below); the other conditions want real motion.
- Preprocessors are built in: Canny (Detect Edges), Depth (Depth Anything 3 / MoGe), Pose
(SDPose, wired in the template). Inpainting: connect
mask(+ optionalsource_video).
What's in this repo
controlnet/ ControlNet-Union-2.0 branch (13.6 GB) + config + license
transformer_diffusers/ H3-Image DiT (PDD, 6-step), unmodified
vae_diffusers/, tokenizer/, processor/, scheduler/, modular_model_index.json
merge/ wiring + reproduction + coverage scripts
comfyui/workflows/ ComfyUI template adapted for the 2.0 union patch
assets/ samples: card-repro, validation, diagnostics
Official controls on the H3-Image DiT (negative result)
The same 8 official control frames through the image DiT (runtime wiring,
merge/validate_controlnet.py), scene-matched prompts, seed 43:
outputs are excellent β but driven by the prompt, not the control. Base (no control) vs
controlled at the same seed:
PSNR(base, ctrl) per condition β large changes, none structurally aligned with the control
(consistent with the hint-collapse diagnostics in assets/diag2/):
| canny | depth | hed | mlsd | pose | scribble | layout | gray |
|---|---|---|---|---|---|---|---|
| 10.3 | 14.3 | 13.0 | 11.1 | 19.7 | 11.0 | 13.0 | 14.5 dB |
All 24 images (controls, base, controlled): assets/image-dit-official/.
Notes
- The ControlNet is a runtime branch of the MiniMax-H3 video DiT (10 blocks injecting
zero-gated skips at layers 0,5,β¦,45) β it cannot be folded into the base weights, and the
image model is not a ControlNet out of the box: on the H3-Image DiT (PDD-distilled, 6
steps) the branch attaches but does not steer without adaptation fine-tuning. Evidence and
diagnostics:
assets/validation*,assets/diag*. - For image use via the video pipeline, static repeated-frame controls transfer for canny and gray (HED partially); the other conditions need real moving control videos.
Sources & license
- Base: https://huggingface.co/MiniMaxAI/MiniMax-H3-Image Β· ControlNet: https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0 Β· Code: https://github.com/aigc-apps/VideoX-Fun
- Both upstream models are under the MiniMax-H3 Community License Agreement (
LICENSE,LICENSE.controlnet); the same terms apply to this unofficial community repackaging. Not endorsed by MiniMax or the VideoX-Fun authors.












