akhaliq's picture
akhaliq HF Staff
Card: skeleton pose-control grid with honest partial verdict
cbd6d3f verified
|
Raw History Blame Contribute Delete
12.4 kB
metadata
license: other
license_name: minimax-h3-community-license-agreement
license_link: LICENSE
library_name: diffusers
base_model:
  - MiniMaxAI/MiniMax-H3-Image
  - alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0
base_model_relation: merge
tags:
  - minimax-h3
  - videox_fun
  - controlnet
  - controlnet-union
  - text-to-image
  - image-to-image
  - ml-intern

MiniMax-H3 + Fun ControlNet-Union 2.0 β€” video-model control, validated

TL;DR: this repo is the ControlNet-Union-2.0 branch working with the MiniMax-H3 video model β€” all 9 official demos reproduced, every condition validated, a ComfyUI workflow, and a repeated-frame recipe that turns the video stack into a still-image controlnet (canny/gray/HED).

What we tried that did NOT work: driving the MiniMax-H3-Image model (the fast 6-step PDD-distilled image DiT) with this branch. The runtime wiring is verified mathematically exact (scale-0 output is bit-identical to base), but the video-trained control blocks collapse the spatial signal on the image model's distilled trajectory β€” it perturbs, never steers (full evidence: assets/diag*, assets/image-dit-official/).

Future work: fine-tune the control branch on the H3-Image DiT (frozen DiT, zero-gated start β€” all machinery in merge/).

Contents: MiniMaxAI/MiniMax-H3-Image packaged with the alibaba-pai/MiniMax-H3-Fun-Controlnet-Union-2.0 control branch (8 union conditions + inpainting), plus verified, reproducible scripts and samples.

Showcase: extreme poses β†’ man in a suit

pose showcase grid

Each row is one extreme pose; the task is the same pose as a man in a tailored suit in a new setting. Columns, left to right: the text-to-video source pose (woman in athletic wear); its canny map; the H3 video model with the source image as a keyframe reference (no ControlNet); the ControlNet-Union-2.0 video stack (22-frame repeated canny, 40 steps, guidance_scale=1.0, seed 904231, control scale 1.0 on row 1, 0.6 on rows 2–4); and H3-Image with the source image as a <Picture 1> reference. With a reference image both baselines approximate the poses β€” H3-Image's multi-reference editing is genuinely strong β€” but only the ControlNet column matches the exact geometry every row (compare row 2's one-arm handstand arm/leg angles). Identity and clothing follow the prompt at control scale 0.6. Variants (0.6/0.8/1.0, animated control) and per-column frames: assets/showcase2/variants/, assets/showcase2/.

Skeleton (pose) control on stills: partial, honestly

skeleton showcase grid

Same task (single-person poses β†’ man in a suit, generic prompt, same seed). Columns: dataset person, pose skeleton input, H3 video + keyframe reference (no ControlNet), ControlNet-Union-2.0 (22-frame repeated skeleton, scale 1.0), H3-Image + <Picture 1> reference. Read it row by row: static skeletons steer only coarse posture classes β€” the seated and kneeling rows produce sitting men while every baseline stands β€” but the lunge rows are ignored, and on those rows H3-Image's reference editing actually lands the crouch better than the sparse skeleton. The video keyframe path mostly ignores the reference or copies its identity instead. For dense, reliable still-image control, canny remains the working condition (grid above); real pose control wants motion (see the official dancer demo below) or the v1-based P-H3 pack. Full-res cells: assets/showcase3/.

Official card demos, reproduced

All 9 official demos from the upstream card, reproduced end-to-end: the official control videos (alibaba-pai/.../asset/) at exact card settings β€” 40 steps, guidance_scale=1.0, control_context_scale=1.0, seed 43, 704Γ—1280 budget, 24 fps β€” one H200, one model load (merge/card_repro_job.py). The upstream card publishes no prompts, so ours are scene-matched. Mid-frames shown; compare with the official result videos on the upstream card.

Condition Control Reproduced frame
Canny astronaut edges g
Depth dog + cat depth g
HED turtle edges g
MLSD architectural lines g
Pose dancer skeleton g
Scribble 3-student sketch g
Layout colored boxes g
Gray coral luminance g
Inpaint source + mask g

First frames, extracted control maps and every sample: assets/card-repro/. More samples (same-seed on/off proof, static-frame coverage, diagnostics): assets/.

Reproduce / use

ComfyUI

ComfyUI β‰₯ 0.35 ships the core Apply MiniMax H3 Fun ControlNet node β€” no custom nodes needed. Drag comfyui/workflows/minimax-h3-fun-controlnet-union-2.0.json onto the canvas (the official template, adapted to load the 2.0 union patch).

File Goes in
minimax_h3_ref2va_pruned_int8_convrot.safetensors (19.5 GB) models/diffusion_models/
minimax_h3_fun_controlnet_union_2.0_pruned_int8_convrot.safetensors (4.5 GB) β€” the 2.0 patch (bf16 also available) models/model_patches/
qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (14.6 GB) models/text_encoders/
minimax_h3_video_vae_int8_convrot.safetensors + minimax_h3_audio_vae_fp32.safetensors models/vae/

Usage notes:

  • Weights caveat: ComfyUI's pruned bases need the curve-form patches from Comfy-Org (table above). The checkpoint in this repo's controlnet/ is the original full-width AdaLN release β€” use it with the VideoX-Fun pipeline (merge/ scripts), not with ComfyUI's pruned models.
  • strength = control_context_scale (1.0 strongest, 0 off); keep guidance at 1.0. Control is windowable via start_percent/end_percent.
  • Frames snap to the 17n+5 grid at 24 fps. For single-image control, repeat the control image into a 22-frame clip β€” static frames steer for canny/gray/HED (see coverage below); the other conditions want real motion.
  • Preprocessors are built in: Canny (Detect Edges), Depth (Depth Anything 3 / MoGe), Pose (SDPose, wired in the template). Inpainting: connect mask (+ optional source_video).

What's in this repo

controlnet/                 ControlNet-Union-2.0 branch (13.6 GB) + config + license
transformer_diffusers/      H3-Image DiT (PDD, 6-step), unmodified
vae_diffusers/, tokenizer/, processor/, scheduler/, modular_model_index.json
merge/                      wiring + reproduction + coverage scripts
comfyui/workflows/          ComfyUI template adapted for the 2.0 union patch
assets/                     samples: card-repro, validation, diagnostics

Official controls on the H3-Image DiT (negative result)

The same 8 official control frames through the image DiT (runtime wiring, merge/validate_controlnet.py), scene-matched prompts, seed 43: outputs are excellent β€” but driven by the prompt, not the control. Base (no control) vs controlled at the same seed:

base_canny ctrl_canny
b c

PSNR(base, ctrl) per condition β€” large changes, none structurally aligned with the control (consistent with the hint-collapse diagnostics in assets/diag2/):

canny depth hed mlsd pose scribble layout gray
10.3 14.3 13.0 11.1 19.7 11.0 13.0 14.5 dB

All 24 images (controls, base, controlled): assets/image-dit-official/.

Notes

  • The ControlNet is a runtime branch of the MiniMax-H3 video DiT (10 blocks injecting zero-gated skips at layers 0,5,…,45) β€” it cannot be folded into the base weights, and the image model is not a ControlNet out of the box: on the H3-Image DiT (PDD-distilled, 6 steps) the branch attaches but does not steer without adaptation fine-tuning. Evidence and diagnostics: assets/validation*, assets/diag*.
  • For image use via the video pipeline, static repeated-frame controls transfer for canny and gray (HED partially); the other conditions need real moving control videos.

Sources & license