Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,150 @@
+---
+license: other
+license_name: minimax-h3-community-license-agreement
+license_link: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE
+base_model:
+  - MiniMaxAI/MiniMax-H3
+  - Comfy-Org/MiniMax-H3
+base_model_relation: merge
+library_name: diffusers
+pipeline_tag: image-text-to-video
+tags:
+  - minimax-h3
+  - modular-diffusers
+  - ref2va
+  - fl2va
+  - merge
+  - synchronized-audio-video
+  - experimental
+inference: false
+---
+
+# MiniMax-H3, one transformer for references *and* keyframes
+
+MiniMax-H3 ships **two** 37.5 GB transformer partitions: `fl2va` for first/last-frame conditioning and `ref2va`
+for reference conditioning. This is **one** checkpoint that serves both, so a deployment holds 37.5 GB instead of
+75 GB. It is the pruned `fl2va` partition with a rank-1024 approximation of the `ref2va − fl2va` weight delta
+**fused into the weights**.
+
+> [!WARNING]
+> Experimental and mechanically derived — the delta was extracted from two released checkpoints by SVD, not
+> trained. This is **not** `transformer_ref`: against the real thing it reaches video-latent cosine 0.875 / 0.897 /
+> 0.691 on three matched reference requests, short of the 0.99 that would make it a drop-in replacement. Use it when
+> one-partition deployment is worth that gap.
+
+> [!IMPORTANT]
+> Governed by the [MiniMax H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE),
+> which carries territory exclusions and redistribution conditions. The upstream `LICENSE` is included and this card
+> is the modification notice.
+
+## Inference
+
+Both workflows resolve to the weights in this repo, and each loads only its own slot:
+
+```python
+import torch
+from diffusers import ComponentsManager, ModularPipeline
+
+REPO = "diffusers-modular/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024"
+
+manager = ComponentsManager()
+pipe = ModularPipeline.from_pretrained(REPO, workflow="ref2va", components_manager=manager,
+                                      trust_remote_code=True)
+pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
+manager.enable_auto_cpu_offload(device="cuda")   # 37.5 GB DiT + 62 GB conditioner + two VAEs
+```
+
+**References** (`workflow="ref2va"`) — image, video and/or audio, in the order they are passed:
+
+```python
+from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3ImageReference
+
+out = pipe(
+    prompt="The woman from the reference stands in a snowy park at dusk, catching snowflakes and laughing.",
+    references=[MiniMaxH3ImageReference.from_file("subject.png")],
+    height=544, width=960, num_frames=124, num_inference_steps=20,
+    generator=torch.Generator("cpu").manual_seed(42),
+)
+```
+
+**Keyframes** (`workflow="fl2va"`) — a first and/or last frame:
+
+```python
+from PIL import Image
+
+out = pipe(                       # same repo, reloaded with workflow="fl2va"
+    prompt="She lifts the teacup and takes a slow sip, then smiles; quiet kitchen room tone.",
+    image=Image.open("first.png"), last_image=Image.open("last.png"),
+    height=544, width=960, num_frames=124, num_inference_steps=20,
+    generator=torch.Generator("cpu").manual_seed(42),
+)
+video, audio, rate = out["videos"][0], out["audio"][0], out["sampling_rate"]
+```
+
+One generation carries **one** shape: the blocks dispatch on their inputs and `references` wins, so passing both
+silently drops the keyframes.
+
+### Few-step generation — keep the turbo LoRA live
+
+```python
+pipe.load_lora_weights("larryvrh/MiniMax-H3-Turbo-Lora", adapter_name="turbo",
+                       load_into_transformer_ref=True)   # `load_into_transformer_ref` only for ref2va
+```
+
+**Do not fuse it.** A distill LoRA's deltas are 2–5e-4 of the weights they modify, and folding them into bf16 keeps
+only **0.67–0.79** — a third of turbo's AdaLN modulation is lost to rounding. Left live it costs ~7–15% wall time.
+For the same reason, an AoT-compiled transformer cannot be combined with a live adapter: the graph is captured from
+the base modules and runs straight past it.
+
+Requires `diffusers` with `MiniMaxH3LoraLoaderMixin` (PR
+[#14408](https://github.com/huggingface/diffusers/pull/14408)) and `trust_remote_code=True` — the pruned layout
+ships a `MiniMaxH3PrunedTransformer3DModel` whose AdaLN is 8 wide, where the stock class expects 2688.
+
+## How it was made
+
+1. **Take the difference between the twins.** `ref2va − fl2va`, tensor by tensor. Both partitions are architecturally
+   identical, so this is what makes one able to use references.
+2. **Compress it.** [ethanfel's](https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental)
+   randomized-SVD extraction at **rank 1024** (9.4 GB), applied at strength 1.0.
+3. **Apply the parts that cannot be compressed exactly.** 267 patches: 211 RMSNorm deltas, 56 biases, and
+   `adaln_t_table` — the timestep coordinate table, which differs between the partitions and which the earlier
+   rank-256 extractions omit.
+4. **Fuse.** `fuse_lora`, then unload the adapter, leaving an ordinary checkpoint. The delta is 0.02–1.9 of the
+   weights it lands in, so it survives a bf16 fold at 1.00 (unlike a distill LoRA — see above).
+5. **Keep the AdaLN affine map.** `adaln_basis` / `adaln_mean` ship as buffers, so LoRAs trained on the released
+   2688-wide AdaLN still project onto this 8-wide one.
+
+Verified by re-downloading this repo and generating: bit-identical (`torch.equal` on video and audio latents) to the
+local build it was made from.
+
+## Measured
+
+Video-latent cosine against the true `ref2va` partition, generated on the same GPU (identical weights on a
+different GPU only agree to 0.959–0.987, so cross-machine anchors are not usable at this precision). Three
+seed-matched requests: photoreal image reference, stylized image reference, video reference.
+
+| applied to pruned FL2VA | size | photoreal | stylized | video ref |
+|---|---|---|---|---|
+| **rank-1024 delta (this repo)** | 9.4 GB | **0.875** | **0.897** | **0.691** |
+| rank-256 delta | 2.4 GB | 0.772 | 0.798 | 0.527 |
+| AdaLN-only, exact | 95 MiB | 0.678 | 0.803 | 0.505 |
+| nothing | 0 | 0.668 | 0.795 | 0.480 |
+
+Rank is what matters, and the bulk is load-bearing: the exact AdaLN half — the tempting 95 MiB shortcut — lands on
+the no-delta baseline, so the trunk attention and MLP deltas are carrying the result.
+
+**Keyframes still work.** Against the *pristine* `fl2va` partition on the same first+last-frame request, this
+checkpoint reaches video cosine **0.930** and is visually indistinguishable. Audio is the weaker half (0.662, and
+it runs louder). Few-step, with turbo live: 64–179 s per 5 s clip against 174–478 s for 20 steps on the true
+partition, at the VRAM of a single partition.
+
+## Credit
+
+Delta extraction [ethanfel](https://huggingface.co/ethanfel/MiniMax-H3-Pruned-Ref2VA-Delta-LoRAs-Experimental);
+approach established by [Kijai](https://huggingface.co/Kijai/MiniMax-H3-experimental); pruned repack
+[Comfy-Org](https://huggingface.co/Comfy-Org/MiniMax-H3); turbo LoRA
+[larryvrh](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora); base model and license
+[MiniMaxAI](https://huggingface.co/MiniMaxAI/MiniMax-H3). Related community work on the same question:
+[lihaoyun6](https://huggingface.co/lihaoyun6/MiniMax-H3-Ref-Patch) (exact-diff-only patch),
+[smhfacct](https://huggingface.co/smhfacct/Minimax-H3-fl2va-ref2va-hybrid-models) (AdaLN block swap),
+[PulpCut](https://huggingface.co/PulpCut/MiniMax-H3-Ref2VA-Turbo-INT8-ConvRot) (turbo merged into Ref2VA).