docs: fix stale comments (20GB→1.2GB, startup merge→on-demand), update codemap.md with turbo
721686d verified | # Repository Atlas: minimax-h3-repo | |
| ## Project Responsibility | |
| The denoising half of a **two-Space MiniMax-H3 deployment** on Hugging Face ZeroGPU: joint video + synchronized soundtrack generation, unquantized bf16. The 195.9 GiB checkpoint cannot fit one ZeroGPU Space (150 GB storage cap), so `MiniMaxH3Blocks` is cut at its `text_encoder` step. This repo holds the transformer + two autoencoders; the 62 GiB Qwen3-VL conditioner runs in the sibling `multimodalart/qwen3vl-conditioner` Space and is called over the gradio API on every request. | |
| ## System Entry Points | |
| - `app.py` — Gradio UI + request orchestration. **Runs `load_models()` at import** (77.3 GB startup load on CPU). Entrypoint per README frontmatter (`app_file: app.py`). | |
| - `requirements.txt` — Dependency manifest. **`diffusers` comes from a pinned WIP PR commit** (`refs/pull/14371/head` @ `665f5782`), not PyPI. | |
| - `README.md` — Space config (frontmatter `sdk_version` must match `gradio==6.20.0`) + design rationale and measured performance. | |
| ## File Map | |
| | File | Responsibility | Detailed Notes | | |
| |------|---------------|----------------| | |
| | `app.py` | Request flow + Gradio UI | `generate` → remote `/encode` → `@spaces.GPU _generate` → `encode_video` mux. Owns `CANVASES` (cross-Space contract), `snap_frames` (17n+5), placement modes (`pack`/`lazy`/`offload`), `get_duration` GPU budget estimator. Turbo LoRA: A/B factors stored on CPU (~1.2 GB), merged on-the-fly via `B @ A` inside `@spaces.GPU`. | | |
| | `h3_split_blocks.py` | Split pipeline definition | `SequentialPipelineBlocks` subclasses for both halves (`Conditioner`/`Generator`) and both partitions (`t2va`/`ref2va`). Defines the wire-format outputs (`prompt_embeds` + `text_token_tags` + resolved height/width/num_frames). | | |
| | `h3_aoti.py` | AoTI acceleration | `maybe_load`/`patch_blocks` = runtime path (strict key `bf16/torch2.11/sm120/dynamic`, else eager). `export_block`/`compile_and_save`/`upload` = build path for the separate `multimodalart/minimax-h3-aoti` repo (never called by the Space). | | |
| | `spaces_constant_binding_patch.py` | AoTI constant binding | Compile-side (`register_loose_tensors`, `write_constant_aliases`) + load-side (`apply_spaces_constant_binding_patch`) glue preventing SIGSEGV from anonymously-lifted export constants. | | |
| | `requirements.txt` | Pip dependencies | All pins deliberate — see notes in file. | | |
| | `packages.txt` | Space apt packages | `ffmpeg` (ZeroGPU system-level, not pip). | | |
| | `.gitattributes` | HF LFS rules | Standard Space LFS filters for model artifacts. | | |
| | `AGENTS.md` | Agent instructions | Critical rules, dependency pins, verification steps. | | |
| ## Data & Control Flow | |
| 1. Browser/client calls `generate(prompt, image?, last_image?, canvas, duration, steps, seed, upsample, turbo)`. | |
| 2. `snap_frames` aligns the duration to `17*n + 5` frames at 24 fps; if `turbo`, steps forced to 7. `encode_remote` calls the conditioner's `/encode` over `gradio_client` (forwarding the caller's own ZeroGPU token). | |
| 3. Conditioner returns one safetensors file carrying `prompt_embeds` `(1, num_text_tokens, 5120)` bf16 + `text_token_tags` int64, with resolved `height`/`width`/`num_frames` in metadata + the plan. | |
| 4. `_generate` (decorated `@spaces.GPU(duration=get_duration, size=xlarge)`) runs the packed-sequence denoise loop + two decoders on GPU. If turbo, A/B factors are moved to CUDA (~1.2 GB) and LoRA is merged/unmerged around the pipeline call. Returns only `(video frames, audio, sampling_rate)` — a `@spaces.GPU` return crosses a process boundary by pickling, so pipeline state stays on the card. | |
| 5. `encode_video` (PyAV) muxes h264 frames + stereo AAC soundtrack into an MP4 in temp storage. | |
| ## Integration Points | |
| - **Cross-Space wire contract**: `CANVASES` label strings in `app.py` go over the wire and must stay identical to the conditioner's table; the wire format (two tensors + metadata) must stay in lockstep. Changing either breaks the split. | |
| - **AoTI package repo**: `multimodalart/minimax-h3-aoti` — consumed at runtime via `snapshot_download` when `H3_AOTI=1`; produced by this repo's build path (requires `HF_TOKEN` + same-arch GPU). | |
| - **Model repo**: `MiniMaxAI/MiniMax-H3` (public diffusers checkpoint, subfolders `transformer/` + `vae/` + `audio_vae/`). | |
| - **Conditioner Space**: `multimodalart/qwen3vl-conditioner` — remote `/encode` endpoint, billed to the requesting user's token. | |
| - **diffusers upstream**: `huggingface/diffusers` PR #14371 — the pinned WIP commit whose block classes `h3_split_blocks.py` subclasses. | |