Blame for COMFYUI.md

# Viggle Remake: ComfyUI, int8 and GGUF

Part of [Viggle Remake](README.md); the reference sampler `sample.py` and its setup are in the README's Quick start.

## ComfyUI

ComfyUI runs MiniMax-H3 natively. Remake adds a small node pack, `comfyui/ComfyUI-Viggle-Remake`, for
four things it does the way `sample.py` does and stock nodes do not: the text-only presentation of the
references, keyframes pinned at real frame indices, the source soundtrack kept as clean audio, and the
starting noise for a seed. Everything else is stock ComfyUI. Tested with ComfyUI 0.42.0.

### Install

```bash
cd ComfyUI

# the base model, as Comfy-Org's single-file repack (about 124 GB)
hf download Comfy-Org/MiniMax-H3 \
  diffusion_models/minimax_h3_fl2va_bf16.safetensors \
  text_encoders/qwen3vl_32b_minimax_h3_bf16.safetensors \
  vae/minimax_h3_video_vae_fp16.safetensors \
  vae/minimax_h3_audio_vae_fp32.safetensors \
  --local-dir models

# the LoRAs, the nodes, the workflows and the example inputs from this repository
hf download Viggle/Viggle-Remake --include "comfyui/*" "examples/inputs/*" --local-dir ./Viggle-Remake
cp -r Viggle-Remake/comfyui/ComfyUI-Viggle-Remake custom_nodes/
cp Viggle-Remake/comfyui/viggle_remake_*_comfyui.safetensors models/loras/
```

| File | Folder | Node |
|---|---|---|
| `minimax_h3_fl2va_bf16.safetensors` | `models/diffusion_models` | Load Diffusion Model |
| `qwen3vl_32b_minimax_h3_bf16.safetensors` | `models/text_encoders` | Load CLIP, type `minimax` |
| `minimax_h3_video_vae_fp16.safetensors`, `minimax_h3_audio_vae_fp32.safetensors` | `models/vae` | Load VAE, one each |
| `viggle_remake_swap_comfyui.safetensors`, `viggle_remake_edit_comfyui.safetensors` | `models/loras` | LoraLoaderBypassModelOnly |

Use the `fl2va` diffusion model, not `ref2va`: the LoRAs apply to the base `transformer`, the same
weights `sample.py` loads. The two ComfyUI LoRA files hold the same weights as `lora/swap` and
`lora/edit`, repacked to ComfyUI's layer names by [`comfyui/to_comfyui.py`](comfyui/to_comfyui.py).

### Run the examples

Copy the inputs under the names the workflows expect:

```bash
cp Viggle-Remake/examples/inputs/swap-pineapple-mascot/source.mp4 input/remake_swap_source.mp4
cp Viggle-Remake/examples/inputs/swap-pineapple-mascot/ref1.png   input/remake_swap_ref1.png
cp Viggle-Remake/examples/inputs/swap-pineapple-mascot/ref2.png   input/remake_swap_ref2.png
cp Viggle-Remake/examples/inputs/edit-remove-turbines/source.mp4  input/remake_edit_source.mp4
cp Viggle-Remake/examples/inputs/edit-remove-turbines/ref1.png    input/remake_edit_ref1.png
cp Viggle-Remake/examples/inputs/edit-remove-turbines/ref2.png    input/remake_edit_ref2.png
```

Then drag `comfyui/workflows/remake_swap_ui.json` or `remake_edit_ui.json` onto the canvas and queue
it. The `*_api.json` files are the same graphs for ComfyUI's `/prompt` API. Each is the same render as
`examples/demo.sh` with seed 0: 226 frames at 832×480, two references pinned as keyframes, the prompt
from `examples/prompts/` already filled in. The video lands in `output/viggle_remake/`.

The graph:

```
Load Diffusion Model → LoraLoaderBypassModelOnly → Viggle Remake Keyframe Until (0.9) → BasicGuider
Load Video → Get Video Components ─┐
Load Image ×2 ─────────────────────┼→ Viggle Remake Conditioning → BasicGuider, Viggle Remake Noise
Load CLIP, Load VAE ×2 ────────────┘
KSamplerSelect (euler) + ManualSigmas → SamplerCustomAdvanced → VAE Decode + VAE Decode Audio → Create Video (24 fps) → Save Video
```

Two settings change with the mode:

| | swap | edit |
|---|---|---|
| LoRA | `viggle_remake_swap_comfyui.safetensors` | `viggle_remake_edit_comfyui.safetensors` |
| ManualSigmas | `1.0, 0.990476, 0.965243, 0.923077, 0.8, 0.631579, 0.0` | `1.0, 0.983607, 0.96, 0.923077, 0.857143, 0.705882, 0.0` |

### Your own clip

**Viggle Remake Conditioning** takes the same inputs as `sample.py`:

- `source`: the clip's frames. They are used as they come, so convert the clip to 24 fps first
  (`ffmpeg -i in.mp4 -r 24 source.mp4`); `sample.py` does this itself. A clip that is not already on the
  canvas is scaled and centre-cropped onto it.
- `audio`: the soundtrack kept in the output, usually the source's own from Get Video Components.
  Silence when it is not connected.
- `ref_1` … `ref_4`: the reference images, `<Picture 1>` … `<Picture 4>` in the prompt.
- `pins`: one output frame per reference, in order, such as `12, 184` (`--pin`). Leave it empty to
  use the references without keyframes.
- `frames`: the output length, 17n+5 from 90 to 345 (`--frames`). The prompt's timestamps are
  written for it.
- `short_edge`: 480 (832×480 for 16:9) or 736 (1280×736).
- `prompt`: the six-section prompt, used verbatim.

### Speed and memory

Measured on one B200 at 832×480 with 226 frames and two pinned references:

| | ComfyUI | `sample.py` |
|---|---|---|
| One render, models already loaded | about 95 s (6 steps at about 13.5 s each, plus encoding and decoding) | about 58 s |
| First render, including loading | about 2.5–3 min | about 2 min |
| GPU memory | runs at full speed in 80 GB | 87.2 GiB allocated, 91.0 GiB reserved (832×480, 226 frames) |
| System RAM | about 130 GB | |

ComfyUI loads weights onto the GPU as it needs them and keeps the rest in system RAM, so it runs on a
smaller card than `sample.py` does. With ComfyUI capped to 92 and 76 GiB (`--reserve-vram` on a
183 GB card), the card peaked at 92.8 and 76.8 GiB and every step took as long as with no cap. Given a
larger card, it keeps more of the weights resident (120–135 GB on the B200).

### Differences from `sample.py`

- **Load the LoRA with LoraLoaderBypassModelOnly, not LoraLoaderModelOnly.** The plain loader adds
  the LoRA into the bf16 weights, and at bf16 precision much of the update rounds away: on average only 37–73% of
  it survives in the attention and MLP layers. On the swap example the merged render is 7% less sharp
  and further from `sample.py` (21.2 dB against 24.7 dB). The bypass loader keeps the LoRA as a
  separate branch, for 2–3% more time per render.
- **Same seed, same video, not the same pixels.** Viggle Remake Noise draws the same starting noise
  as `sample.py` for a given seed, so the two renders show the same result. They are not bit-identical,
  because the attention kernels and the order of operations differ. On the examples, ComfyUI's frames
  are 41.6 dB (edit) and 24.7 dB (swap) from `sample.py`'s. The swap clip is the sensitive one: in
  ComfyUI alone, a change in the text embeddings the size of bf16 rounding moves it by about as much
  (27 dB). Sharpness is within 3% of `sample.py`, and the pinned keyframes are reproduced to within
  0.2 dB.
- **Slower.** ComfyUI tries flash attention before cuDNN attention, which `sample.py` uses. On B200,
  putting cuDNN first in `SDPA_BACKEND_PRIORITY` in `comfy/ops.py` halves the step time (6.8 s
  against 13.5 s) and the render matches `sample.py` as closely (25.4 and 41.7 dB). We have only
  measured this on B200.
- **Audio.** Viggle Remake Conditioning encodes the soundtrack itself. ComfyUI's stock audio encode
  trims a track to whole 800-sample blocks by cutting from both ends, which shifts it by up to 400
  samples against the video. The node pads the end instead, as `sample.py` does.

## Smaller GPUs: int8 and GGUF

For cards below 80 GB, each mode also comes as a single transformer file for ComfyUI. Each file is the
base `transformer` with that mode's LoRA already merged in, then quantized. It replaces both the base
diffusion model and the LoRA loader in the ComfyUI graph above, so do not load a Remake LoRA on top of it.

The merge is done once, at fp32, and the sum is quantized directly. No bf16 copy is made in between,
so these files do not lose part of the LoRA the way LoraLoaderModelOnly does.

### Files

| file | size | ComfyUI node |
|---|---|---|
| `int8/viggle_remake_swap_pruned_int8_convrot.safetensors` | 21.0 GB | Load Diffusion Model (core) |
| `int8/viggle_remake_edit_pruned_int8_convrot.safetensors` | 21.0 GB | Load Diffusion Model (core) |
| `gguf/viggle_remake_swap-Q8_0.gguf`, `gguf/viggle_remake_edit-Q8_0.gguf` | 22.3 GB each | Unet Loader (GGUF), from [ComfyUI-GGUF](https://github.com/city96/ComfyUI-GGUF) |
| `gguf/viggle_remake_swap-Q6_K.gguf`, `gguf/viggle_remake_edit-Q6_K.gguf` | 17.6 GB each | Unet Loader (GGUF) |
| `gguf/viggle_remake_swap-Q4_K_M.gguf`, `gguf/viggle_remake_edit-Q4_K_M.gguf` | 14.3 GB each | Unet Loader (GGUF) |

- **int8** uses ComfyUI's native int8 format, with the same layout as Comfy-Org's `minimax_h3_fl2va_pruned_int8_convrot`.
  - The four attention and MLP linears in each of the 50 blocks are int8 with one scale per output row (`int8_tensorwise`, `convrot`, group 256).
  - Every other tensor keeps the base file's precision.
  - adaLN is Comfy-Org's compact `pruned` form, unchanged, because the LoRAs do not touch adaLN.
- **GGUF** quantizes the same four linears per block.
  - Q8_0 and Q6_K use that type for all four.
  - Q4_K_M uses Q5_K for the qkv projection, Q6_K for the second MLP layer and Q4_K for the other two. This follows llama.cpp's treatment of the matching layers.
  - These stay unquantized: patch embedding, output layer, adaLN, norms, token refiner and condition projection.
- **The GGUF files carry the `wan` architecture tag.** ComfyUI-GGUF only checks this tag against a list of names it accepts, and ComfyUI identifies the model as MiniMax-H3 from its tensor names.

### Install

```bash
cd ComfyUI

# int8: no extra node needed
hf download Viggle/Viggle-Remake --include "int8/*" --local-dir ./Viggle-Remake
mv Viggle-Remake/int8/*.safetensors models/diffusion_models/

# or GGUF, for example Q6_K
git clone https://github.com/city96/ComfyUI-GGUF custom_nodes/ComfyUI-GGUF
pip install -r custom_nodes/ComfyUI-GGUF/requirements.txt
hf download Viggle/Viggle-Remake --include "gguf/*Q6_K*" --local-dir ./Viggle-Remake
mv Viggle-Remake/gguf/*.gguf models/diffusion_models/
```

The text encoder, the VAEs, the node pack and the inputs are the same as in the ComfyUI section.

**Change two nodes in the workflow.**
1. Remove Load Diffusion Model and LoraLoaderBypassModelOnly.
2. Add Load Diffusion Model with the int8 file, or Unet Loader (GGUF) with a `.gguf` file.
3. Connect its MODEL output to Viggle Remake Keyframe Until.

Leave everything else as it is. Use the swap file with the swap sigmas and the edit file with the edit sigmas.

### Which file for which card

We measured on B200s. A second process held the rest of the card, so ComfyUI saw only the stated amount of free memory. The test was the edit example at 226 frames with two pinned references and the bf16 text encoder. ComfyUI ran with default settings unless a flag is shown.

| card | 832×480 | 1280×736 |
|---|---|---|
| 12 GB | Q4_K_M or Q8_0 with `--reserve-vram 8`. int8: 124 frames only | not tested |
| 16 GB | int8. Q4_K_M or Q6_K with `--reserve-vram 6` | Q4_K_M with `--reserve-vram 10`: 124 frames. At 226 frames it aborted even with `--reserve-vram 12` |
| 24 GB | int8 or Q4_K_M. Q6_K or Q8_0 with `--reserve-vram 6` | Q4_K_M or Q8_0 with `--reserve-vram 20`. int8: 124 frames only |
| 32 GB | int8 or Q8_0 | Q8_0 with `--reserve-vram 20`. int8: up to 175 frames |
| 48 GB | any | Q8_0. int8 runs out of memory at 226 frames |
| 80 GB | bf16 + LoRA, as in the ComfyUI section | bf16 + LoRA. int8 runs out of memory at 226 frames |

- **At 832×480 on 16 GB or more, start with int8.**
  - On these examples it scores the same as Q8_0, and it needs no flags.
  - ComfyUI loads it with dynamic VRAM: it moves each weight onto the card when it is needed and keeps the rest in system RAM.
- **For 12 GB, or for 1280×736 below 80 GB, use a GGUF file with `--reserve-vram`.** With these settings ComfyUI keeps the weights in system RAM and streams them to the card every step. That costs little time:
  - at 832×480, 15.9 s per step against about 14 s with the weights on the card;
  - at 1280×736, 67–72 s per step against 64 s.
- **Why GGUF needs the flag.** ComfyUI-GGUF puts as much of the model on the card as ComfyUI's memory estimate allows, and for MiniMax-H3 that estimate leaves out the reference and keyframe tokens.
  - When the weights fit but the activations do not, ComfyUI aborts with `CUDA error: invalid argument` instead of falling back to offloading.
  - `--reserve-vram N` keeps N GB free for the activations. If a run aborts, raise N.
- **Why int8 stops at 1280×736 × 226 frames.** Without a CUDA 13 build of PyTorch, comfy-kitchen falls back to a plain int8 matmul. That matmul keeps 32-bit sums for the whole layer and assembles the output in pieces, so one layer briefly needs several times the memory of its output. At 1280×736 × 226 frames this ran out of memory even with 80 GB free, while bf16 + LoRA fit in 80 GB and Q8_0 ran in 24 GB with `--reserve-vram 20`. We have not tested a CUDA 13 build.
- **Q4_K_M is the smallest file.** It is visibly further from the reference than int8 (see Quality).

### Quality

Each file rendered both examples with seed 0 at 832×480 and 226 frames. We compared each render with
`sample.py`'s bf16 + LoRA render of the same inputs. The first comparison runs both in the same diffusers
pipeline, so only the weights differ. PSNR is over every frame. Sharpness is the variance of the Laplacian,
as a percentage of the reference's.

| weights | swap, pinned | swap, refs only | edit, pinned | edit, refs only | sharpness |
|---|---|---|---|---|---|
| bf16 + LoRA (reference) | — | — | — | — | 100% |
| int8 | 27.2 dB | 34.0 dB | 46.8 dB | 49.7 dB | 100–104% |
| Q8_0 | 26.9 dB | 36.6 dB | 49.8 dB | 52.0 dB | 99–102% |
| Q6_K | 24.6 dB | 30.5 dB | 44.6 dB | 46.0 dB | 99–102% |
| Q4_K_M | 20.6 dB | 23.8 dB | 38.3 dB | 40.7 dB | 97–109% |
| *for scale: same bf16 weights, rounded differently* | *29.6 dB* | | *50.4 dB* | | *101%* |

- **Pinned** is the way the examples run. **Refs only** drops the keyframe pins, so the LoRA alone has to carry the edit. It is the harder test for a quantized file.
- **Every file makes the edit**: the turbines are removed and the pizza maker is replaced. Sharpness stays between 3% below and 9% above the reference.
- **The swap clip is sensitive.** The last row is the same model with its bf16 weights rounded a second, slightly different way, and that change alone moves the swap render to 29.6 dB. Read swap differences of a few dB as noise. The edit column separates the formats more clearly.
- **In ComfyUI** we compared each file with the LoRA applied through LoraLoaderBypassModelOnly, on the same graph:
  - int8 scored 24.0 dB on swap and 46.5 dB on edit.
  - Q8_0 scored 23.8 / 48.9.
  - Q6_K scored 20.0 / 43.6.
  - Q4_K_M scored 20.0 / 38.5.
  - For comparison, LoraLoaderModelOnly, which merges the LoRA at bf16, scored 22.0 / 46.1.
  - Contact sheets of every render match the reference by eye.
- **At 1280×736** (edit, 226 frames, in ComfyUI against bypass-LoRA bf16), Q8_0 scored 42.8 dB and Q4_K_M 31.6 dB. Sharpness was 101% and 98% of the reference.
- **Card size and offloading do not change the output.** The same file and seed gave the same frames on a 12 GB budget as with the whole card free.

### Text encoder

MiniMax-H3 uses the first 50 layers of Qwen3-VL 32B, and Comfy-Org's text-encoder files contain only those layers. They come in three versions:
- `qwen3vl_32b_minimax_h3_bf16` (51.5 GB);
- `_int8_convrot` (27.1 GB);
- `_nvfp4_awq` (15.7 GB).

ComfyUI runs the encoder once, before sampling, and streams it through the card, so it does not decide which card you need. Every result in the card table used the bf16 encoder, except the 12 GB int8 runs, which used int8.

A smaller encoder saves disk and loading time. On 16 GB, the first render took 144 s with nvfp4 against 192 s with bf16.

We rendered both examples with each encoder and the int8 transformer, and compared each render with the bypass-LoRA render:

| encoder | edit | swap |
|---|---|---|
| bf16 | 46.5 dB | 24.0 dB |
| int8 | 47.0 dB | 25.0 dB |
| nvfp4 | 46.7 dB | 23.1 dB |

Neither smaller encoder costs anything we can measure. Community GGUF versions of the encoder exist, but we have not tested them.

### Speed and memory

All times are from one B200.

- **832×480, 226 frames.**
  - With the whole file on the card, a step takes about 14 s for every file, so the six steps take about 85 s.
  - With the weights streamed from RAM, a step takes 15.9–17.4 s.
  - A first render, which includes loading the bf16 text encoder from local NVMe, takes 170–210 s from queue to saved frames.
- **1280×736, 226 frames.** A step takes 64–72 s for every file and for bf16 + LoRA. A render takes 510–585 s.
- **We have not timed consumer cards.** Expect them to be slower.
- **System RAM.** We measured the peak resident memory of the ComfyUI process. Memory-mapped weight files count toward it.
  - 832×480: 47–55 GB, whichever encoder.
  - 1280×736 with an int8 or GGUF file: 55–80 GB.
  - 1280×736 with bf16 + LoRA: 127 GB.
- **int8 ran on comfy-kitchen's fallback path.** It ran on a PyTorch built for CUDA 12.8. There, comfy-kitchen's optimized int8 kernels are disabled and ComfyUI logs "You need pytorch with cu130 or higher". The fallback path gives correct output. We have not measured the optimized kernels.

### Caveats

- **Tested with** ComfyUI 0.42.0, ComfyUI-GGUF at commit `6ea2651`, PyTorch 2.9.1+cu128 and comfy-kitchen 0.2.40, on B200s only.
- **The quality numbers come from the two example clips.** The card table comes from the edit example only.
- **The small cards were emulated.** ComfyUI still reported the B200's full 183 GB as total VRAM, so it ran in its normal-VRAM mode. On a real 12–24 GB card, ComfyUI may pick its low-VRAM defaults by itself, and the flags above may need adjusting.
- **GGUF runs log a harmless warning**, "Mismatch dtype between input and weight … Cannot dispatch to fused implementation". It comes from the norm weights, which are kept at F32.