ClipProj 4B encoder variant: 23.47 GB -> 13.16 GB, 2x faster, still four bits everywhere

README.md

@@ -61,7 +61,9 @@ not ported.
 
 | file | size | |
 |---|---|---|
-| `mmh3-turbo-fl2va-q4tp.cmf` | 23.94 GB | **use this one** |
+| `mmh3-turbo-fl2va-q4tp.cmf` | 23.94 GB | **use this one** — text-to-video AND keyframe-to-video |
+| `mmh3-turbo-q4tp.cmf` | 23.47 GB | the same without the vision tower: text-to-video only |
+| `mmh3-turbo-clipproj4b-q4tp.cmf` | **13.16 GB** | text-to-video with a 4B stand-in prompt encoder. Peaks at 15.1 GB of VRAM, so a 20 GB card holds the whole run; 2x faster, and still four bits everywhere. Costs some set dressing — see below |
 | `mmh3-turbo-fl2va-q2tp.cmf` | 18.74 GB | two bits on the gate/up planes. Smaller, faster, and it stops following the prompt — kept for anyone who wants to push on it, not for rendering. See below |
 
 ## Keyframe to video
@@ -194,6 +196,11 @@ RAM at least the file's size — 24 GB — or every step faults on non-resident
 pages; the weights are memory-mapped, not read. Disk: 24 GB. A GPU is optional
 and wants ~14 GB of VRAM for the DiT's planes. No network access at run time.
 
+`mmh3-turbo-clipproj4b-q4tp.cmf` asks for 14 GB of RAM and disk instead, and
+peaks at 15.1 GB of VRAM — a whole-run maximum, polled, not a snapshot. That is
+what makes it the one to reach for on a 20 GB card: below that the prompt
+encoder pages, and on this workload paging costs more than any kernel.
+
 ## Two bits: smaller, faster, and answering a different question
 
 The obvious next cut is the DeepSeek-V4 policy: gate/up at two bits,
@@ -230,6 +237,121 @@ measuring next — the packer's policy is one predicate,
 that experiment starts from something rather than nothing; four bits is
 what to render with.
 
+## A smaller encoder instead of fewer bits
+
+The two-bit section above ends on a diagnosis: half this file is the PROMPT
+ENCODER, and squeezing it is what broke the prompt. There is a second way to act
+on that diagnosis — **don't compress the encoder, replace it.**
+
+[ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj) fits a ridge
+regression from a small Qwen3-VL's hidden state into the space the DiT was
+conditioned on. Same tokenizer, same family, one affine map:
+
+```text
+cond = ((h - mean_in) / std_in) @ W [+ GELU residual] * std_out + mean_out
+```
+
+`mmh3-turbo-clipproj4b-q4tp.cmf` is the text-to-video file with Qwen3-VL-4B
+tapped at layer 24 standing in for the 32B tapped at 50, plus the 304 MB
+projection kept **exact** — f32, no quantization on the piece that carries the
+whole substitution. Everything else is copied through byte for byte: DiT and
+both VAE decoders, still `q4tp`. Nothing anywhere is two bits.
+
+**23.47 GB → 13.16 GB.** The encoder went from 552 tensors to 277.
+
+### Does it still mean the same thing?
+
+That is measurable without rendering a frame: dump what the DiT actually
+receives (`CMF_TE_DUMP=<path>`) from both files on one prompt, and take the
+cosine per token.
+
+| | mean cosine to the 32B | worst token |
+|---|---|---|
+| 4B tapped at **24** | **0.9198** | 0.7982 |
+| 4B tapped at 25 | 0.8452 | 0.5910 |
+| *floor — two different tokens of the 32B itself* | *0.5914* | |
+
+The tap index is 0-based, and off-by-one is a real failure mode rather than a
+theoretical one: at 25 the worst token sits **on the floor**, uncorrelated.
+Token 0 is not projected at all but replaced by a stored `sink_out` — the
+attention sink is an outlier no regression fits — and it lands at cosine
+**0.9999** against the teacher's, which is the sharpest single confirmation
+that the mechanism is wired right.
+
+### What it costs
+
+Same prompt, same seed, same four steps, one RTX 3090, both files under
+identical settings:
+
+| | encode | denoise | video VAE | audio VAE | **total** |
+|---|---|---|---|---|---|
+| `mmh3-turbo-q4tp` | 49.2 s | 116.4 s | 101.9 s | 6.7 s | **281 s** |
+| `mmh3-turbo-clipproj4b-q4tp` | **1.9 s** | 84.5 s | 45.6 s | 5.9 s | **140 s** |
+
+Encoding is 26x faster, which is the part ClipProj actually replaced. The rest
+is residency: 13.16 GB leaves the decoder room on a 24 GB card where 23.47 GB
+does not, and the video VAE more than halves without a line of its own
+changing. On the 20 GB card this variant exists for, that gap is the whole
+point — a reader of this repo measured 20 minutes for one 512x288 clip on an
+RTX 3080 20 GB, paging the encoder in and out of a ~19 GB weight budget.
+
+The parity probe takes the device arm on both files at the same `rel rms
+4.65e-3`, which is its own small proof that the DiT came through untouched.
+
+### What it costs in quality
+
+Judge it on the clip, not on a frame: a still catches the pancake at rest and
+reads as a loss that is not there.
+
+| four bits, 32B encoder | four bits, 4B + ClipProj | two bits, 32B encoder |
+|---|---|---|
+| ![32B](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/ab_q4tp.gif) | ![ClipProj](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/ab_clipproj.gif) | ![two bits](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/ab_q2tp.gif) |
+| 23.94 GB | **13.16 GB** | 18.74 GB |
+
+Across the 39 frames the 4B does the whole job: corgi, chef hat, flat clean
+style, and the pancake **leaves the surface and comes back**, the dish empty
+underneath at the top of the toss. The verb survives — that is what `q2tp`
+lost, along with the animal and the pan.
+
+What drifts is set dressing. The 32B renders the cooking surface as a griddle
+and puts patterned wallpaper behind; the 4B gives a rimmed white plate and a
+plain ground. A 0.92 cosine is close, not equal, and where it is not equal is
+the scene's furniture rather than its subject or its action.
+
+So the ordering is not "smaller is worse". The two-bit file answers a different
+question; this one answers the right question with a plainer set, for 56% of
+the size and half the time. If you have the VRAM for the 32B encoder, use it —
+its framing is richer. If you are paging, this is the better trade, and unlike
+the two-bit build it is a trade rather than a loss.
+
+Sound is untouched either way: the audio branch never sees the prompt encoder.
+[The clip with its audio.](https://huggingface.co/infosave/MiniMax-H3-Turbo-cmf/resolve/main/samples/ab_clipproj.mp4)
+
+### Building one
+
+```bash
+cortiq animate-pack \
+  --in  mmh3-turbo-q4tp.cmf \
+  --te  qwen3vl_4b_bf16.safetensors --te-layers 24 \
+  --clip-proj mmh3-4b-ClipProj-celeb-mlp.safetensors \
+  --quant q4tp --out mmh3-turbo-clipproj4b-q4tp.cmf
+```
+
+`--te-layers` is not a size knob, it is the tap: layers above it never execute,
+so packing them is pure file. `--clip-proj` carries the projection in exact and
+replaces the encoder **as a whole component** — matching tensor names alone
+would leave `te.layers.24..49` of the 32B behind, six gigabytes that
+`num_hidden_layers` then excludes from the forward while disk and VRAM budget
+still pay for them.
+
+The encoder is [`Comfy-Org/Qwen3-VL`](https://huggingface.co/Comfy-Org/Qwen3-VL)
+`text_encoders/qwen3vl_4b_bf16.safetensors`, and the projection is
+[`NicoLab28/ClipProj-MiniMax-H3`](https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3)
+`mmh3-4b-ClipProj-celeb-mlp.safetensors` — use the file the projection was
+fitted on, since the map is only valid for those exact weights. An 8B
+projection is published too and scores higher on its author's corpus (0.8037
+against 0.7930); it costs about 4 GB more.
+
 ## What it costs to run
 
 The file is memory-mapped, so plan on RAM at least its size or every step