ClipProj v0.1.0: projection matrices and controls
Browse files- README.md +109 -0
- h3_control_identity.pt +3 -0
- h3_control_zero.pt +3 -0
- h3_qwen3vl_4b_tap24.pt +3 -0
- h3_qwen3vl_4b_tap24_200prompts.pt +3 -0
README.md
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
tags:
|
| 4 |
+
- comfyui
|
| 5 |
+
- minimax-h3
|
| 6 |
+
- text-to-video
|
| 7 |
+
- qwen3-vl
|
| 8 |
+
- text-encoder
|
| 9 |
+
base_model:
|
| 10 |
+
- Comfy-Org/MiniMax-H3
|
| 11 |
+
- Qwen/Qwen3-VL-4B-Instruct
|
| 12 |
+
library_name: comfyui
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B
|
| 16 |
+
|
| 17 |
+
**Projection matrices that let a Qwen3-VL-4B replace the Qwen3-VL-32B text encoder of MiniMax H3.**
|
| 18 |
+
|
| 19 |
+
**15.7 GB → 5.2 GB of VRAM**, with no change to the diffusion model, the VAEs or the sampler.
|
| 20 |
+
|
| 21 |
+
> ⚠️ **Proof of concept — working, but a proof of concept.** It runs and produces good video, and every number below was measured on real hardware. Built and tested on a single setup (Windows 11, NVIDIA, ComfyUI 0.31.0) with deliberately limited exploration.
|
| 22 |
+
|
| 23 |
+
These files are useless on their own. They require the custom node:
|
| 24 |
+
**[github.com/nicolab28/ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)**
|
| 25 |
+
|
| 26 |
+
## What this is
|
| 27 |
+
|
| 28 |
+
MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a `[seq, 5120]` tensor. This repository provides a learned linear map that lets the much smaller Qwen3-VL-4B (2560 dims) produce the same conditioning:
|
| 29 |
+
|
| 30 |
+
```
|
| 31 |
+
cond = ((h - mean_in) / std_in) @ W * std_out + mean_out
|
| 32 |
+
```
|
| 33 |
+
|
| 34 |
+
It works because the 4B and the 32B share the **same tokenizer** (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. Calibration is plain **ridge regression** — no gradients, no epochs, no learning rate.
|
| 35 |
+
|
| 36 |
+
## Files
|
| 37 |
+
|
| 38 |
+
| File | Size | Description |
|
| 39 |
+
|---|---|---|
|
| 40 |
+
| `h3_qwen3vl_4b_tap24.pt` | 50 MB | **The learned matrix.** Calibrated on 2 000 prompts / 288 608 tokens, tap 24 of 36 |
|
| 41 |
+
| `h3_qwen3vl_4b_tap24_200prompts.pt` | 50 MB | Same, calibrated on 200 prompts only — kept for comparison |
|
| 42 |
+
| `h3_control_identity.pt` | 50 MB | **Control**: raw copy of the 2560 dims, no learning |
|
| 43 |
+
| `h3_control_zero.pt` | 50 MB | **Control**: `W = 0`, carries no prompt information |
|
| 44 |
+
|
| 45 |
+
Put them in `ComfyUI/models/clip_projections/`.
|
| 46 |
+
|
| 47 |
+
## Measured results
|
| 48 |
+
|
| 49 |
+
| Corpus | Tokens | Cross-prompt CKA | Test cosine | Test R² |
|
| 50 |
+
|---|---|---|---|---|
|
| 51 |
+
| 200 prompts | 37 361 | 0.95 | 0.699 | 0.490 |
|
| 52 |
+
| 2 000 prompts | 288 608 | 0.92 | **0.712** | 0.507 |
|
| 53 |
+
|
| 54 |
+
Eight times the data buys 1.8 % of cosine: the linear projection is **at its ceiling**, not starved of data. Going further needs an MLP, not more prompts.
|
| 55 |
+
|
| 56 |
+
A cosine of 0.71 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts (four distinct cuts with no bleed between them), and fl2va with first and last frame, even though the matrix only ever saw text positions.
|
| 57 |
+
|
| 58 |
+
The matrix is also **robust to a change of encoder weights**: one calibrated on bf16 works on an abliterated fp8 variant, with a measured cosine gap of 0.0023. One matrix covers every Qwen3-VL-4B variant.
|
| 59 |
+
|
| 60 |
+
## Run the controls first
|
| 61 |
+
|
| 62 |
+
The two control matrices exist to prove the learned matrix is doing the work rather than the diffusion model. Same prompt, same seed, only the matrix changes:
|
| 63 |
+
|
| 64 |
+
| Matrix | Output for *"a red ball on a wood table"* |
|
| 65 |
+
|---|---|
|
| 66 |
+
| `h3_control_zero.pt` | a countryside landscape — the prompt is entirely ignored |
|
| 67 |
+
| `h3_control_identity.pt` | a golden object in flames — unusable |
|
| 68 |
+
| `h3_qwen3vl_4b_tap24.pt` | the red ball on a wood table |
|
| 69 |
+
|
| 70 |
+
`‖W_identity‖ = 50.6` against `‖W_learned‖ = 52.4` — near-identical energy, so the difference is structural, not a matter of scale.
|
| 71 |
+
|
| 72 |
+
**If the identity control ever looks fine, the learned matrix adds nothing — and you want to know that before trusting it.**
|
| 73 |
+
|
| 74 |
+
## Known limitations
|
| 75 |
+
|
| 76 |
+
The 4B holds up well enough on everything tried so far, but the trials were not pushed far, and the honest summary is that **you lose knowledge the 32B has**.
|
| 77 |
+
|
| 78 |
+
**Named references disappear.** Some real people come through, others do not: in testing, one well-known actor rendered correctly while another was simply absent, replaced by a generic figure. The same almost certainly applies to landmarks, artworks, brands, film styles and any other named reference. A 4B stores far fewer facts than a 32B, and no projection can restore knowledge that was never encoded. We did not map which references survive — assume any proper noun is at risk.
|
| 79 |
+
|
| 80 |
+
Other known gaps:
|
| 81 |
+
|
| 82 |
+
- **ref2va** (video / audio references) is untested and refused by the node
|
| 83 |
+
- image references work in **fl2va**, but the matrix was calibrated on text positions only, so vision positions are projected out of their training distribution
|
| 84 |
+
|
| 85 |
+
## Required models
|
| 86 |
+
|
| 87 |
+
| Role | Model |
|
| 88 |
+
|---|---|
|
| 89 |
+
| Diffusion model + VAEs | [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) |
|
| 90 |
+
| Text encoder (small) | [Comfy-Org/Krea-2](https://huggingface.co/Comfy-Org/Krea-2) → `text_encoders/qwen3vl_4b_fp8_scaled.safetensors` |
|
| 91 |
+
|
| 92 |
+
The 32B text encoder is **no longer needed** — that is the entire point.
|
| 93 |
+
|
| 94 |
+
## Licence and responsibility
|
| 95 |
+
|
| 96 |
+
These matrices are released under **MIT**, like the node.
|
| 97 |
+
|
| 98 |
+
They are derived from the activations of both models, and their legal status is unclear. They are provided as-is, for research, with no claim of ownership over anything derived from the underlying models.
|
| 99 |
+
|
| 100 |
+
- **Qwen3-VL** is published by Alibaba under **Apache 2.0**. Read and comply with its terms and acceptable-use policy.
|
| 101 |
+
- **MiniMax H3** ships under a **custom licence**. Read it before any use, particularly commercial.
|
| 102 |
+
|
| 103 |
+
This project is **not affiliated with, endorsed by, or connected to** Alibaba / Qwen, MiniMax, or Comfy Org.
|
| 104 |
+
|
| 105 |
+
You remain responsible for what you generate and for complying with the licences of every model you load.
|
| 106 |
+
|
| 107 |
+
## Credits
|
| 108 |
+
|
| 109 |
+
Vibe-coded with **Anthropic Claude Code (Opus 5)**. Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected.
|
h3_control_identity.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:48c70ae8c619c2818a4e4b1d19ebcfef9ce339979030aa4f7ebbd1f6ac0b6f79
|
| 3 |
+
size 52493396
|
h3_control_zero.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:99cac87a9f75e65ad45c06822cc7a280c51c681e197bc45ba04a47312ad898a4
|
| 3 |
+
size 52493418
|
h3_qwen3vl_4b_tap24.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:85db0214838af840e5de75cfc6abc4faff5f1948a8d0918173373e6fae961358
|
| 3 |
+
size 52493451
|
h3_qwen3vl_4b_tap24_200prompts.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5789fd6f533a8985360047c53a06fe12e84ac97d10093a7c70dedf57a52cc405
|
| 3 |
+
size 52493063
|