NicoLab28 commited on
Commit
7c99d41
·
verified ·
1 Parent(s): d239611

ClipProj v0.1.0: projection matrices and controls

Browse files
README.md ADDED
@@ -0,0 +1,109 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - comfyui
5
+ - minimax-h3
6
+ - text-to-video
7
+ - qwen3-vl
8
+ - text-encoder
9
+ base_model:
10
+ - Comfy-Org/MiniMax-H3
11
+ - Qwen/Qwen3-VL-4B-Instruct
12
+ library_name: comfyui
13
+ ---
14
+
15
+ # ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B
16
+
17
+ **Projection matrices that let a Qwen3-VL-4B replace the Qwen3-VL-32B text encoder of MiniMax H3.**
18
+
19
+ **15.7 GB → 5.2 GB of VRAM**, with no change to the diffusion model, the VAEs or the sampler.
20
+
21
+ > ⚠️ **Proof of concept — working, but a proof of concept.** It runs and produces good video, and every number below was measured on real hardware. Built and tested on a single setup (Windows 11, NVIDIA, ComfyUI 0.31.0) with deliberately limited exploration.
22
+
23
+ These files are useless on their own. They require the custom node:
24
+ **[github.com/nicolab28/ComfyUI-ClipProj](https://github.com/nicolab28/ComfyUI-ClipProj)**
25
+
26
+ ## What this is
27
+
28
+ MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a `[seq, 5120]` tensor. This repository provides a learned linear map that lets the much smaller Qwen3-VL-4B (2560 dims) produce the same conditioning:
29
+
30
+ ```
31
+ cond = ((h - mean_in) / std_in) @ W * std_out + mean_out
32
+ ```
33
+
34
+ It works because the 4B and the 32B share the **same tokenizer** (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. Calibration is plain **ridge regression** — no gradients, no epochs, no learning rate.
35
+
36
+ ## Files
37
+
38
+ | File | Size | Description |
39
+ |---|---|---|
40
+ | `h3_qwen3vl_4b_tap24.pt` | 50 MB | **The learned matrix.** Calibrated on 2 000 prompts / 288 608 tokens, tap 24 of 36 |
41
+ | `h3_qwen3vl_4b_tap24_200prompts.pt` | 50 MB | Same, calibrated on 200 prompts only — kept for comparison |
42
+ | `h3_control_identity.pt` | 50 MB | **Control**: raw copy of the 2560 dims, no learning |
43
+ | `h3_control_zero.pt` | 50 MB | **Control**: `W = 0`, carries no prompt information |
44
+
45
+ Put them in `ComfyUI/models/clip_projections/`.
46
+
47
+ ## Measured results
48
+
49
+ | Corpus | Tokens | Cross-prompt CKA | Test cosine | Test R² |
50
+ |---|---|---|---|---|
51
+ | 200 prompts | 37 361 | 0.95 | 0.699 | 0.490 |
52
+ | 2 000 prompts | 288 608 | 0.92 | **0.712** | 0.507 |
53
+
54
+ Eight times the data buys 1.8 % of cosine: the linear projection is **at its ceiling**, not starved of data. Going further needs an MLP, not more prompts.
55
+
56
+ A cosine of 0.71 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts (four distinct cuts with no bleed between them), and fl2va with first and last frame, even though the matrix only ever saw text positions.
57
+
58
+ The matrix is also **robust to a change of encoder weights**: one calibrated on bf16 works on an abliterated fp8 variant, with a measured cosine gap of 0.0023. One matrix covers every Qwen3-VL-4B variant.
59
+
60
+ ## Run the controls first
61
+
62
+ The two control matrices exist to prove the learned matrix is doing the work rather than the diffusion model. Same prompt, same seed, only the matrix changes:
63
+
64
+ | Matrix | Output for *"a red ball on a wood table"* |
65
+ |---|---|
66
+ | `h3_control_zero.pt` | a countryside landscape — the prompt is entirely ignored |
67
+ | `h3_control_identity.pt` | a golden object in flames — unusable |
68
+ | `h3_qwen3vl_4b_tap24.pt` | the red ball on a wood table |
69
+
70
+ `‖W_identity‖ = 50.6` against `‖W_learned‖ = 52.4` — near-identical energy, so the difference is structural, not a matter of scale.
71
+
72
+ **If the identity control ever looks fine, the learned matrix adds nothing — and you want to know that before trusting it.**
73
+
74
+ ## Known limitations
75
+
76
+ The 4B holds up well enough on everything tried so far, but the trials were not pushed far, and the honest summary is that **you lose knowledge the 32B has**.
77
+
78
+ **Named references disappear.** Some real people come through, others do not: in testing, one well-known actor rendered correctly while another was simply absent, replaced by a generic figure. The same almost certainly applies to landmarks, artworks, brands, film styles and any other named reference. A 4B stores far fewer facts than a 32B, and no projection can restore knowledge that was never encoded. We did not map which references survive — assume any proper noun is at risk.
79
+
80
+ Other known gaps:
81
+
82
+ - **ref2va** (video / audio references) is untested and refused by the node
83
+ - image references work in **fl2va**, but the matrix was calibrated on text positions only, so vision positions are projected out of their training distribution
84
+
85
+ ## Required models
86
+
87
+ | Role | Model |
88
+ |---|---|
89
+ | Diffusion model + VAEs | [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) |
90
+ | Text encoder (small) | [Comfy-Org/Krea-2](https://huggingface.co/Comfy-Org/Krea-2) → `text_encoders/qwen3vl_4b_fp8_scaled.safetensors` |
91
+
92
+ The 32B text encoder is **no longer needed** — that is the entire point.
93
+
94
+ ## Licence and responsibility
95
+
96
+ These matrices are released under **MIT**, like the node.
97
+
98
+ They are derived from the activations of both models, and their legal status is unclear. They are provided as-is, for research, with no claim of ownership over anything derived from the underlying models.
99
+
100
+ - **Qwen3-VL** is published by Alibaba under **Apache 2.0**. Read and comply with its terms and acceptable-use policy.
101
+ - **MiniMax H3** ships under a **custom licence**. Read it before any use, particularly commercial.
102
+
103
+ This project is **not affiliated with, endorsed by, or connected to** Alibaba / Qwen, MiniMax, or Comfy Org.
104
+
105
+ You remain responsible for what you generate and for complying with the licences of every model you load.
106
+
107
+ ## Credits
108
+
109
+ Vibe-coded with **Anthropic Claude Code (Opus 5)**. Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected.
h3_control_identity.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:48c70ae8c619c2818a4e4b1d19ebcfef9ce339979030aa4f7ebbd1f6ac0b6f79
3
+ size 52493396
h3_control_zero.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:99cac87a9f75e65ad45c06822cc7a280c51c681e197bc45ba04a47312ad898a4
3
+ size 52493418
h3_qwen3vl_4b_tap24.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:85db0214838af840e5de75cfc6abc4faff5f1948a8d0918173373e6fae961358
3
+ size 52493451
h3_qwen3vl_4b_tap24_200prompts.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5789fd6f533a8985360047c53a06fe12e84ac97d10093a7c70dedf57a52cc405
3
+ size 52493063