Note about Vision & Images
Tried your approaching when working with vision. So making this post as a sort of "word of warning" to anyone else walking this road.
Qwen3-vl-32b & Qwen3-vl-8b tend to focus different activation energy on entirely different tokens. So when one starts testing for rough correlation between 32b & 8b you'll find the distribution is strongly bi-modal. This is because AFAIT around L29 32b & 8b start "diverging" in what "KV/Tokens" they do/do-not "pay attention to" to paraphrase.
So a simple affine map from say 8b -> 32b straight up doesn't exist (with vision tokens). It has to have (basically) an attention mechanism to redistribute activation energy.
One can "very nearly" for an affine map at 8BL16 -> 32BL29, but if you include text tokens, doesn't work "that cleanly".
Does it means the smaller encoder is better to use in scene like image to video, other than text to video, because the image provide more context？
Does it means the smaller encoder is better to use in scene like image to video, other than text to video, because the image provide more context？
No. It means a smaller encoder is pretty fundamentally broken for handling image & video input without a secondary attention mechanism to "translate" the different learned attention patterns which Qwen3-vl-8b & Qwen3-vl-32b have encoded into them.
Text (and text alone) works fine, I have no clue why, the shared geometry is great. But multiple/mixed media plainly needs a non-affine activation aware transform.
Thanks for this — it changed what we trained on.
We are finalizing an MLP rather than a plain affine map. It comes out slightly better than the affine transform, but not dramatically so. The bulk of what a projection can recover seems to be within reach of a linear map already; the MLP buys refinement, not a different regime.
The concrete effect of your post was that we added reference images to the training corpus, which had been text-only until then. That turned out to matter a lot more than the choice of architecture. On 100 held-out images, the cosine on the conditioning improved from 0.71 to 0.85, and every single one of the 100 improved — no exceptions. Text quality did not regress.
My reading is that the map needs to see a diversity of token types to cover the space it is asked to fill. Vision tokens land in a region of the hidden space that a text-only corpus simply never visits, so the map was extrapolating there. It is not that images taught it something about images specifically — it is that they filled in a part of the domain that was empty.
Which is also why I am cautious about your bimodality point. I could not reproduce it on my side: measuring 32B against 4B hidden states on the same tokens, imposing the target's tokenization on the student so the sequences cannot drift, I get a unimodal distribution. That does not settle your mechanism argument, but it does mean the practical conclusion — that no affine map exists — does not hold: fitting one on data that includes vision tokens moves it substantially.
Where I think you are right is on the residual. What is left after the projection is exactly the kind of failure a position-wise map should have: small discrete attributes get lost — a hat, a colour — while motion, action and overall composition survive. A map that treats each token independently has no way to bind an attribute to the thing it belongs to.
Does it means the smaller encoder is better to use in scene like image to video, other than text to video, because the image provide more context？
No. It means a smaller encoder is pretty fundamentally broken for handling image & video input without a secondary attention mechanism to "translate" the different learned attention patterns which Qwen3-vl-8b & Qwen3-vl-32b have encoded into them.
Text (and text alone) works fine, I have no clue why, the shared geometry is great. But multiple/mixed media plainly needs a non-affine activation aware transform.
Oh, I thought that smaller encoder may not be able to capture abstract semantics, such names of people, etc. Using image to video, maybe this won't be a big problem, as we usually do not need to provide such prompts. I saw some comparison using image to video, the result is quite similar with the original encoder.
As the tests did not has complex scene, maybe more training on the small encoder is still needed.
Results -> https://huggingface.co/valarauca1/qwen3-vl-25m-stitched-8bL16-to-32bL29
I found surprisingly good coherence even to very long 100,000 token prompts. It isn't "prefect" and the auto warps. Still keeps around 2/3rds of qwen3-vl-32b, which is "not great"
Nice — 100k tokens holding together is impressive on its own.
Out of curiosity, what did you calibrate the 25M mapping on?
Asking because the same plateau hit me, and it turned out to be the corpus rather than the method. Going from v3 to v3.1 I gained 29 % on phoneme errors overall, and 60 to 74 % on European languages — no architectural change at all, just a wider calibration set.
I have a fair amount of prompt corpus lying around from that: 8 632 full H3 prompts expanded with Mistral and Gemini, plus tagged prompts per language. Happy to share it if that would be useful to you.
Out of curiosity, what did you calibrate the 25M mapping on?
I trained against a random combination of:
Then evaluated each training epoch against
Asking because the same plateau hit me, and it turned out to be the corpus rather than the method
Vidpro & TIP-I2V are probably what you're looking for. They have fairly good multi-language support.
Vidpro does have a non-commerical license tho, so you have to be developing this as a hobbyist, not for a employer.
On the datasets: COCO and VidProM will get you image and video semantics, but neither will get you multilingual anything — COCO captions are English, VidProM is overwhelmingly English too. That was exactly my v3 problem, and it turned out to be the whole story.
The fix was not more data or a bigger model. It was rebalancing the calibration corpus so every writing system carries roughly the same weight, English aside since the tag grammar is defined in English. Same architecture, same tap, same rank — only the corpus changed.
Result, measured over 297 speech renders across three seeds and eleven languages: 29% fewer phoneme errors overall, and 60 to 74% fewer in Spanish, French, German and Italian. The image scores did not move at all — v3 and v3.1 overlap completely there. The entire gain is in pronunciation.
Which says something about what the projection is actually fitting: a direction the corpus never exercises is a direction the ridge has no reason to preserve. The model knows the language fine; the matrix just was not asked to carry it.
Full report with the per-language tables: https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3/blob/main/bench3.1/README.md
The offer stands for the corpus — the part that is mine to share is the expanded prompt set and the tagged prompts, which is where the language balancing lives.