| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - image-text-to-text |
| base_model: |
| - Qwen/Qwen3-VL-2B-Instruct |
| --- |
| # Qwen3-VL-2B-Instruct — ExecuTorch |
|
|
| A vision-language model as one `.pte` with three entry points. Ask it about a photograph on |
| the device; nothing leaves it. |
|
|
| - **Source**: Qwen/Qwen3-VL-2B-Instruct — 2.13B parameters, 28-layer decoder, 24-layer vision tower |
| - **License**: apache-2.0 |
| - **Input**: a 512x512 RGB photograph and a question |
| - **Output**: text |
|
|
| ## Methods |
|
|
| | method | takes | returns | |
| |---|---|---| |
| | `vision_encoder` | `pixel_values [1024, 1536]` | `merged [256, 2048]` **and three deepstack maps `[256, 2048]`** | |
| | `token_embeddings` | `ids [1, seq]` | `rows [1, seq, 2048]` | |
| | `text_model` | `rows`, `positions [3, 1, seq]`, `slots [1, seq]`, three deepstack maps `[1, seq, 2048]` | `logits [1, seq, 151936]` | |
|
|
| Three methods rather than three files because ExecuTorch copies a mutable buffer into every |
| method that names it, so a prefill method and a decode method would each get their own KV |
| cache and neither would see the other's writes. One graph serves a 277-token prompt and a |
| single token. |
|
|
| ## What the caller owns |
|
|
| Three things, and getting any of them wrong does not throw — it answers about the wrong |
| thing: |
|
|
| 1. **Splice** the 256 vision rows into the prompt embedding at the image token positions. |
| 2. **Scatter** the three deepstack maps to those same positions, zero everywhere else. |
| Qwen3-VL adds features from vision blocks 5, 11 and 17 into decoder layers 0, 1 and 2, |
| and only at image positions. Upstream writes that as |
| `hidden_states[visual_pos_masks, :] += embeds`, which does not export; here the maps |
| arrive already placed, so the graph just adds them. |
| 3. **Pass Qwen's own 3-D positions** from `get_rope_index`, not `arange`. The rope is |
| interleaved M-RoPE with `mrope_section [24, 20, 20]` — time, height, width. A 1-D |
| position array gives every image patch the same angle and reads as a mild degradation |
| rather than an error. |
|
|
| `convert/run_qwen3_vl.py` does all three in 40 lines. |
|
|
| ## Verification (Mac arm64, executorch 1.4.0, torch 2.13.0) |
|
|
| Parity is the last-row logits of a real prompt against `Qwen3VLForConditionalGeneration` in |
| eager fp32, per question, because a model of this size can hold a scene and lose the small |
| print. Both questions are about the same photograph. |
|
|
| | build | file | size (MB) | corr, "name on the sign" | corr, "describe it" | vision (ms) | prefill (ms) | decode | |
| |---|---|---|---|---|---|---|---| |
| | 8da8w | `qwen3_vl_2b_instruct_xnnpack_8da8w.pte` | 2453.1 | 0.993435 | 0.992611 | 568.7 | 590.6 | 17.1 tok/s | |
| | 8da4w *(not shipped)* | `qwen3_vl_2b_instruct_xnnpack_8da4w.pte` | 1697.5 | 0.739964 | 0.948887 | 566.7 | 739.4 | 19.8 tok/s | |
|
|
| The prompt is 277 tokens, of which 256 are the picture. `vision_encoder` runs |
| once per image, `text_model` once per prompt and once per generated token. |
|
|
| ### What it says |
|
|
| Asked for the name on a pub sign, against the eager model's *"Lamb & Flag"*: |
|
|
| - **8da8w** — Based on the image provided, the name of the pub is **The Lamb & Flag**. |
|
|
| This name is visible on the sign above the entrance to the establishment. The sign also includes the address " |
| - **8da4w** — Based on the text visible on the sign above the entrance of the establishment, the name of the pub is **Pine Purpose**. |
|
|
| The sign is located on the front of the building, just above |
|
|
| ## The rewrites, and what each was checked against |
|
|
| Neither this repo's generic VL bundle nor `export_llm` reaches Qwen3-VL. Five things had to |
| change, each measured against the module it replaced: |
|
|
| | what | why | checked | |
| |---|---|---| |
| | vision constants precomputed | position embeddings are bilinearly interpolated from `grid_thw` on every call | part of the 0.000e+00 below | |
| | variable-length attention → plain SDPA | one image is one sequence, so `cu_seqlens` is `[0, 1024]` | exact | |
| | patch embedding `Conv3d` → `F.linear` | portable convolution takes 3-D or 4-D only: `Expect input tensor to be 3-D or 4-D, but got, 5` | max_abs_diff 4.9e-04 | |
| | deepstack mask-assignment → add | boolean-mask assignment does not export | first-step logits corr 1.000000 | |
| | embedding table quantised by hand | see below | corr 0.999965 | |
|
|
| With the first two in place and no quantisation, the vision wrapper reproduces |
| `Qwen3VLVisionModel` at **max_abs_diff 0.000e+00** across all four outputs, and the decoder |
| wrapper reproduces the full model's first-step logits at **corr 1.000000**. |
|
|
| ## Two traps worth knowing |
|
|
| **The token embedding table escapes quantisation.** It is an `nn.Embedding`, so a filter |
| written as `isinstance(module, nn.Linear)` skips it — and `tie_word_embeddings` does not |
| save it, because `quantize_` swaps `lm_head.weight` for a new tensor while the embedding |
| keeps pointing at the old fp32 one. At 151,936 x 2048 that is **1.24 GB left in fp32**: the |
| first export of this model came out at 2630 MB. torchao's own `IntxWeightOnlyConfig` does |
| reach an `nn.Embedding` but will not lower — `Missing out variants: |
| {'torchao::dequantize_affine'}` — so the quantisation is written out by hand, one scale |
| per row. |
| |
| **512x512 is where small text lives or dies, and the resampling filter decides.** Asked |
| for the name on the sign, **the eager fp32 model** answers: |
| |
| | resize filter | eager fp32 | |
| |---|---| |
| | BICUBIC | "Lamb & Flag" — right | |
| | BILINEAR | "The Lamb & Flag" — right | |
| | LANCZOS | "The Pigeon House" — wrong | |
| | NEAREST | "Pompadour" — wrong | |
| |
| That is before any quantisation. If fine print matters for your use, the preprocessing is |
| not a detail; `convert/export_qwen3_vl.py` puts the resize behind one helper for that reason. |
| |
| ## Not shipped |
| |
| **8da4w** — the same bundle with a 4-bit decoder — is 1697.5 MB against |
| 2453.1 MB, and it does not hold: logits correlation 0.740 on the sign question |
| against 0.993, and it reads the sign as *"Pine Purpose"*. Measured where the loss is, |
| one arm at a time: the vision tower at int8 costs corr 0.990 against fp32's 0.99982, the |
| hand-quantised table costs 0.99997, and the 4-bit decoder is the rest. Reported here rather |
| than dropped, because a 1.7 GB 2B vision-language model is worth wanting and this is what it |
| costs. |
| |
| ## Conversion |
| |
| ```bash |
| python convert/export_qwen3_vl.py # check both halves against eager |
| python convert/export_qwen3_vl.py --export # write the bundle |
| python convert/check_qwen3_vl.py 8da8w # parity and speed |
| python convert/run_qwen3_vl.py photo.jpg "what is on the sign?" |
| ``` |
| |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models)) |
| |
