| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - text-generation |
| base_model: |
| - Qwen/Qwen3.5-4B |
| --- |
| # Qwen3.5-4B — ExecuTorch XNNPACK 8da4w + 8-bit embedding |
|
|
| `qwen3_5_4b_xnnpack_8da4w_e8.pte` (2810.9 MB) |
|
|
| - **Source**: Qwen/Qwen3.5-4B |
| - **License**: Apache-2.0 |
| - **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`). The |
| 248320-row embedding is the reason: left in fp32 it is a gigabyte on its own, and at this |
| vocabulary size that is most of what a small model weighs. |
| - **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048, |
| XNNPACK extended_ops |
| - **Config**: `llm_params/qwen3_5_4b_xnnpack_8da4w_e8.yaml` |
|
|
| ## Verification (Mac arm64, 2026-08-21) |
|
|
| `llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per |
| prompt so no answer is read through the previous one's cache: |
|
|
| | prompt | answer | decode | |
| |---|---|---| |
| | capital of France? | opens a `<think>` block, reasons, answers Paris | 19.10 tok/s | |
| | 日本の首都は? | reads the question back in both languages, then answers | 19.10 tok/s | |
|
|
| This size reaches for its thinking mode on questions the 2B answers outright, so a short |
| `max_new` may cut it off mid-thought. Budget for it. |
|
|
| Chat template: ChatML, bos 248045, eos [248046, 248044]. |
|
|
| Not measured on a phone, and at 2.8 GB it is the largest text model on this shelf. The |
| [0.8B](https://huggingface.co/mlboydaisuke/Qwen3.5-0.8B-ExecuTorch) was measured at 10.5 |
| tok/s on an iPhone 17 Pro, but on its earlier build, before `use_sdpa_with_kv_cache` was |
| turned on; the [2B](https://huggingface.co/mlboydaisuke/Qwen3.5-2B-ExecuTorch) is the middle |
| of the three. |
|
|
| Both rates above are from one run with nothing else on the machine. |
|
|
| ## The setting that doubles it |
|
|
| `use_sdpa_with_kv_cache: True`. Upstream's own `qwen3_5` config has it off with no reason |
| given, while the equally hybrid `lfm2` config has it on. Measured on the 2B in one run: |
| **8.20 tok/s off, 16.64 on**. |
|
|
| That it is only a kernel swap and not a different model was checked three ways: three of |
| four prompts came out token-identical; the first-step logits agree at **corr 0.995** with |
| the same top-1; and 338 tokens deep they agree at **corr 0.990**, same top-1 and top-2. The |
| fourth prompt differs where a thinking model decides how long to think, which is a near-tie |
| either way. |
|
|
| ## Running it |
|
|
| ```bash |
| python llm_params/gen_static.py \ |
| --pte qwen3_5_4b_xnnpack_8da4w_e8.pte \ |
| --tokenizer tokenizer.json \ |
| --prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \ |
| --eos_ids "[248046,248044]" |
| ``` |
|
|
| The 8-bit embedding needs `from executorch.kernels import quantized` before the program is |
| loaded. Without it the method will not even load — `kernel |
| 'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken |
| export rather than a runtime missing its kernels. |
| |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) · |
| iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples)) |
| |
