File size: 3,119 Bytes
d9d0200 618a94f d9d0200 618a94f d9d0200 618a94f d9d0200 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- text-generation
base_model:
- Qwen/Qwen3.5-4B
---
# Qwen3.5-4B — ExecuTorch XNNPACK 8da4w + 8-bit embedding
`qwen3_5_4b_xnnpack_8da4w_e8.pte` (2810.9 MB)
- **Source**: Qwen/Qwen3.5-4B
- **License**: Apache-2.0
- **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`). The
248320-row embedding is the reason: left in fp32 it is a gigabyte on its own, and at this
vocabulary size that is most of what a small model weighs.
- **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048,
XNNPACK extended_ops
- **Config**: `llm_params/qwen3_5_4b_xnnpack_8da4w_e8.yaml`
## Verification (Mac arm64, 2026-08-21)
`llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per
prompt so no answer is read through the previous one's cache:
| prompt | answer | decode |
|---|---|---|
| capital of France? | opens a `<think>` block, reasons, answers Paris | 19.10 tok/s |
| 日本の首都は? | reads the question back in both languages, then answers | 19.10 tok/s |
This size reaches for its thinking mode on questions the 2B answers outright, so a short
`max_new` may cut it off mid-thought. Budget for it.
Chat template: ChatML, bos 248045, eos [248046, 248044].
Not measured on a phone, and at 2.8 GB it is the largest text model on this shelf. The
[0.8B](https://huggingface.co/mlboydaisuke/Qwen3.5-0.8B-ExecuTorch) was measured at 10.5
tok/s on an iPhone 17 Pro, but on its earlier build, before `use_sdpa_with_kv_cache` was
turned on; the [2B](https://huggingface.co/mlboydaisuke/Qwen3.5-2B-ExecuTorch) is the middle
of the three.
Both rates above are from one run with nothing else on the machine.
## The setting that doubles it
`use_sdpa_with_kv_cache: True`. Upstream's own `qwen3_5` config has it off with no reason
given, while the equally hybrid `lfm2` config has it on. Measured on the 2B in one run:
**8.20 tok/s off, 16.64 on**.
That it is only a kernel swap and not a different model was checked three ways: three of
four prompts came out token-identical; the first-step logits agree at **corr 0.995** with
the same top-1; and 338 tokens deep they agree at **corr 0.990**, same top-1 and top-2. The
fourth prompt differs where a thinking model decides how long to think, which is a near-tie
either way.
## Running it
```bash
python llm_params/gen_static.py \
--pte qwen3_5_4b_xnnpack_8da4w_e8.pte \
--tokenizer tokenizer.json \
--prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \
--eos_ids "[248046,248044]"
```
The 8-bit embedding needs `from executorch.kernels import quantized` before the program is
loaded. Without it the method will not even load — `kernel
'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken
export rather than a runtime missing its kernels.
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))
|