| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - text-generation |
| base_model: |
| - Qwen/Qwen2.5-1.5B |
| --- |
| # Qwen2.5-1.5B — ExecuTorch XNNPACK 8da4w + 8-bit embedding |
|
|
| `qwen2_5_1_5b_xnnpack_8da4w_e8.pte` (1035.3 MB) |
|
|
| - **Source**: Qwen/Qwen2.5-1.5B — the **base** model, not Instruct. It continues text; it |
| does not answer a chat template. |
| - **License**: Apache-2.0 |
| - **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`) |
| - **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048, |
| XNNPACK extended_ops |
| - **Config**: `llm_params/qwen2_5_1_5b_xnnpack_8da4w_e8.yaml` |
|
|
| ## Verification (Mac arm64, 2026-08-21) |
|
|
| Completions, against the untouched model on the same prompts. A base model is checked this |
| way because a chat template gets it nowhere: fed one, it echoes the question back. |
|
|
| | prompt | this file | eager fp32 | |
| |---|---|---| |
| | "The capital of France is" | turns it into a multiple-choice question and answers A, Paris | " Paris. The capital of France is also the capital of the European Union…" | |
| | "Water boils at a temperature of" | "100°C and water freezes at a temperature of 0°C" | "212 °F or 100 °C and ice melts at a temperature of 32 °" | |
| | "Seventeen times four equals" | "what number? To determine what number 17 times 4 equals, we need to perform the multiplication" | "what number? To find the product of 17 and 4, we perform the multiplication: 17 *" | |
| |
| The continuations differ, which they will once the weights are 4-bit, but nothing here is |
| wrong where eager is right. Decode **69.5 tok/s**, from one pass with nothing else running. |
| |
| **The 0.5B of this pair is not on the shelf.** It fails that comparison rather than passing |
| it: asked where water boils it says 215 degrees Fahrenheit, where eager says 212 °F or |
| 100 °C. Half a billion parameters do not survive 4-bit weights, and XNNPACK has no 8-bit |
| path that works — `torchao:8da8w` cannot be combined with the delegate and `qmode: int8` |
| falls over on grouped-query attention. At that size |
| [LFM2-350M](https://huggingface.co/mlboydaisuke/LFM2-350M-ExecuTorch) is smaller, faster and |
| correct. |
| |
| ## Converting it |
| |
| ```bash |
| python convert/export_from_safetensors.py qwen2_5_1_5b |
| ``` |
| |
| Not `export_llm` directly: this repository ships safetensors only, and |
| `load_checkpoint_from_pytorch_model` reads `pytorch_model.bin`, so the code falls back to |
| torchtune's checkpointer. Qwen2.5's converter wants one thing from torchtune — |
| `get_mapped_key` — and ExecuTorch has its own copy of that function, so the script hands the |
| real one over and stubs the rest. |
| |
| ## Running it |
| |
| ```bash |
| python llm_params/gen_static.py \ |
| --pte qwen2_5_1_5b_xnnpack_8da4w_e8.pte \ |
| --tokenizer tokenizer.json \ |
| --prompt 'The capital of France is' \ |
| --eos_ids "[151643]" |
| ``` |
| |
| The 8-bit embedding needs `from executorch.kernels import quantized` before the program is |
| loaded, and `portable_lib._load_for_executorch` rather than `executorch.runtime`. |
| |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) · |
| iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples)) |
| |
