File size: 3,047 Bytes
d1ec632 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
license: mit
tags:
- executorch
- xnnpack
- pte
- on-device
- text-generation
base_model:
- microsoft/Phi-4-mini-instruct
---
# Phi-4-mini-instruct — ExecuTorch XNNPACK 8da4w + 8-bit embedding
`phi_4_mini_xnnpack_8da4w_e8.pte` (2597.9 MB)
- **Source**: Qwen/Phi-4-mini-instruct
- **License**: Apache-2.0
- **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`)
- **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048,
XNNPACK extended_ops
- **Config**: `llm_params/phi_4_mini_xnnpack_8da4w_e8.yaml`
## Verification (Mac arm64, 2026-08-21)
`llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per
prompt so no answer is read through the previous one's cache:
| prompt | answer | decode |
|---|---|---|
| capital of France? | "The capital of France is Paris." then keeps going | 49.3 tok/s |
| 17 times 4? | "17 times 4 is equal to 68." and stops | 49.7 tok/s |
Stopping is the part worth looking at. Chat template is `<|user|>...<|end|><|assistant|>`,
bos 199999, eos [200020, 199999] — the model's own
`generation_config.json`. The upstream example config
(`examples/models/phi_4_mini/config/phi_4_mini_xnnpack.yaml`) carries 151643 for both, which
is Qwen's, and a model that never sees its end token does not stop.
Not measured on a phone. Worth noting against the rest of the shelf: this decodes at 49
tok/s where [Qwen3.5-4B](https://huggingface.co/mlboydaisuke/Qwen3.5-4B-ExecuTorch) manages
8.6 at a comparable 2.8 GB. The Qwen3.5 exports run with `use_sdpa_with_kv_cache: False`
and this one with it on.
## Converting it
```bash
python convert/export_phi_4_mini.py
```
Not `export_llm` directly, because its download path cannot reach this model:
`load_checkpoint_from_pytorch_model` reads `pytorch_model.bin` and this repository ships
safetensors only, so the code falls back to torchtune's checkpointer — a dependency with
its own torch pin, for a key rename and two splits. The script reads the safetensors, calls
upstream's own `phi_4_hf_to_meta`, and ties the output embedding back on: input and output
embeddings are tied here, so there is no `lm_head.weight` to map to `output.weight`, and the
model refuses a checkpoint without it. The torchtune converter beside it ties them; the
safetensors one does not.
## Running it
```bash
python llm_params/gen_static.py \
--pte phi_4_mini_xnnpack_8da4w_e8.pte \
--tokenizer tokenizer.json \
--prompt '<|user|>What is the capital of France?<|end|><|assistant|>' \
--eos_ids "[200020,199999]"
```
The 8-bit embedding needs `from executorch.kernels import quantized` before the program is
loaded. Without it the method will not even load — `kernel
'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken
export rather than a runtime missing its kernels.
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))
|