Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,75 @@
+---
+license: mit
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- text-generation
+base_model:
+- microsoft/Phi-4-mini-instruct
+---
+# Phi-4-mini-instruct — ExecuTorch XNNPACK 8da4w + 8-bit embedding
+
+`phi_4_mini_xnnpack_8da4w_e8.pte` (2597.9 MB)
+
+- **Source**: Qwen/Phi-4-mini-instruct
+- **License**: Apache-2.0
+- **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`)
+- **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048,
+  XNNPACK extended_ops
+- **Config**: `llm_params/phi_4_mini_xnnpack_8da4w_e8.yaml`
+
+## Verification (Mac arm64, 2026-08-21)
+
+`llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per
+prompt so no answer is read through the previous one's cache:
+
+| prompt | answer | decode |
+|---|---|---|
+| capital of France? | "The capital of France is Paris." then keeps going | 49.3 tok/s |
+| 17 times 4? | "17 times 4 is equal to 68." and stops | 49.7 tok/s |
+
+Stopping is the part worth looking at. Chat template is `<|user|>...<|end|><|assistant|>`,
+bos 199999, eos [200020, 199999] — the model's own
+`generation_config.json`. The upstream example config
+(`examples/models/phi_4_mini/config/phi_4_mini_xnnpack.yaml`) carries 151643 for both, which
+is Qwen's, and a model that never sees its end token does not stop.
+
+Not measured on a phone. Worth noting against the rest of the shelf: this decodes at 49
+tok/s where [Qwen3.5-4B](https://huggingface.co/mlboydaisuke/Qwen3.5-4B-ExecuTorch) manages
+8.6 at a comparable 2.8 GB. The Qwen3.5 exports run with `use_sdpa_with_kv_cache: False`
+and this one with it on.
+
+## Converting it
+
+```bash
+python convert/export_phi_4_mini.py
+```
+
+Not `export_llm` directly, because its download path cannot reach this model:
+`load_checkpoint_from_pytorch_model` reads `pytorch_model.bin` and this repository ships
+safetensors only, so the code falls back to torchtune's checkpointer — a dependency with
+its own torch pin, for a key rename and two splits. The script reads the safetensors, calls
+upstream's own `phi_4_hf_to_meta`, and ties the output embedding back on: input and output
+embeddings are tied here, so there is no `lm_head.weight` to map to `output.weight`, and the
+model refuses a checkpoint without it. The torchtune converter beside it ties them; the
+safetensors one does not.
+
+## Running it
+
+```bash
+python llm_params/gen_static.py \
+  --pte phi_4_mini_xnnpack_8da4w_e8.pte \
+  --tokenizer tokenizer.json \
+  --prompt '<|user|>What is the capital of France?<|end|><|assistant|>' \
+  --eos_ids "[200020,199999]"
+```
+
+The 8-bit embedding needs `from executorch.kernels import quantized` before the program is
+loaded. Without it the method will not even load — `kernel
+'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken
+export rather than a runtime missing its kernels.
+
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
+iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))