| --- |
| license: other |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - text-generation |
| base_model: |
| - LiquidAI/LFM2-350M |
| --- |
| # LFM2-350M — ExecuTorch XNNPACK 8da4w + 8-bit embedding |
|
|
| `lfm2_350m_xnnpack_8da4w_e8.pte` (252.8 MB) |
|
|
| - **Source**: LiquidAI/LFM2-350M |
| - **License**: LFM Open License v1.0 |
| - **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`) |
| - **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048, |
| XNNPACK extended_ops |
| - **Config**: `llm_params/lfm2_350m_xnnpack_8da4w_e8.yaml` |
|
|
| ## Verification (Mac arm64, 2026-08-21) |
|
|
| `llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per |
| prompt so no answer is read through the previous one's cache: |
|
|
| | prompt | answer | |
| |---|---| |
| | capital of France? | "The capital of France is Paris." | |
| | 17 times 4? | "17 times 4 is 68." and stops | |
|
|
| Decode **144.8 tok/s**, from one pass over every model on this shelf with nothing else |
| running. That matters more than it sounds: the same file measured a quarter of its rate |
| while an export was running alongside. |
|
|
| Chat template: ChatML, bos 1, eos [7]. |
|
|
| Not measured on a phone. |
|
|
| ## Three things checked before exporting |
|
|
| - **`use_sdpa_with_kv_cache` is on.** Upstream's `qwen3_5` config leaves it off with no |
| reason given while the equally hybrid `lfm2` config has it on; measured on Qwen3.5-2B in |
| one run, that is 8.20 tok/s against 16.64. |
| - **`dim` and `hidden_dim` both divide by the quantizer's group size.** 8da4w only touches a |
| linear whose in_features divide by it, and skips the rest silently — SmolLM2-135M, which |
| is 576 wide, came out at 475 MB against fp32's 540 with no warning at all. |
| - **Every field of the params json is read by the generic path**, via |
| `convert/check_params_used.py`. SmolLM3 sets `no_rope_layer_interval`, which `ModelArgs` |
| declares and only the MLX and Qualcomm backends read, and it exports fine and then repeats |
| a single word forever. |
| |
| ## Running it |
| |
| ```bash |
| python llm_params/gen_static.py \ |
| --pte lfm2_350m_xnnpack_8da4w_e8.pte \ |
| --tokenizer tokenizer.json \ |
| --prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \ |
| --eos_ids "[7]" |
| ``` |
| |
| The 8-bit embedding needs `from executorch.kernels import quantized` before the program is |
| loaded, and `portable_lib._load_for_executorch` rather than `executorch.runtime`. Without |
| that the method will not load at all — `kernel |
| 'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken |
| export rather than a runtime missing its kernels. |
| |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) · |
| iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples)) |
| |
