File size: 2,769 Bytes
dac5dff | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
license: other
tags:
- executorch
- xnnpack
- pte
- on-device
- text-generation
base_model:
- LiquidAI/LFM2-700M
---
# LFM2-700M — ExecuTorch XNNPACK 8da4w + 8-bit embedding
`lfm2_700m_xnnpack_8da4w_e8.pte` (486.5 MB)
- **Source**: LiquidAI/LFM2-700M
- **License**: LFM Open License v1.0
- **Quantization**: 8da4w linear + 8-bit embedding (`embedding_quantize: "8,0"`)
- **Export**: executorch 1.4.0 `export_llm`, static shape (seq_len=1), max_seq_length 2048,
XNNPACK extended_ops
- **Config**: `llm_params/lfm2_700m_xnnpack_8da4w_e8.yaml`
## Verification (Mac arm64, 2026-08-21)
`llm_params/gen_static.py`, token-by-token prefill then greedy decode, a fresh process per
prompt so no answer is read through the previous one's cache:
| prompt | answer |
|---|---|
| capital of France? | "The capital city of France is Paris." |
| 17 times 4? | "17 times 4 is 68." |
Decode **91.1 tok/s**, from one pass over every model on this shelf with nothing else
running. That matters more than it sounds: the same file measured a quarter of its rate
while an export was running alongside.
Chat template: ChatML, bos 1, eos [7].
Not measured on a phone.
## Three things checked before exporting
- **`use_sdpa_with_kv_cache` is on.** Upstream's `qwen3_5` config leaves it off with no
reason given while the equally hybrid `lfm2` config has it on; measured on Qwen3.5-2B in
one run, that is 8.20 tok/s against 16.64.
- **`dim` and `hidden_dim` both divide by the quantizer's group size.** 8da4w only touches a
linear whose in_features divide by it, and skips the rest silently — SmolLM2-135M, which
is 576 wide, came out at 475 MB against fp32's 540 with no warning at all.
- **Every field of the params json is read by the generic path**, via
`convert/check_params_used.py`. SmolLM3 sets `no_rope_layer_interval`, which `ModelArgs`
declares and only the MLX and Qualcomm backends read, and it exports fine and then repeats
a single word forever.
## Running it
```bash
python llm_params/gen_static.py \
--pte lfm2_700m_xnnpack_8da4w_e8.pte \
--tokenizer tokenizer.json \
--prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \
--eos_ids "[7]"
```
The 8-bit embedding needs `from executorch.kernels import quantized` before the program is
loaded, and `portable_lib._load_for_executorch` rather than `executorch.runtime`. Without
that the method will not load at all — `kernel
'quantized_decomposed::embedding_byte.dtype_out' not found` — which reads like a broken
export rather than a runtime missing its kernels.
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))
|