mlboydaisuke/LFM2-700M-ExecuTorch

LFM2-700M — ExecuTorch XNNPACK 8da4w + 8-bit embedding

lfm2_700m_xnnpack_8da4w_e8.pte (486.5 MB)

• Source: LiquidAI/LFM2-700M
• License: LFM Open License v1.0
• Quantization: 8da4w linear + 8-bit embedding (embedding_quantize: "8,0")
• Export: executorch 1.4.0 export_llm, static shape (seq_len=1), max_seq_length 2048, XNNPACK extended_ops
• Config: llm_params/lfm2_700m_xnnpack_8da4w_e8.yaml

Verification (Mac arm64, 2026-08-21)

llm_params/gen_static.py, token-by-token prefill then greedy decode, a fresh process per prompt so no answer is read through the previous one's cache:

prompt	answer
capital of France?	"The capital city of France is Paris."
17 times 4?	"17 times 4 is 68."

Decode 91.1 tok/s, from one pass over every model on this shelf with nothing else running. That matters more than it sounds: the same file measured a quarter of its rate while an export was running alongside.

Chat template: ChatML, bos 1, eos [7].

Not measured on a phone.

Three things checked before exporting

• use_sdpa_with_kv_cache is on. Upstream's qwen3_5 config leaves it off with no reason given while the equally hybrid lfm2 config has it on; measured on Qwen3.5-2B in one run, that is 8.20 tok/s against 16.64.
• dim and hidden_dim both divide by the quantizer's group size. 8da4w only touches a linear whose in_features divide by it, and skips the rest silently — SmolLM2-135M, which is 576 wide, came out at 475 MB against fp32's 540 with no warning at all.
• Every field of the params json is read by the generic path, via convert/check_params_used.py. SmolLM3 sets no_rope_layer_interval, which ModelArgs declares and only the MLX and Qualcomm backends read, and it exports fine and then repeats a single word forever.

Running it

python llm_params/gen_static.py \
  --pte lfm2_700m_xnnpack_8da4w_e8.pte \
  --tokenizer tokenizer.json \
  --prompt $'<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n' \
  --eos_ids "[7]"

The 8-bit embedding needs from executorch.kernels import quantized before the program is loaded, and portable_lib._load_for_executorch rather than executorch.runtime. Without that the method will not load at all — kernel 'quantized_decomposed::embedding_byte.dtype_out' not found — which reads like a broken export rather than a runtime missing its kernels.

(conversion scripts: executorch-models · iOS sample: executorch-samples)