runanywhere/lfm2_5_350m_HNPU

LFM2.5-350M — QHexRT NPU bundle (Hexagon v79 + v81)

Precompiled LiquidAI LFM2.5-350M (Lfm2ForCausalLM) for the QHexRT runtime on Qualcomm Hexagon v79 (Snapdragon 8 Elite / SM8750, e.g. Galaxy S25). A hybrid LLM with 10 short-conv layers + 6 GQA-attention layers — the transformer and the tied lm-head run on the NPU (HMX/HVX), driven by the lfm_generate host-op through QHexRT's manifest plan-interpreter; the host does only embed lookup, RoPE/mask, the cache copies, and the argmax scan. Device-validated end-to-end (greedy decode matches the fp32 reference for the first 6 generated tokens, then tracks coherently under fp16).

Contents (v79/)

file	what	size
lfm2-5-350m-2048.json	QHexRT manifest — balanced (recommended) (MAXCTX=2048, ~29 tok/s); matches Genie's context.size	—
lfm2-5-350m-512s.json	QHexRT manifest — fast (MAXCTX=512, ~41 tok/s)	—
lfm2-5-350m-4k.json	QHexRT manifest — long context (MAXCTX=4096, ~14 tok/s)	—
lfm_2048_shared.bin	weight-shared context holding batched-prefill (AR=512) + 512-decode + 2048-decode graphs	583 MB
lfm_512_4k_shared.bin	weight-shared context holding batched-prefill (AR=512) + 512-decode + 4096-decode graphs	586 MB
lfm_lmh.bin	NPU lm-head graph — hidden[1,1024] → logits[1,65536] (tied), fp16 on HMX	134 MB
lfm_embed_f16.bin	token embedding table [65536,1024] f16 (host embed lookup)	134 MB
tokenizer.json	LFM2 tokenizer (vocab 65536)	3.3 MB

Three context windows — pick the manifest. -2048 is the recommended default (balanced speed + real long context, matches Genie's 2048); -512s is fastest for short turns; -4k is for genuinely long inputs. Each manifest loads only its own weight-shared bin, so a run pulls one ~585 MB context — the extra decode-width graphs cost only a few MB each (identical fp16 weights stored once). Batched prefill seeds the decode caches in one forward → O(1) TTFT for prompts up to 512 tokens; decode then runs autoregressively on the NPU lm-head. A sliding-window KV ring lets the conversation continue past the window (the model keeps the most recent MAXCTX tokens). All three are device-validated coherent (incl. verified long-range recall at 4096).

Run (QHexRT CLI)

hf download runanywhere/lfm2_5_350m_HNPU --local-dir lfm2_5_350m_HNPU
# qhx_generate from a QHexRT build; QNN libs from the QAIRT SDK (lib/aarch64-android) + v79 HTP skel.
adb push lfm2_5_350m_HNPU/v79 /data/local/tmp/wq/lfm
adb shell "cd /data/local/tmp/wq && export ADSP_LIBRARY_PATH='/data/local/tmp/wq/dsp;/data/local/tmp/wq;/vendor/dsp/cdsp'; \
  LD_LIBRARY_PATH=. ./qhx_generate lfm/lfm2-5-350m-2048.json libQnnHtp.so libQnnSystem.so lfm 40 'The capital of France is'"
# -> coherent continuation; [lfm] prefill(TTFT) ~52 ms (any prompt) + decode ~34 ms/tok (~29 tok/s)

Performance (measured on-device, v79 / Samsung S25)

Context	TTFT	Decode	tok/s	Peak RSS	KV cache	Coherent
512	54 ms	24.6 ms/tok	40.7	734 MB	6 MB	✅
2048	52 ms	34.4 ms/tok	29.1	731 MB	24 MB	✅
4096	52 ms	71.6 ms/tok	14.0	734 MB	48 MB	✅

• TTFT is O(1) (~52–54 ms) — same batched-prefill graph; constant in prompt length (S=6 → 52 ms, S=260 → 61 ms; vs ~3.9 s for the old decode-over-prompt).
• Decode is flat across position (2048: 34.4 → 37.1 ms/tok over 4× more tokens; 4096: no growth) — per-token latency is set by the graph's fixed attention width, not by how many tokens already exist.
• KV cache = 6 attn layers × max_ctx × 512 × 2 B × 2(K,V) (only 6 of 16 layers are attention; the 10 short-conv layers add a negligible 61 KB). It is DDR-streamed, not VTCM-resident — which is why peak RSS stays flat (732 MB, weight-dominated) even at 4096.
• Long-range recall verified @4096: recalls a number stated before a distractor sentence → attention genuinely retrieves from earlier KV positions, not just local conv.

Notes

• Arch: v79 only — context binaries are dsp-arch-pinned.
• No custom op-package — pure-native HTP graph (the conv1d short-conv is a native op; QK-norm + GQA + 1D RoPE θ=1e6, head_dim 64). The host does only embed lookup, the full-dim RoPE table, the cache mask, and the tied lm-head argmax.
• Base model (not instruction-tuned) — it continues text rather than chatting.
• Three compiled context windows (512 / 2048 / 4096); a sliding-window KV ring lets generation continue past the window (keeping the most recent MAXCTX tokens). Beyond 4096 needs a re-export.
• Source: LiquidAI/LFM2.5-350M-Base, compiled with QAIRT 2.45 for qualcomm-snapdragon-8-elite-for-galaxy.

v81 (SM8850 / soc_model 87)

The v81/ bundle is device-validated on SM8850: coherent fp16 generation, decode ~14 ms/tok, prefill TTFT ~49 ms (MAXCTX=2048). Greedy tracks the fp32 HF gold for the first 6 tokens then makes the same kind of fp16 near-tie the v79 bundle does (e.g. "A) Paris" vs "A: Paris") and continues correctly — a second prompt ("Photosynthesis is the process by which…") returns a fluent, factually-correct continuation. Built via the config-driven LFM2 export recipe (decode 8/8 + prefill cosine 1.000000 vs the lfm_net oracle) and compiled with the model-lib f16 + {O:3,vtcm_mb:8,dlbc:1} route.

Files (v81/) — separate prefill/decode/lmhead bins (not weight-shared)

file	what
lfm2-5-350m-2048.json	QHexRT manifest (MAXCTX=2048; lfm_generate over the 3 graphs below)
lfm_pf_f16.bin	batched prefill graph (AR=512), fp16
lfm_dec_f16.bin	autoregressive decode graph (MAXCTX 2048, GQA-native), fp16
lfm_lmh_f16.bin	NPU lm-head hidden[1,1024]→logits[1,65536] (tied), fp16
lfm_embed_f16.bin	token embedding table [65536,1024] f16 (host lookup)
tokenizer.json	LFM2 tokenizer