mlboydaisuke/gemma-3-270m-it-NPU-LiteRT

gemma-3-270m-it — NPU (SRQ) bundle, compiled on-device

A static-range-quantized (SRQ) .litertlm of google/gemma-3-270m-it that a Qualcomm device compiles for its own Hexagon NPU (HTP) at load time (JIT). The bundle contains no precompiled QNN context — one file serves any supported SoC, and no Qualcomm-generated artifact ships inside it.

Verified on: Snapdragon SM8850 (Hexagon v81), QAIRT 2.47, LiteRT-LM 0.16-line build with the NPU backend. Other SoCs and runtime versions are untested.

Run it

Your app needs ten Qualcomm/LiteRT runtime libraries in the APK and two Environment options. The full recipe — including the silent-CPU-fallback trap that makes a misconfigured run look successful — is one page:

Android NPU recipe

Know before you ship:

• Every load recompiles the model — ≈16 s on SM8850. The LLM runtime currently has no compile cache for the NPU path. This is per process start, not per prompt.
• The transformer runs fully on the NPU (888/888 prefill + 869/869 decode ops partitioned to HTP). The tiny auxiliary graphs (RoPE/mask/cache-update, embedder) run on CPU; measured cost ≈2.8% of decode time.
• The KV cache length is 896 by design, not 1024: on SM8850 the decode mask concat crossing 1 MiB makes the HTP apply the mask to the wrong rows and quality collapses (observed on SM8850, reported upstream: litert-torch #1184).

Measured performance

build	prefill (tok/s)	decode (tok/s)	load
JIT, cache-1024 variant	1899	219.7	≈16 s (compiles)
AOT (offline-compiled) control, cache 896	1747	199.8	≈1 s

Single runs on Galaxy S26 (SM8850), QAIRT 2.47, LiteRT-LM 0.16-line build, 16-token prompt, 64 output tokens, per-prompt process start. The 16-token prompt makes the prefill column a floor, not a throughput claim. The two rows differ in cache length as noted, so they are indicative, not a controlled JIT-vs-AOT comparison.

Quality

On-device 45-prompt gate (SM8850, 64 max output tokens, greedy):

arm	correct	note
fp32 reference (host)	28/45
JIT, cache 896 (this configuration)	29/45	generations string-identical to the AOT build on 45/45 prompts
JIT/AOT, cache 1024 or 960	12/45	the 1 MiB mask threshold above — why this bundle uses 896

The uploaded file itself was additionally gated on the host interpreter: 28/45 correct (equal to fp32), same verdict on 41/45. The conversion pipeline is byte-reproducible in every weight and quantization parameter — two fresh builds differ only in a timestamp, a UUID, and the serialization order of one stop-token list (242 bytes of 456,853,286).

Reproduce

Stages 1–3 of litert-torch's experimental npu_export pipeline (tree upstream/main @ 8379afb plus the open fix PR #1178), gemma3-270m SM8850 preset, cache_length=896, 10 calibration prompts × 32 decode steps, stopping after the static-range quantize stage. Step-by-step: recipe §4.

File

file	size (B)	sha256
gemma-3-270m-it_npu-srq_c896.litertlm	456,853,286	baafebc1c68ab3667740d478dfe8d1bb5be75689cce06108d10a3188b3e02bcc

---

Gemma is provided under and subject to the Gemma Terms of Use. This repository distributes a quantized conversion of google/gemma-3-270m-it; use of the model remains subject to those terms and the Gemma Prohibited Use Policy.