README.md

---
license: gemma
base_model: google/gemma-4-E2B-it
tags:
  - qualcomm
  - hexagon
  - npu
  - qnn
  - snapdragon
  - quantized
  - int16-activations
library_name: qai-hub
---

# Gemma-4-E2B-it — 8-bit and 4-bit quantization for the Qualcomm Hexagon NPU

`google/gemma-4-E2B-it` quantized two ways, both running on the Hexagon NPU:

- **8-bit quant** — int8 weights with int16 activations, compiled to **QNN context binaries**
  that run **100% on the Hexagon NPU** with no CPU or GPU fallback. Needs the QAIRT SDK.
- **4-bit quant** — `q4_0` group-32 weights as a **GGUF executed on the NPU through llama.cpp's
  `ggml-hexagon` backend**. No QAIRT SDK, smaller on disk, and faster on decode.

Most shipped mobile builds of this model use int8 activations. Keeping activations at 16 bits
costs some memory and holds onto accuracy: on a held-out 400-question MMLU slice the 8-bit
build is **statistically indistinguishable from the unquantized model**.

> **Read the verification table before using a binary.** Not every target below has been run
> on physical hardware, and this README states exactly which have. As of 2026-07-27 the 8-bit
> **prefill** binary is verified on both v79 and v81 silicon; **v81 decode is not verified.**

---

## 8-bit — verification status per target

Everything below was measured on **physical hardware** — Snapdragon 8 Elite (SM8750, HTP v79)
via Qualcomm Device Cloud, device `41c5710f`, 2026-07-26, and Snapdragon 8 Elite Gen 5
(SM8850, HTP v81) via Qualcomm AI Hub inference, 2026-07-27. Each table says which.

### v79 (hexagon-v79 / SM8750 / Snapdragon 8 Elite) — verified

| binary | role | on hardware | perf |
|---|---|---|---|
| `gemma4_decode_wgqa_int8kv_a16w8_v79.bin` | decode (recommended) | ✅ exact float match | **65.3 ms/step · 15.3 tok/s** |
| `gemma4_decode_wgqa_a16w8_v79.bin` | decode (no int8-KV) | ✅ exact float match, deterministic | 69.8 ms/step · 14.3 tok/s |
| `gemma4_trunk_a16w8_v79.bin` | prefill, fixed SEQ=128 | ✅ 12/12 next-token, cos 0.992 | — |

**Decode** — the generated text is token-for-token identical to the float ONNX reference:

```
prompt : "The capital of France is"   (Gemma-4 chat template)
device : 'The capital of France is **Paris**.'
float  : 'The capital of France is **Paris**.'
```

`The`/` capital`/` of`/` France`/` is`/` **`/`Paris`/`**.`, terminating correctly on `<turn|>`.
Two consecutive runs produced byte-identical output, so decoding is deterministic on device.

**Prefill** — 12 held-out chat prompts, each one forward pass, compared against float:
**12/12 (100%)** next-token top-1, hidden cosine mean 0.992 / min 0.986.

### Speed ladder (v79, 8-bit, AI Hub profile on real hardware)

| decode graph | ms/step | tok/s | speedup |
|---|---|---|---|
| naive full-KV | 307.9 | 3.25 | 1.0× |
| + windowed KV + broadcast-GQA | 69.8 | 14.3 | 4.41× |
| **+ int8-KV (full-attention slots)** | **65.3** | **15.3** | **4.71×** |

int8-KV cost no accuracy on the held-out check (32/32 content-token agreement, unchanged).

### v81 (hexagon-v81 / SM8850 / 8 Elite Gen 5) — prefill verified, decode still untested

Measured on **real Snapdragon 8 Elite Gen 5 silicon** via AI Hub inference, 2026-07-27.

| binary | role | on hardware | perf |
|---|---|---|---|
| `gemma4_trunk_a16w8_v81.bin` | prefill, fixed SEQ=128 | ✅ 11/12 next-token, cos mean 0.991 / min 0.981 | — |
| `gemma4_decode_wgqa_int8kv_a16w8_v81.bin` | decode | ⚠️ **still unverified** — see below | profiles at 57.4 ms/step (17.4 tok/s) |

**Prefill on v81 matches v79 bit-for-bit.** The 12 held-out prompts were run through the v81
trunk binary, and the *same* harness was run against the v79 trunk binary as a control, so the
two are directly comparable rather than being compared across measurement paths:

| trunk binary | device | prompts | next-token top-1 | hidden cos (mean / min) |
|---|---|---|---|---|
| `gemma4_trunk_a16w8_v81.bin` | SM8850 (v81), AI Hub | 12 | 11/12 | 0.99077 / 0.98050 |
| `gemma4_trunk_a16w8_v79.bin` | SM8750 (v79), AI Hub | 2 (control) | 1/2 — same prompt flips, same cosines | 0.99234, 0.98050 |
| `gemma4_trunk_a16w8_v79.bin` | SM8750 (v79), adb / Device Cloud | 12 | 12/12 | 0.992 / 0.986 |

The v79 rows are a **control on the measurement path**, not a second verification: the AI Hub
row deliberately re-ran only the harness-validation prompt and the one prompt v81 flipped.

The single v81 disagreement is `"Name a planet with rings."`, where float opens `'**'` and the
device opens `'The'` at cos 0.9805 — a near-tie between two plausible sentence openings, not a
degradation. **Re-running that exact prompt on the v79 binary reproduced the identical
mismatch at an identical cosine of 0.9804982542991638**, so it is a property of the 8-bit
quantization, not of v81. On this evidence the two architectures are numerically
indistinguishable on the trunk.

**Decode on v81 remains unverified.** An earlier spot-check of the v81 decode binary showed
hidden cosine degrading 0.912 → 0.848 → 0.794 → 0.839 across prefill steps 0–3, with the
hardware norm about half of float at step 2, and was stopped before the token comparison. That
result has **not** been explained or reproduced, and the trunk result above does not clear it:
the trunk is a different graph. Two candidate explanations remain open — the decode harness
itself, and the fact that the v81 decode binary was compiled from the **int8-KV** export
(`decode_wgqa_A16W8_int8kv_full`) whereas the token-exact v79 verification used the plain
WGQA export. Treat `gemma4_decode_wgqa_int8kv_a16w8_v81.bin` as *"failed a spot-check, cause
unknown"* — not as broken, and not as usable.

Why decode is expensive to verify: Qualcomm Device Cloud provisions **v79** parts only, so
there is no adb-attached v81 device, and AI Hub inference bills one farm job per decode step
(~23 jobs for one short sentence). The trunk is stateless — one forward per prompt — which is
why prefill could be verified for 12 jobs and decode was not.

**If you have v81 hardware**, the useful next step is decode: run the loop per *How to run*
and compare against float. If it diverges, re-verify against a v81 build of the *non*-int8-KV
export to isolate whether int8-KV interacts badly with v81, and check that your QAIRT install
ships a `hexagon-v81` skel matching the compile.

### What is *not* verified

- **Long-context / ring-wrap.** The windowed graph uses 512-entry ring buffers on the sliding
  layers. Prompts long enough to wrap the ring (>512 tokens) were **not** exercised on
  hardware. Verified prompts were ~14–18 tokens.
- **Prefill beyond 128 tokens.** The trunk graph is a fixed SEQ=128 window. Longer prompts are
  not covered by it at all.
- **Generative benchmarks (GSM8K etc).** Not measured. MMLU is one forward pass per question;
  multi-step generative reasoning compounds error over hundreds of steps and is untested here.
- **Throughput is NPU inference time** from an AI Hub profile job, not end-to-end tokens/s.
  Host-side embedding lookup and `lm_head` are excluded; a real application adds those, and the
  net-run harness used for correctness reloads the context each step so its wall-clock
  (~3.6 s/step) is **not** a throughput number.

---

## 8-bit accuracy

Held-out MMLU, 0-shot, chat-formatted, 400 questions disjoint from all calibration data:

| | accuracy |
|---|---|
| base model (float, ≡ HF) | 56.75% ± 2.48 |
| **8-bit (this build)** | **59.25% ± 2.46** |
| delta | +2.50 pp |
| random baseline | 25.00% |

The +2.50 pp delta is about one standard error — **not** evidence that quantization improves
the model. The correct reading is that 8-bit costs no measurable MMLU accuracy. Note the two
models disagree on ~24% of individual questions; they match in aggregate, not per-question.

---

## Two things you must get right

**1. Use the chat template.** The raw completion format makes this instruction-tuned model
degenerate. Verified on the *unquantized* model, so this is not a quantization artifact:

| format | output |
|---|---|
| raw + greedy | `' France is France is France is…'` |
| raw + temperature / top-p | byte-identical degeneration |
| raw + repetition_penalty 1.2 | byte-identical degeneration |
| **chat template + plain greedy** | **`'The capital of France is **Paris**.'`** |

Token layout (verified byte-exact against `transformers.apply_chat_template`):

```
[2 <bos>, 105 <|turn>, 2364 'user', 107 '\n'] + PROMPT + [106 <turn|>, 107, 105, 4368 'model', 107]
```

Stop generation on `106` (`<turn|>`) or `1` (`<eos>`).

**2. Match the mask constant.** Attention masks use a finite `NEG = -1e4`, not `-inf` or
`float32.min`. `-inf` cannot survive int16 activation quantization — it blows out the range so
real scores round to zero. `-1e4` still zeroes the softmax while leaving real scores resolved.
**The host must use the same value the model was calibrated with.**

---

## Architecture

The graph is split so the >2 GB vocab tensors never enter it:

- **Host (CPU/ARM):** token embedding lookup, per-layer embedding lookup, and the tied
  `lm_head` with `30·tanh(x/30)` softcap.
- **NPU:** the transformer decode graph, with the KV cache resident on device.

Gemma-4-E2B is dense: 35 layers, hidden 1536, GQA 8 query heads → 1 KV head, head_dim 256,
262144-token vocab, and hybrid attention (28 sliding-window layers of window 512, interleaved
with 7 full-attention; KV shared across the last 20 layers, so only 15 layers store KV).

### Why the fast graph is fast

Decode is **KV-attention-bound, not weight-bound**. The published binary uses two changes over
a naive full-KV decode graph:

- **Windowed KV** — sliding-window layers use a 512-entry ring buffer instead of a full 4096
  buffer; only the 3 full-attention layers keep 4096. KV traffic per step drops from ~288 MB
  to ~63 MB. The ring index is computed *inside* the graph as `cache_position % buf`, so the
  host just passes `pos`.
- **Broadcast GQA** — the `expand` op that materialized 1 KV head into 8 copies is removed
  (verified: 0 `Expand` nodes in the exported ONNX).

Net effect on v79: **307.9 ms → 69.8 ms per decode step (4.41×)**.

Credit: these two levers come from the `tps/` work in
[gemma-4-e2b-hexagon-npu](https://github.com/) — this repo contributes a corrected
quantization of that graph.

---

## Files

```
gemma4_decode_wgqa_int8kv_a16w8_v79.bin  1.9 GB  8-bit decode, v79  (recommended)
gemma4_decode_wgqa_a16w8_v79.bin         1.9 GB  8-bit decode, v79, no int8-KV
gemma4_trunk_a16w8_v79.bin               1.9 GB  8-bit prefill, v79, fixed SEQ=128
gemma4_decode_wgqa_int8kv_a16w8_v81.bin  1.9 GB  8-bit decode, v81   [UNVERIFIED - see above]
gemma4_trunk_a16w8_v81.bin               1.9 GB  8-bit prefill, v81  (verified on v81 silicon)

gemma4-e2b-w4.gguf                       2.6 GB  4-bit, runs via ggml-hexagon
gemma4-e2b-w4-mtp.gguf                    57 MB  4-bit MTP drafter (speculative decoding)

llama.cpp/bin/{llama-cli,llama-server,llama-bench}   arm64 Android, the 4-bit runtime
llama.cpp/lib/*.so                        22 MB  11 arm64 libs + libggml-htp-v79/v81.so
llama.cpp/LICENSE                                MIT (llama.cpp 0ef6e55) + DSP-library note

host-model/embed_tokens_weight.bf16            769 MB  token embeddings
host-model/embed_tokens_per_layer_weight.bf16  4.4 GB  per-layer embeddings
host-model/tokenizer.json                       31 MB
host-model/norm_weight.bf16                              final norm (diagnostics)

runtime/hostlib.py               host embeddings, chat template, lm_head + softcap
runtime/run_gate.py              host orchestrator (the autoregressive loop)
runtime/verify_trunk.py          prefill checker vs a float reference
runtime/stage_device.sh          push everything to an adb device
runtime/gate_ondevice_wgqa.sh    on-device decode step + KV rotation
runtime/gate_ondevice_int8kv.sh  same, int8-KV binary
runtime/gate_ondevice_trunk.sh   on-device prefill pass (no KV)
requirements.txt
```

The three `host-model` tensors are ~5.2 GB and stay on the host by design — putting the
262144-token vocab in the graph blows past ONNX's 2 GB protobuf limit.

---

# How to run the 8-bit build

## What you need that is NOT in this repo

**You cannot run this from this repo alone.** One dependency is missing by necessity:

1. **Qualcomm AI Engine Direct (QAIRT / QNN) SDK** — supplies `qnn-net-run` and
   `libQnnHtp*.so` plus the HTP `Stub`/`Skel` pair for your Hexagon version. These are
   Qualcomm-licensed and not redistributable here, so you must install the SDK yourself
   (free, from Qualcomm). Built and tested against **QAIRT 2.45**.
   - v79 needs `libQnnHtpV79Stub.so` + `libQnnHtpV79.so` / `libQnnHtpV79Skel.so`
   - v81 needs the **V81** equivalents. Check your SDK actually ships `hexagon-v81`;
     older installs do not.
2. **A device**: Snapdragon 8 Elite (SM8750, v79) or 8 Elite Gen 5 (SM8850, v81), reachable
   over `adb`. Qualcomm Device Cloud works — that is what this was verified on.
3. **Host Python 3.9+** with `numpy` and `tokenizers` (`pip install -r requirements.txt`).
   No torch, no transformers needed to *run* — only to reproduce the quantization.
4. **~6 GB free on the host** for the embedding tensors, **~4 GB free on the device**
   (`/data/local/tmp`) per pair of context binaries.

## Step 1 — get the repo

```bash
pip install -r requirements.txt
git lfs install
git clone https://huggingface.co/h2loop-ai/gemma-4-e2b-hexagon
cd gemma-4-e2b-hexagon
```

## Step 2 — connect the device

```bash
adb devices -l          # confirm your serial
```

On Qualcomm Device Cloud, tunnel the adb server first, then point adb at it:

```bash
ssh -i <your-qdc-key>.pem -L 5037:<QDC_HOST>:5037 -N sshtunnel@ssh.qdc.qualcomm.com &
export ADB_SERVER_SOCKET=tcp:127.0.0.1:5037
adb devices -l
```

Never run `adb kill-server` against that tunnel — it kills the *remote* pod's adb server,
which you cannot restart without portal access.

## Step 3 — stage onto the device

```bash
export QAIRT_DIR=/path/to/qairt/2.45.0.xxxxxx     # your SDK install
./runtime/stage_device.sh <serial> v79             # or: v81
```

This pushes `qnn-net-run`, the HTP libs/skels, the matching `*_v79.bin` context binaries, and
the on-device step scripts. Two 1.9 GB pushes over adb take a while; over a QDC tunnel a
single stream runs ~1 MB/s, so expect ~30 min unless you parallelise (see *Slow adb* below).

KV buffers are **not** pushed — `run_gate.py` creates them on device with `dd`.

## Step 4 — run the autoregressive loop

```bash
python runtime/run_gate.py \
  --prompt "The capital of France is" \
  --ntokens 14 \
  --adb-serial <serial> \
  --chat --wgqa \
  --script gate_ondevice_int8kv.sh
```

Expected output:

```
continuation: 'The capital of France is **Paris**.'
```

Flags that matter:

| flag | why |
|---|---|
| `--chat` | **required.** Without it the `-it` model degenerates into `' France is France is …'` |
| `--wgqa` | **required** for these binaries — selects 512-entry ring buffers and the 512-wide sliding mask |
| `--script` | pick the binary: `gate_ondevice_int8kv.sh` (recommended) or `gate_ondevice_wgqa.sh` |

Drop `--script` to use the non-int8-KV binary.

## Step 5 (optional) — check prefill

`verify_trunk.py` compares the trunk against a float reference. Producing that reference
needs `torch` + `transformers` on a host that knows the `gemma4` architecture (transformers
≥ 5.12 — older versions raise `KeyError: 'gemma4'`), so it is a reproduction step rather than
part of normal use.

---

# 4-bit — q4_0 weights on the Hexagon NPU

The 4-bit path uses a different runtime. It is not a QNN context binary: it is a **GGUF executed
on the Hexagon NPU through llama.cpp's `ggml-hexagon` backend**, which drives the DSP directly
over FastRPC instead of through QNN. It needs no QAIRT SDK.

On the same physical v79 silicon it is **2.5× faster on decode than the 8-bit build**, fits
entirely in one 2.44 GiB file with no host-side tensors, and costs ~0.4 pp of MMLU:

| | **4-bit / ggml-hexagon** | 8-bit / QNN |
|---|---|---|
| model on disk | **2.44 GiB total** | 3.54 GiB + 5.2 GB host tensors |
| decode, sustained on device | **37.8 – 38.5 tok/s** | 15.3 tok/s |
| prefill, NPU | **1145 tok/s** | — |
| peak device RSS | **3.84 GiB** | not measured |
| held-out MMLU | **57.76% ±0.99** | 59.25% ±2.46 |
| vs unquantized base | **−0.44 pp** | +2.50 pp |
| GSM8K (5-shot, chat) | **0.5200 / 0.6467** | not measured |
| embeddings | **inside the file** | 5.2 GB on host |
| QAIRT SDK required | **no** | yes |

**Granularity, not bit-width, is what decides accuracy here.** This build is group-32; at the
same 4 bits a per-channel grid costs ~5.3 pp of MMLU, measured by quantizing the same parent
checkpoint both ways on the same harness:

| grid, same parent checkpoint | MMLU (n=2280) |
|---|---|
| unquantized parent | 57.68% ±0.99 |
| group-32 (this build) | **57.76% ±0.99** |
| per-channel int4 | 52.46% ±1.01 |

Group-wise 4-bit was previously ruled out on this graph because LPBQ stores 4-bit values in an
int8 container and runs 9.3× slower. **That conclusion still holds for QNN** — it does not apply
here, because `ggml-hexagon` consumes `q4_0` blocks natively with its own HTP kernels rather
than going through QNN's blockwise path.

## 4-bit verification status

Measured on **physical Snapdragon 8 Elite (SM8750, HTP v79)** via Qualcomm Device Cloud, device
`87b3a4aa`, 2026-08-05. Every run started from 42 °C.

### Throughput

| dev | prefill pp128 | decode tg32 |
|---|---|---|
| **HTP0 (NPU)** | **1145.39 ± 7.89 tok/s** | 28.50 ± 0.06 tok/s |
| CPU (6 threads) | 225.17 ± 0.46 tok/s | 42.48 ± 0.08 tok/s |

**NPU prefill is 5.1× CPU; CPU decode is 1.5× NPU.** Decode is weight-bandwidth-bound, so the
NPU's compute advantage does not help it — prefill is compute-bound, where it does. A hybrid
placement (NPU prefill, CPU decode) is the strongest configuration on this hardware.

### Sustained decode — and a measurement discrepancy you should know about

Five consecutive 128-token requests through `llama-server` on HTP0:

| run | tok/s | peak RSS | hottest thermal zone |
|---|---|---|---|
| 1 | 38.47 | 4,029,912 kB | 79.2 °C |
| 2 | 38.27 | 4,031,796 kB | 83.4 °C |
| 3 | 38.13 | 4,031,796 kB | 83.8 °C |
| 4 | 38.06 | 4,031,796 kB | 86.1 °C |
| 5 | 37.79 | 4,031,796 kB | 87.2 °C |


### Latency and memory

- **TTFT 61.19 ms** for an 8-token prompt. Not comparable to context-512 TTFT figures.
- **Peak RSS 3.84 GiB** (`VmHWM` 4,031,796 kB) for a 2.44 GiB file — ~1.4 GiB of host-side
  buffers on top of the weights.
- The `ggml-hex` log reports the DSP session mapping budget as `vmem 3355443200` =
  **3.125 GiB**. RSS exceeds it, so not everything is resident in the session. A model whose
  weights alone approach 3.1 GiB will need multi-device layer splitting.

### Thermals

42 °C → **87.2 °C** peak across 5×128-token bursts, with **1.8% decode decay**. Hot but not
throttling within that window. A multi-minute soak has **not** been run — do not assume this
holds for continuous generation.

### Generation quality on device

Greedy, raw completion (no chat template):

| prompt | output |
|---|---|
| "The capital of France is" | " Paris." |
| "Who wrote Hamlet?" | "The answer is William Shakespeare." |
| "What color is grass?" | *(empty)* |
| "Name a country in South America." | **"Brazil"** |

The empty response on "What color is grass?" is a **prompt artifact, not a quantization
defect**: raw completion emits an immediate end-of-turn, and Google's own `q4_0` GGUF returns the
identical empty string on that prompt. Use the chat template.

### Accuracy — measured on CPU, not on device

| benchmark | 4-bit | parent (unquantized) | base fp32 |
|---|---|---|---|
| MMLU, 0-shot, n=2280 | **57.76% ±0.99** | 57.68% ±0.99 | 58.20% ±0.99 |
| GSM8K 5-shot chat, strict | **0.5200 ±0.041** | 0.5200 ±0.041 | — |
| GSM8K 5-shot chat, flexible | **0.6467 ±0.039** | 0.6467 ±0.039 | — |
| top-1 agreement vs parent | **98.2%** | — | — |
| KL divergence vs parent | **0.00173** | — | — |

GSM8K is **identical to the unquantized parent to the digit**, standard errors included. MMLU is
+0.08 pp — noise. Token-level top-1 agreement is 98.2% and KL is 1.7×10⁻³, so the two models are
not bit-identical per token; they match on the benchmarks in aggregate.

**How this was measured, and its limit:** the GGUF was dequantized and its tensors substituted
into the parent architecture, then evaluated with `lm-evaluation-harness` on CPU. That measures
the **weight grid**, which is what quantization changes — it does **not** exercise the HTP
kernels. On-device accuracy at scale is unverified; the four coherence prompts above are the only
device-side quality evidence.

Two harness details that otherwise waste time: MMLU needs `tokenizer.add_bos_token=True` or
loglikelihood-MC scores at chance; GSM8K needs `apply_chat_template=True` **and**
`add_bos_token` left off, because the template already emits `<bos>` and forcing it
double-prefixes. GSM8K run 0-shot raw scores `strict-match = 0.0` — the model never emits the
`#### N` format, so that is a broken measurement, not a bad model.

### What is not verified

- **v81.** `libggml-htp-v81.so` builds and is auto-selected by arch, but has never been run —
  Device Cloud provisions v79 only. Qualcomm's published figures for a `q4_0` GGUF on v81 show
  NPU decode *lower* than v79 (17.61 vs 26.66 tok/s at ctx 512) with prefill 36% higher, so do
  not assume v81 is an upgrade for decode.
- **Long context.** Nothing beyond depth 4096 exercised. The `tg32 @d4096` figure was itself
  unstable across sessions (21.90 ±0.97 vs 27.47 previously), so it is not quoted here.
- **Sustained multi-minute load**, batch > 1, and on-device accuracy at scale.

## How to run the 4-bit build

### What you need that is not in this repo

Only a device — **no QAIRT/QNN SDK, and no build step.** The `llama.cpp/` directory in this repo
is a prebuilt arm64-Android runtime, so unlike the 8-bit path this one is self-contained:

1. **A device**: SM8750 (v79) or SM8850 (v81) over `adb`.
2. **~3 GB free on the device**, ~4 GB RAM headroom for the process.

### What is in `llama.cpp/`

```
llama.cpp/bin/llama-cli  llama-server  llama-bench     arm64 Android, the three tools below
llama.cpp/lib/*.so                             22 MB   11 arm64 libs + the v79/v81 DSP libs
llama.cpp/LICENSE                                      MIT, plus the DSP-library note
```

Built from upstream llama.cpp **`0ef6e55`** in `ghcr.io/snapdragon-toolchain/arm64-android:v0.7`
(NDK r28b, Hexagon SDK 6.6). Hexagon support and the `gemma4` architecture both need a build from
2026-06 or later, so an older llama.cpp will not work. The commit is recorded inside
`libllama-common.so` if you want to check what you got.

The arm64 libraries are stripped of debug info (175 MB → 20 MB); every `PT_LOAD` segment and
every `SHF_ALLOC` section is byte-identical to the unstripped build, and the DSP libraries are
shipped as built. This is the exact dependency closure of the three tools — nothing spare, and
nothing missing.

`libggml-htp-v79.so` / `-v81.so` are the DSP-side kernels; the right one is selected
automatically from the detected arch, so unlike QNN there is no version pinning to get right.
They need **no Qualcomm libraries at runtime** — their `qurt_*`, `compute_resource_*` and
`dspqueue_*` symbols resolve on-device against the vendor DSP image. That is also why
`ADSP_LIBRARY_PATH` must include the vendor paths (Step 2). If you want v73/v75 as well, or a
build without the unused `libggml-opencl.so` dependency, build it yourself:

```bash
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cp docs/backend/snapdragon/CMakeUserPresets.json .

docker run --rm -u $(id -u):$(id -g) -v $(pwd):/workspace --platform linux/amd64 \
  ghcr.io/snapdragon-toolchain/arm64-android:v0.7 \
  bash -lc 'cd /workspace \
    && cmake --preset arm64-android-snapdragon-release -B build-snapdragon \
    && cmake --build build-snapdragon -j 32 \
    && cmake --install build-snapdragon --prefix pkg-snapdragon/llama.cpp'
```

### Step 1 — fetch just the 4-bit files, and push

A full clone of this repo is ~17 GB, almost all of it the 8-bit binaries and their host tensors.
The 4-bit path needs 2.7 GB of it, so fetch only that:

```bash
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/h2loop-ai/gemma-4-e2b-hexagon
cd gemma-4-e2b-hexagon
git lfs pull --include="gemma4-e2b-w4*.gguf,llama.cpp/lib/*"
```

```bash
adb push llama.cpp /data/local/tmp/
adb shell chmod +x /data/local/tmp/llama.cpp/bin/llama-cli \
                   /data/local/tmp/llama.cpp/bin/llama-server \
                   /data/local/tmp/llama.cpp/bin/llama-bench
adb shell mkdir -p /data/local/tmp/gguf
adb push gemma4-e2b-w4.gguf /data/local/tmp/gguf/
```

`adb push` does not preserve the executable bit, hence the `chmod`. Over a QDC tunnel expect
2.6–7.6 MB/s, so 10–20 min for the model — the runtime itself is 22 MB and lands in seconds. The
*Slow adb* section below applies to the model.

### Step 2 — the one non-obvious requirement

**`ADSP_LIBRARY_PATH` is mandatory.** Without it the DSP loader cannot find the skel, and the
error looks like a hardware or arch problem but is not:

```
ggml-hex: failed to open session 0 : error 0x80000406
```

```bash
adb shell
cd /data/local/tmp/llama.cpp
export LD_LIBRARY_PATH=/data/local/tmp/llama.cpp/lib
export ADSP_LIBRARY_PATH="/data/local/tmp/llama.cpp/lib;/system/lib/rfsa/adsp;/vendor/lib/rfsa/adsp"
```

A healthy session logs:

```
ggml-hex: Hexagon Arch version v79
ggml-hex: HTP0 hwinfo: threads 6, hvx 6, hmx 1, vtcm 8 MB
ggml-hex: HTP0 new session : ... file:///libggml-htp-v79.so ... _dom=cdsp
```

### Step 3 — run

Benchmark NPU and CPU in one invocation:

```bash
./bin/llama-bench -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf \
  -p 128 -n 32 -t 6 -r 2 -fa 1 -dev HTP0,none
```

Interactive generation on the NPU:

```bash
./bin/llama-cli -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf -dev HTP0 -t 6 -c 4096 -fa on
```

Server — the path the quoted decode figures come from:

```bash
./bin/llama-server -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf \
  -dev HTP0 -t 6 -c 4096 -fa on --parallel 1 --no-cont-batching --port 8099
```

```bash
curl -s localhost:8099/completion -H 'Content-Type: application/json' \
  -d '{"prompt":"Write a paragraph about on-device inference.","n_predict":128,"temperature":0}'
```

The `timings` block carries `predicted_per_second`, `prompt_ms` (TTFT) and, with a drafter
loaded, `draft_n` / `draft_n_accepted`.

| flag | why |
|---|---|
| `-dev HTP0` | run on the NPU. `-dev none` = CPU. `-dev HTP0,none` benchmarks both |
| `-fa on` | flash attention; required for the windowed-KV path |
| `-t 6` | matches the HTP's 6 HVX contexts |
| `--parallel 1 --no-cont-batching` | single-stream, matching the quoted figures |
| `-c 4096` | context; nothing beyond this is verified |

Quantized KV (`-ctk q8_0 -ctv q8_0`) is **not supported** on this backend — it aborts with
`GGML_ASSERT(offset == 0)` in `ggml-hexagon.cpp`. On CPU it works but is slower than `f16`
(25.66 vs 34.38 tok/s at depth 4096).

### Step 4 — speculative decoding with the MTP drafter

`gemma4-e2b-w4-mtp.gguf` is a Multi-Token Prediction drafter (~57 MB) that shares the target's
KV cache. It needs a llama.cpp build newer than 2026-06-08 (arch `gemma4-assistant`).

```bash
adb push gemma4-e2b-w4-mtp.gguf /data/local/tmp/gguf/
```

```bash
./bin/llama-server -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf \
  -md /data/local/tmp/gguf/gemma4-e2b-w4-mtp.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -dev HTP0 -t 6 -c 4096 -fa on --port 8099
```

The drafter cannot be instantiated standalone (`failed to create context with model` — it
requires the target's context), and the startup warning `[spec] failed to measure draft model
memory` is benign; the drafter loads afterwards and `common_speculative_init_result` confirms it.

### 4-bit troubleshooting

| symptom | cause |
|---|---|
| `Permission denied` running `./bin/llama-cli` | `adb push` drops the executable bit; `chmod +x` the three tools (Step 1) |
| `library "libllama-common.so" not found` | `LD_LIBRARY_PATH` not pointing at `llama.cpp/lib` (Step 2) |
| `failed to open session 0 : error 0x80000406` | `ADSP_LIBRARY_PATH` not set — the DSP cannot find the skel |
| `GGML_ASSERT(offset == 0)` in `ggml-hexagon.cpp` | quantized KV cache (`-ctk`/`-ctv`) unsupported on HTP |
| `failed to create context with model 'gemma4-e2b-w4-mtp.gguf'` | the drafter needs the target's context; do not load it alone |
| `503 {"error":"Loading model"}` | server not ready. `/health` returns 503 while loading, so poll for HTTP 200, not just any response |
| decode slower on NPU than CPU | expected — decode is bandwidth-bound. Use the NPU for prefill |
| `llama-cli` idles at a `>` prompt | it entered conversation mode; use `llama-server`, or check this build's single-turn flag |
| arch `gemma4` / `gemma4-assistant` unknown | llama.cpp too old |

## 4-bit limitations

- **Decode figure is unsettled** — 28.50 (llama-bench) vs ~38 tok/s (llama-server), cause
  unresolved. See above.
- **On-device accuracy is unverified at scale.** MMLU/GSM8K were measured on CPU against the
  weight grid, not through the HTP kernels.
- **v81 has never been run**, and Qualcomm's own v81 figures suggest decode is *worse* there.
- **No compiled artifact.** Unlike the 8-bit context binaries, there is no AOT compile
  step — `ggml-hexagon` reads the GGUF at runtime. Converting this to a QNN context binary is
  not a shortcut: QAIRT's `gguf_builder` dequantizes rather than preserving the group-32 grid,
  which is the property doing the work.
- **Quantized KV cache unsupported** on the HTP backend.
- **Beyond 4096 context, batch > 1, and multi-minute sustained load** are all untested.

---

## How the loop actually works

`run_gate.py` is the reference implementation, and deliberately simple:

1. Tokenize with the chat template (`hostlib.encode_chat`).
2. Zero the KV buffers on device.
3. For each position: look up embeddings on the host, write the six small per-step tensors,
   run `qnn-net-run` once on device, pull back `hidden` (1536 floats).
4. Apply the tied `lm_head` + `30·tanh(x/30)` softcap on the host, take the argmax.
5. The on-device script renames `present_* → past_*` so KV never crosses adb.
6. Stop on `<turn|>` (106) or `<eos>` (1).

**This harness is for correctness, not speed.** It re-loads the 1.9 GB context binary every
step, so its wall clock (~3.6 s/token) is ~50× worse than the NPU's actual 65 ms. A real
application loads the context **once**, keeps KV device-resident, and does embeddings +
`lm_head` in-process. Building that is left to you.

## Slow adb

A single adb stream over a QDC tunnel is bandwidth-delay-product limited (~1 MB/s), not
bandwidth limited. Splitting the binary and pushing chunks over separate SSH tunnels (one
local port each) reached ~7 MB/s:

```bash
split -n 6 -d gemma4_decode_wgqa_int8kv_a16w8_v79.bin chunk.
# ...one `ssh -L 503X:$HOST:5037` per chunk, then push each with its own
#    ADB_SERVER_SOCKET=tcp:127.0.0.1:503X, then on device:
adb shell 'cd /data/local/tmp/gemma/artifacts && cat chunk.* > out.bin && rm chunk.*'
```

Verify the checksum afterwards (`SHA256SUMS`) — and wait for **all** pushes to finish before
concatenating, or you will silently assemble a truncated file.

## Troubleshooting

| symptom | cause |
|---|---|
| `Could not create context from binary` | HTP arch mismatch — a v81 binary will not load on a v79 device, or vice versa |
| `Cannot assign data from unexpected type. Expected int32, got int64` | binaries are built with `--truncate_64bit_io`, so `position_ids`/`cache_position` are **int32** |
| Output repeats `' France is France is …'` | `--chat` missing |
| Fluent but wrong answer | binary/mask mismatch — the host `NEG` must be `-1e4`, matching calibration |
| Garbage tokens, hidden norm ≈ 0 | wrong context binary, or KV buffers not zeroed before position 0 |
| Different output across identical runs | a KV buffer was corrupted mid-push; re-seed (`dd` on device) and retry |
| Fluent garbage from a binary you compiled yourself | local `qnn-context-binary-generator` emits `UFIXED_POINT_16` IO; the harness writes fp32. Use the AI Hub build, or check `graphInputs[].dataType` |
| `Could not create context from binary` on a binary that used to work | QAIRT runtime version does not match the version that compiled it |

## Reproducing the 8-bit quantization

AIMET `QuantizationSimModel`, `param_type=int8`, `activation_type=int16`,
`quant_scheme=min_max`, calibrated on **real activations captured from chat-formatted decode
loops** — not random noise, and not raw-format text.

Both of those details matter and each caused a distinct on-device failure:

- Calibrating on `np.random.randn` produced a binary whose residual stream collapsed to zero
  on hardware (final hidden norm 0.0000 → pure noise tokens), even though it compiled fine.
- Calibrating on *raw-format* text and then running *chat-format* prompts produced fluent but
  unfaithful output on hardware — the chat template's special tokens hit activation ranges the
  quantizer never observed.

`min_max` outperformed `tf_enhanced` here: on int16 there are 65k levels, so range *coverage*
matters more than outlier clipping, and `tf_enhanced` mis-estimated the range badly enough to
inflate hidden norms ~10×.

## 8-bit limitations

- Fixed 4096 context.
- Prefill is not included in this repo; the decode graph can prefill token-by-token, which is
  slow for long prompts.
- Batch size 1 only.
- `--truncate_64bit_io` at compile time means index inputs (`position_ids`, `cache_position`)
  are **int32** on the compiled binary, though the float ONNX takes int64.