Add README, host runtime, and device staging scripts

.gitattributes

@@ -1,35 +1,3 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.bf16 filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

MANIFEST.txt

@@ -0,0 +1,4 @@
+# Large files are NOT copied into this directory (they already exist elsewhere on disk and
+# are 1.9GB / 5.2GB). upload.sh streams them straight to the Hub from these paths.
+gemma4_decode_wgqa_a16w8_v79.bin  <- edge-ai-recon/qdc-gemma-v79/artifacts/gemma4_decode_wgqa_a16w8_v79.bin
+host-model/                       <- edge-ai-recon/qdc-gemma-v79/host-model/

README.md

@@ -0,0 +1,355 @@
+---
+license: gemma
+base_model: google/gemma-4-E2B-it
+tags:
+  - qualcomm
+  - hexagon
+  - npu
+  - qnn
+  - snapdragon
+  - quantized
+  - int16-activations
+library_name: qai-hub
+---
+
+# Gemma-4-E2B-it — A16W8 for the Qualcomm Hexagon NPU
+
+`google/gemma-4-E2B-it` quantized to **A16W8** (int16 activations, int8 weights) and compiled
+to **QNN context binaries** that run **100% on the Hexagon NPU** — no CPU or GPU fallback.
+
+Most shipped mobile builds of this model use int8 activations (GGUF-style `q4_0`). Keeping
+activations at 16 bits costs some memory and holds onto accuracy: on a held-out 400-question
+MMLU slice this build is **statistically indistinguishable from the unquantized model**.
+
+> **Read the verification table before using a binary.** Not every target below has been run
+> on physical hardware, and this README states exactly which have.
+
+---
+
+## Verification status per target
+
+Everything below was measured on a **physical Snapdragon 8 Elite (SM8750, HTP v79)** via
+Qualcomm Device Cloud, device `41c5710f`, 2026-07-26 — except where the table says otherwise.
+
+### v79 (hexagon-v79 / SM8750 / Snapdragon 8 Elite) — verified
+
+| binary | role | on hardware | perf |
+|---|---|---|---|
+| `gemma4_decode_wgqa_int8kv_a16w8_v79.bin` | decode (recommended) | ✅ exact float match | **65.3 ms/step · 15.3 tok/s** |
+| `gemma4_decode_wgqa_a16w8_v79.bin` | decode (no int8-KV) | ✅ exact float match, deterministic | 69.8 ms/step · 14.3 tok/s |
+| `gemma4_trunk_a16w8_v79.bin` | prefill, fixed SEQ=128 | ✅ 12/12 next-token, cos 0.992 | — |
+
+**Decode** — the generated text is token-for-token identical to the float ONNX reference:
+
+```
+prompt : "The capital of France is"   (Gemma-4 chat template)
+device : 'The capital of France is **Paris**.'
+float  : 'The capital of France is **Paris**.'
+```
+
+`The`/` capital`/` of`/` France`/` is`/` **`/`Paris`/`**.`, terminating correctly on `<turn|>`.
+Two consecutive runs produced byte-identical output, so decoding is deterministic on device.
+
+**Prefill** — 12 held-out chat prompts, each one forward pass, compared against float:
+**12/12 (100%)** next-token top-1, hidden cosine mean 0.992 / min 0.986.
+
+### Speed ladder (v79, A16W8, AI Hub profile on real hardware)
+
+| decode graph | ms/step | tok/s | speedup |
+|---|---|---|---|
+| naive full-KV | 307.9 | 3.25 | 1.0× |
+| + windowed KV + broadcast-GQA | 69.8 | 14.3 | 4.41× |
+| **+ int8-KV (full-attention slots)** | **65.3** | **15.3** | **4.71×** |
+
+int8-KV cost no accuracy on the held-out check (32/32 content-token agreement, unchanged).
+
+### v81 (hexagon-v81 / SM8850 / 8 Elite Gen 5)
+
+See the per-file table above for what is published. v81 status is stated separately below
+because it could not be verified the same way: **Qualcomm Device Cloud only provisions v79
+parts**, so no adb-attached v81 device was available.
+
+### What is *not* verified
+
+- **Long-context / ring-wrap.** The windowed graph uses 512-entry ring buffers on the sliding
+  layers. Prompts long enough to wrap the ring (>512 tokens) were **not** exercised on
+  hardware. Verified prompts were ~14–18 tokens.
+- **Prefill beyond 128 tokens.** The trunk graph is a fixed SEQ=128 window. Longer prompts are
+  not covered by it at all.
+- **Generative benchmarks (GSM8K etc).** Not measured. MMLU is one forward pass per question;
+  multi-step generative reasoning compounds error over hundreds of steps and is untested here.
+- **Throughput is NPU inference time** from an AI Hub profile job, not end-to-end tokens/s.
+  Host-side embedding lookup and `lm_head` are excluded; a real application adds those, and the
+  net-run harness used for correctness reloads the context each step so its wall-clock
+  (~3.6 s/step) is **not** a throughput number.
+
+---
+
+## Accuracy
+
+Held-out MMLU, 0-shot, chat-formatted, 400 questions disjoint from all calibration data:
+
+| | accuracy |
+|---|---|
+| base model (float, ≡ HF) | 56.75% ± 2.48 |
+| **A16W8 (this build)** | **59.25% ± 2.46** |
+| delta | +2.50 pp |
+| random baseline | 25.00% |
+
+The +2.50 pp delta is about one standard error — **not** evidence that quantization improves
+the model. The correct reading is that A16W8 costs no measurable MMLU accuracy. Note the two
+models disagree on ~24% of individual questions; they match in aggregate, not per-question.
+
+---
+
+## Two things you must get right
+
+**1. Use the chat template.** The raw completion format makes this instruction-tuned model
+degenerate. Verified on the *unquantized* model, so this is not a quantization artifact:
+
+| format | output |
+|---|---|
+| raw + greedy | `' France is France is France is…'` |
+| raw + temperature / top-p | byte-identical degeneration |
+| raw + repetition_penalty 1.2 | byte-identical degeneration |
+| **chat template + plain greedy** | **`'The capital of France is **Paris**.'`** |
+
+Token layout (verified byte-exact against `transformers.apply_chat_template`):
+
+```
+[2 <bos>, 105 <|turn>, 2364 'user', 107 '\n'] + PROMPT + [106 <turn|>, 107, 105, 4368 'model', 107]
+```
+
+Stop generation on `106` (`<turn|>`) or `1` (`<eos>`).
+
+**2. Match the mask constant.** Attention masks use a finite `NEG = -1e4`, not `-inf` or
+`float32.min`. `-inf` cannot survive int16 activation quantization — it blows out the range so
+real scores round to zero. `-1e4` still zeroes the softmax while leaving real scores resolved.
+**The host must use the same value the model was calibrated with.**
+
+---
+
+## Architecture
+
+The graph is split so the >2 GB vocab tensors never enter it:
+
+- **Host (CPU/ARM):** token embedding lookup, per-layer embedding lookup, and the tied
+  `lm_head` with `30·tanh(x/30)` softcap.
+- **NPU:** the transformer decode graph, with the KV cache resident on device.
+
+Gemma-4-E2B is dense: 35 layers, hidden 1536, GQA 8 query heads → 1 KV head, head_dim 256,
+262144-token vocab, and hybrid attention (28 sliding-window layers of window 512, interleaved
+with 7 full-attention; KV shared across the last 20 layers, so only 15 layers store KV).
+
+### Why the fast graph is fast
+
+Decode is **KV-attention-bound, not weight-bound**. The published binary uses two changes over
+a naive full-KV decode graph:
+
+- **Windowed KV** — sliding-window layers use a 512-entry ring buffer instead of a full 4096
+  buffer; only the 3 full-attention layers keep 4096. KV traffic per step drops from ~288 MB
+  to ~63 MB. The ring index is computed *inside* the graph as `cache_position % buf`, so the
+  host just passes `pos`.
+- **Broadcast GQA** — the `expand` op that materialized 1 KV head into 8 copies is removed
+  (verified: 0 `Expand` nodes in the exported ONNX).
+
+Net effect on v79: **307.9 ms → 69.8 ms per decode step (4.41×)**.
+
+Credit: these two levers come from the `tps/` work in
+[gemma-4-e2b-hexagon-npu](https://github.com/) — this repo contributes a corrected
+quantization of that graph.
+
+---
+
+## Files
+
+```
+gemma4_decode_wgqa_int8kv_a16w8_v79.bin  1.9 GB  decode, v79  (recommended)
+gemma4_decode_wgqa_a16w8_v79.bin         1.9 GB  decode, v79, no int8-KV
+gemma4_trunk_a16w8_v79.bin               1.9 GB  prefill, v79, fixed SEQ=128
+gemma4_decode_wgqa_int8kv_a16w8_v81.bin  1.9 GB  decode, v81
+gemma4_trunk_a16w8_v81.bin               1.9 GB  prefill, v81
+
+host-model/embed_tokens_weight.bf16            769 MB  token embeddings
+host-model/embed_tokens_per_layer_weight.bf16  4.4 GB  per-layer embeddings
+host-model/tokenizer.json                       31 MB
+host-model/norm_weight.bf16                              final norm (diagnostics)
+
+runtime/hostlib.py               host embeddings, chat template, lm_head + softcap
+runtime/run_gate.py              host orchestrator (the autoregressive loop)
+runtime/verify_trunk.py          prefill checker vs a float reference
+runtime/stage_device.sh          push everything to an adb device
+runtime/gate_ondevice_wgqa.sh    on-device decode step + KV rotation
+runtime/gate_ondevice_int8kv.sh  same, int8-KV binary
+runtime/gate_ondevice_trunk.sh   on-device prefill pass (no KV)
+requirements.txt
+```
+
+The three `host-model` tensors are ~5.2 GB and stay on the host by design — putting the
+262144-token vocab in the graph blows past ONNX's 2 GB protobuf limit.
+
+---
+
+# How to run
+
+## What you need that is NOT in this repo
+
+**You cannot run this from this repo alone.** One dependency is missing by necessity:
+
+1. **Qualcomm AI Engine Direct (QAIRT / QNN) SDK** — supplies `qnn-net-run` and
+   `libQnnHtp*.so` plus the HTP `Stub`/`Skel` pair for your Hexagon version. These are
+   Qualcomm-licensed and not redistributable here, so you must install the SDK yourself
+   (free, from Qualcomm). Built and tested against **QAIRT 2.45**.
+   - v79 needs `libQnnHtpV79Stub.so` + `libQnnHtpV79.so` / `libQnnHtpV79Skel.so`
+   - v81 needs the **V81** equivalents. Check your SDK actually ships `hexagon-v81`;
+     older installs do not.
+2. **A device**: Snapdragon 8 Elite (SM8750, v79) or 8 Elite Gen 5 (SM8850, v81), reachable
+   over `adb`. Qualcomm Device Cloud works — that is what this was verified on.
+3. **Host Python 3.9+** with `numpy` and `tokenizers` (`pip install -r requirements.txt`).
+   No torch, no transformers needed to *run* — only to reproduce the quantization.
+4. **~6 GB free on the host** for the embedding tensors, **~4 GB free on the device**
+   (`/data/local/tmp`) per pair of context binaries.
+
+## Step 1 — get the repo
+
+```bash
+pip install -r requirements.txt
+git lfs install
+git clone https://huggingface.co/r-4ti0/gemma-4-e2b-a16w8-hexagon
+cd gemma-4-e2b-a16w8-hexagon
+```
+
+## Step 2 — connect the device
+
+```bash
+adb devices -l          # confirm your serial
+```
+
+On Qualcomm Device Cloud, tunnel the adb server first, then point adb at it:
+
+```bash
+ssh -i <your-qdc-key>.pem -L 5037:<QDC_HOST>:5037 -N sshtunnel@ssh.qdc.qualcomm.com &
+export ADB_SERVER_SOCKET=tcp:127.0.0.1:5037
+adb devices -l
+```
+
+Never run `adb kill-server` against that tunnel — it kills the *remote* pod's adb server,
+which you cannot restart without portal access.
+
+## Step 3 �� stage onto the device
+
+```bash
+export QAIRT_DIR=/path/to/qairt/2.45.0.xxxxxx     # your SDK install
+./runtime/stage_device.sh <serial> v79             # or: v81
+```
+
+This pushes `qnn-net-run`, the HTP libs/skels, the matching `*_v79.bin` context binaries, and
+the on-device step scripts. Two 1.9 GB pushes over adb take a while; over a QDC tunnel a
+single stream runs ~1 MB/s, so expect ~30 min unless you parallelise (see *Slow adb* below).
+
+KV buffers are **not** pushed — `run_gate.py` creates them on device with `dd`.
+
+## Step 4 — run the autoregressive loop
+
+```bash
+python runtime/run_gate.py \
+  --prompt "The capital of France is" \
+  --ntokens 14 \
+  --adb-serial <serial> \
+  --chat --wgqa \
+  --script gate_ondevice_int8kv.sh
+```
+
+Expected output:
+
+```
+continuation: 'The capital of France is **Paris**.'
+```
+
+Flags that matter:
+
+| flag | why |
+|---|---|
+| `--chat` | **required.** Without it the `-it` model degenerates into `' France is France is …'` |
+| `--wgqa` | **required** for these binaries — selects 512-entry ring buffers and the 512-wide sliding mask |
+| `--script` | pick the binary: `gate_ondevice_int8kv.sh` (recommended) or `gate_ondevice_wgqa.sh` |
+
+Drop `--script` to use the non-int8-KV binary.
+
+## Step 5 (optional) — check prefill
+
+`verify_trunk.py` compares the trunk against a float reference. Producing that reference
+needs `torch` + `transformers` on a host that knows the `gemma4` architecture (transformers
+≥ 5.12 — older versions raise `KeyError: 'gemma4'`), so it is a reproduction step rather than
+part of normal use.
+
+## How the loop actually works
+
+`run_gate.py` is the reference implementation, and deliberately simple:
+
+1. Tokenize with the chat template (`hostlib.encode_chat`).
+2. Zero the KV buffers on device.
+3. For each position: look up embeddings on the host, write the six small per-step tensors,
+   run `qnn-net-run` once on device, pull back `hidden` (1536 floats).
+4. Apply the tied `lm_head` + `30·tanh(x/30)` softcap on the host, take the argmax.
+5. The on-device script renames `present_* → past_*` so KV never crosses adb.
+6. Stop on `<turn|>` (106) or `<eos>` (1).
+
+**This harness is for correctness, not speed.** It re-loads the 1.9 GB context binary every
+step, so its wall clock (~3.6 s/token) is ~50× worse than the NPU's actual 65 ms. A real
+application loads the context **once**, keeps KV device-resident, and does embeddings +
+`lm_head` in-process. Building that is left to you.
+
+## Slow adb
+
+A single adb stream over a QDC tunnel is bandwidth-delay-product limited (~1 MB/s), not
+bandwidth limited. Splitting the binary and pushing chunks over separate SSH tunnels (one
+local port each) reached ~7 MB/s:
+
+```bash
+split -n 6 -d gemma4_decode_wgqa_int8kv_a16w8_v79.bin chunk.
+# ...one `ssh -L 503X:$HOST:5037` per chunk, then push each with its own
+#    ADB_SERVER_SOCKET=tcp:127.0.0.1:503X, then on device:
+adb shell 'cd /data/local/tmp/gemma/artifacts && cat chunk.* > out.bin && rm chunk.*'
+```
+
+Verify the checksum afterwards (`SHA256SUMS`) — and wait for **all** pushes to finish before
+concatenating, or you will silently assemble a truncated file.
+
+## Troubleshooting
+
+| symptom | cause |
+|---|---|
+| `Could not create context from binary` | HTP arch mismatch — a v81 binary will not load on a v79 device, or vice versa |
+| `Cannot assign data from unexpected type. Expected int32, got int64` | binaries are built with `--truncate_64bit_io`, so `position_ids`/`cache_position` are **int32** |
+| Output repeats `' France is France is …'` | `--chat` missing |
+| Fluent but wrong answer | binary/mask mismatch — the host `NEG` must be `-1e4`, matching calibration |
+| Garbage tokens, hidden norm ≈ 0 | wrong context binary, or KV buffers not zeroed before position 0 |
+| Different output across identical runs | a KV buffer was corrupted mid-push; re-seed (`dd` on device) and retry |
+
+## Reproducing the quantization
+
+AIMET `QuantizationSimModel`, `param_type=int8`, `activation_type=int16`,
+`quant_scheme=min_max`, calibrated on **real activations captured from chat-formatted decode
+loops** — not random noise, and not raw-format text.
+
+Both of those details matter and each caused a distinct on-device failure:
+
+- Calibrating on `np.random.randn` produced a binary whose residual stream collapsed to zero
+  on hardware (final hidden norm 0.0000 → pure noise tokens), even though it compiled fine.
+- Calibrating on *raw-format* text and then running *chat-format* prompts produced fluent but
+  unfaithful output on hardware — the chat template's special tokens hit activation ranges the
+  quantizer never observed.
+
+`min_max` outperformed `tf_enhanced` here: on int16 there are 65k levels, so range *coverage*
+matters more than outlier clipping, and `tf_enhanced` mis-estimated the range badly enough to
+inflate hidden norms ~10×.
+
+## Limitations
+
+- Fixed 4096 context.
+- Prefill is not included in this repo; the decode graph can prefill token-by-token, which is
+  slow for long prompts.
+- Batch size 1 only.
+- `--truncate_64bit_io` at compile time means index inputs (`position_ids`, `cache_position`)
+  are **int32** on the compiled binary, though the float ONNX takes int64.

SHA256SUMS

@@ -0,0 +1 @@
+9e784f7dd2240dacaf721a7f82101730f614f0b96330643a4ad25225cdf2c722  gemma4_decode_wgqa_a16w8_v79.bin

requirements.txt

@@ -0,0 +1,2 @@
+numpy>=1.24
+tokenizers>=0.15

runtime/gate_ondevice_int8kv.sh

@@ -0,0 +1,58 @@
+#!/system/bin/sh
+# On-device single-step decode via qnn-net-run against the prebuilt A16W8 v79 context binary.
+# Runs entirely on the device: KV present->past rotation is local file mv (no host transfer).
+# Host pushes only the tiny per-step inputs (inputs_embeds, per_layer_inputs, position_ids,
+# cache_position, full_mask, sliding_mask) and pulls only hidden.raw.
+#
+# Layout on device (BASE=/data/local/tmp/gemma):
+#   $BASE/bin/qnn-net-run, $BASE/lib/*.so, $BASE/dsp/*.so
+#   $BASE/artifacts/gemma4_decode_a16w8_v79.bin
+#   $BASE/kv/past_{k,v}_<i>.raw     (persist across steps; seeded to zeros at step 0)
+#   $BASE/step/<small per-step input .raw files>
+#   $BASE/step/in.txt               (input_list: one line, space-separated name:=path)
+#   $BASE/out/                      (net-run output dir -> hidden.raw + present_*.raw)
+set -e
+BASE=/data/local/tmp/gemma
+export LD_LIBRARY_PATH=$BASE/lib:/system/lib64:/vendor/lib64
+export ADSP_LIBRARY_PATH="$BASE/dsp;/vendor/dsp/cdsp;/vendor/lib/rfsa/cdsp;/vendor/dsp"
+export LD_PRELOAD=/system/lib64/libbinder.so   # Android 16 linker fix (from qdc-session-notes)
+
+STEP_DIR=$BASE/step
+KV=$BASE/kv
+OUT=$BASE/out
+rm -rf "$OUT"; mkdir -p "$OUT"
+
+# Build the input_list line. Order matches decode-io.tsv IN rows.
+# net-run input_list format: "name1:=path1 name2:=path2 ..."
+LINE="inputs_embeds:=$STEP_DIR/inputs_embeds.raw"
+LINE="$LINE per_layer_inputs:=$STEP_DIR/per_layer_inputs.raw"
+LINE="$LINE position_ids:=$STEP_DIR/position_ids.raw"
+LINE="$LINE cache_position:=$STEP_DIR/cache_position.raw"
+LINE="$LINE full_mask:=$STEP_DIR/full_mask.raw"
+LINE="$LINE sliding_mask:=$STEP_DIR/sliding_mask.raw"
+i=0
+while [ $i -lt 15 ]; do
+  LINE="$LINE past_k_$i:=$KV/past_k_$i.raw past_v_$i:=$KV/past_v_$i.raw"
+  i=$((i+1))
+done
+echo "$LINE" > "$STEP_DIR/in.txt"
+
+"$BASE/bin/qnn-net-run" \
+  --backend "$BASE/lib/libQnnHtp.so" \
+  --retrieve_context "$BASE/artifacts/gemma4_decode_wgqa_int8kv_a16w8_v79.bin" \
+  --input_list "$STEP_DIR/in.txt" \
+  --output_dir "$OUT" \
+  --use_native_input_files \
+  --use_native_output_files \
+  --log_level error
+
+# net-run writes outputs under $OUT/Result_0/
+RES=$OUT/Result_0
+# Rotate present_* -> past_* for the next step (on-device, no transfer).
+i=0
+while [ $i -lt 15 ]; do
+  mv -f "$RES/present_k_$i.raw" "$KV/past_k_$i.raw"
+  mv -f "$RES/present_v_$i.raw" "$KV/past_v_$i.raw"
+  i=$((i+1))
+done
+echo "STEP_OK"

runtime/gate_ondevice_trunk.sh

@@ -0,0 +1,38 @@
+#!/system/bin/sh
+# On-device single PREFILL pass via qnn-net-run against the A16W8 trunk context binary.
+#
+# Unlike the decode graph this has NO KV in or out: one forward over a fixed SEQ=128 window
+# produces the full [1,128,1536] hidden. So verifying it is one net-run per prompt, not a loop.
+#
+# Layout on device (BASE=/data/local/tmp/gemma):
+#   $BASE/bin/qnn-net-run, $BASE/lib/*.so, $BASE/dsp/*.so
+#   $BASE/artifacts/gemma4_trunk_a16w8_v79.bin
+#   $BASE/tstep/<per-prompt input .raw files>
+#   $BASE/tout/                      (net-run output dir -> hidden.raw)
+set -e
+BASE=/data/local/tmp/gemma
+export LD_LIBRARY_PATH=$BASE/lib:/system/lib64:/vendor/lib64
+export ADSP_LIBRARY_PATH="$BASE/dsp;/vendor/dsp/cdsp;/vendor/lib/rfsa/cdsp;/vendor/dsp"
+export LD_PRELOAD=/system/lib64/libbinder.so   # Android 16 linker fix
+
+STEP=$BASE/tstep
+OUT=$BASE/tout
+rm -rf "$OUT"; mkdir -p "$OUT"
+
+LINE="inputs_embeds:=$STEP/inputs_embeds.raw"
+LINE="$LINE per_layer_inputs:=$STEP/per_layer_inputs.raw"
+LINE="$LINE position_ids:=$STEP/position_ids.raw"
+LINE="$LINE full_mask:=$STEP/full_mask.raw"
+LINE="$LINE sliding_mask:=$STEP/sliding_mask.raw"
+echo "$LINE" > "$STEP/in.txt"
+
+"$BASE/bin/qnn-net-run" \
+  --backend "$BASE/lib/libQnnHtp.so" \
+  --retrieve_context "$BASE/artifacts/gemma4_trunk_a16w8_v79.bin" \
+  --input_list "$STEP/in.txt" \
+  --output_dir "$OUT" \
+  --use_native_input_files \
+  --use_native_output_files \
+  --log_level error
+
+echo "TRUNK_OK"

runtime/gate_ondevice_wgqa.sh

@@ -0,0 +1,58 @@
+#!/system/bin/sh
+# On-device single-step decode via qnn-net-run against the prebuilt A16W8 v79 context binary.
+# Runs entirely on the device: KV present->past rotation is local file mv (no host transfer).
+# Host pushes only the tiny per-step inputs (inputs_embeds, per_layer_inputs, position_ids,
+# cache_position, full_mask, sliding_mask) and pulls only hidden.raw.
+#
+# Layout on device (BASE=/data/local/tmp/gemma):
+#   $BASE/bin/qnn-net-run, $BASE/lib/*.so, $BASE/dsp/*.so
+#   $BASE/artifacts/gemma4_decode_a16w8_v79.bin
+#   $BASE/kv/past_{k,v}_<i>.raw     (persist across steps; seeded to zeros at step 0)
+#   $BASE/step/<small per-step input .raw files>
+#   $BASE/step/in.txt               (input_list: one line, space-separated name:=path)
+#   $BASE/out/                      (net-run output dir -> hidden.raw + present_*.raw)
+set -e
+BASE=/data/local/tmp/gemma
+export LD_LIBRARY_PATH=$BASE/lib:/system/lib64:/vendor/lib64
+export ADSP_LIBRARY_PATH="$BASE/dsp;/vendor/dsp/cdsp;/vendor/lib/rfsa/cdsp;/vendor/dsp"
+export LD_PRELOAD=/system/lib64/libbinder.so   # Android 16 linker fix (from qdc-session-notes)
+
+STEP_DIR=$BASE/step
+KV=$BASE/kv
+OUT=$BASE/out
+rm -rf "$OUT"; mkdir -p "$OUT"
+
+# Build the input_list line. Order matches decode-io.tsv IN rows.
+# net-run input_list format: "name1:=path1 name2:=path2 ..."
+LINE="inputs_embeds:=$STEP_DIR/inputs_embeds.raw"
+LINE="$LINE per_layer_inputs:=$STEP_DIR/per_layer_inputs.raw"
+LINE="$LINE position_ids:=$STEP_DIR/position_ids.raw"
+LINE="$LINE cache_position:=$STEP_DIR/cache_position.raw"
+LINE="$LINE full_mask:=$STEP_DIR/full_mask.raw"
+LINE="$LINE sliding_mask:=$STEP_DIR/sliding_mask.raw"
+i=0
+while [ $i -lt 15 ]; do
+  LINE="$LINE past_k_$i:=$KV/past_k_$i.raw past_v_$i:=$KV/past_v_$i.raw"
+  i=$((i+1))
+done
+echo "$LINE" > "$STEP_DIR/in.txt"
+
+"$BASE/bin/qnn-net-run" \
+  --backend "$BASE/lib/libQnnHtp.so" \
+  --retrieve_context "$BASE/artifacts/gemma4_decode_wgqa_a16w8_v79.bin" \
+  --input_list "$STEP_DIR/in.txt" \
+  --output_dir "$OUT" \
+  --use_native_input_files \
+  --use_native_output_files \
+  --log_level error
+
+# net-run writes outputs under $OUT/Result_0/
+RES=$OUT/Result_0
+# Rotate present_* -> past_* for the next step (on-device, no transfer).
+i=0
+while [ $i -lt 15 ]; do
+  mv -f "$RES/present_k_$i.raw" "$KV/past_k_$i.raw"
+  mv -f "$RES/present_v_$i.raw" "$KV/past_v_$i.raw"
+  i=$((i+1))
+done
+echo "STEP_OK"

runtime/hostlib.py

@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+"""Host-side pieces of the Gemma-4-E2B A16W8 v79 runtime, matched to the gemma3n architecture.
+
+Graph boundary (from decode_fixed.py / host_generate.py):
+  host: token id -> inputs_embeds (embed_tokens[id] * sqrt(H))
+                 -> per_layer_inputs (embed_tokens_per_layer[id].reshape(NL,PLD) * sqrt(PLD))
+  NPU decode graph: takes inputs_embeds, per_layer_inputs, position_ids(int32), cache_position(int32),
+                    full_mask, sliding_mask, 15x past_k/v  ->  hidden (final-normed), 15x present_k/v
+  host: hidden -> logits = hidden @ embed_tokens.T  (tied, UNSCALED)
+              -> softcap: 30*tanh(logits/30) -> argmax
+
+Embedding scales are read straight from the gemma3n source:
+  embed_tokens          embed_scale = hidden_size ** 0.5              (H=1536 -> ~39.1918)
+  embed_tokens_per_layer embed_scale = hidden_size_per_layer ** 0.5   (PLD=256 -> 16.0)
+lm_head is tied to embed_tokens.weight and applied WITHOUT the embed scale.
+"""
+import json, pathlib, numpy as np
+
+HERE = pathlib.Path(__file__).resolve().parent
+HM = HERE.parent / "host-model"
+
+# dims
+H = 1536
+PLD = 256
+NL = 35
+CTX = 4096
+VOCAB = 262144
+SOFTCAP = 30.0
+EMB_SCALE = float(np.sqrt(H))          # 39.19183...
+PLE_SCALE = float(np.sqrt(PLD))        # 16.0
+# Finite mask value — MUST match the value used at quantization calibration
+# (decode_pipeline_v2.py NEG=-1e4). -inf/finfo.min cannot survive int16 activation
+# quantization (blows out the range so real scores round to 0); -1e4 still zeroes
+# softmax (exp(-1e4)=0) while leaving real scores (~+-50) well resolved.
+NEG = -1e4
+
+# Gemma-4 chat-template token ids (verified against transformers apply_chat_template).
+BOS_ID = 2          # <bos>
+TURN_START = 105    # <|turn>
+TURN_END = 106      # <turn|>   -- also the generation stop token
+NL_ID = 107         # '\n'
+ROLE_USER = 2364    # 'user'
+ROLE_MODEL = 4368   # 'model'
+STOP_IDS = {TURN_END, 1}   # <turn|> or <eos>
+
+# KV layout: head dim 512 for layers 4,9,14; else 256 (from decode-io.tsv, 15 non-shared layers)
+KV_HD = [256]*4 + [512] + [256]*4 + [512] + [256]*4 + [512]
+NC = 15
+
+# --- WGQA (windowed + broadcast-GQA) decode graph ---------------------------------------
+# Sliding-window layers keep a WIN-entry RING buffer instead of a full CTX one; only the
+# 3 full-attention layers (4, 9, 14) keep CTX. The ring write index is computed INSIDE the
+# graph (cache_position % buf), so the host still just passes pos. This cuts KV traffic from
+# ~288MB/step to ~63MB/step and, with the GQA `expand` removed, is 4.4x faster on v79
+# (307.9ms -> 69.8ms/step).
+WIN = 512
+KV_BUF = [WIN]*4 + [CTX] + [WIN]*4 + [CTX] + [WIN]*4 + [CTX]
+
+
+def _load_bf16(path, shape):
+    raw = np.fromfile(path, dtype=np.uint16)
+    f32 = (raw.astype(np.uint32) << 16).view(np.float32)
+    return f32.reshape(shape)
+
+
+def _bf16_row(mm, idx):
+    """Convert one bf16 row (uint16 memmap slice) -> float32."""
+    return (mm[idx].astype(np.uint32) << 16).view(np.float32)
+
+
+class HostModel:
+    def __init__(self):
+        # memmap as uint16 so we never materialize the full float32 tensors (~11GB spike).
+        self.embed = np.memmap(HM / "embed_tokens_weight.bf16", dtype=np.uint16,
+                               mode="r", shape=(VOCAB, H))            # [V,H] bf16
+        self.ple = np.memmap(HM / "embed_tokens_per_layer_weight.bf16", dtype=np.uint16,
+                             mode="r", shape=(VOCAB, NL*PLD))         # [V, NL*PLD] bf16
+        # tokenizer
+        from tokenizers import Tokenizer
+        self.tok = Tokenizer.from_file(str(HM / "tokenizer.json"))
+
+    # ---- tokenization ----
+    def encode(self, text):
+        return self.tok.encode(text).ids
+
+    def encode_chat(self, user_text):
+        """Gemma-4 canonical chat template, built from raw token ids.
+
+        The `tokenizers` library has no chat-template support, so we assemble the same
+        id sequence transformers' apply_chat_template() produces. Verified byte-exact
+        against transformers 5.12 for google/gemma-4-E2B-it:
+          '<bos><|turn>user\\nTHE PROMPT<turn|>\\n<|turn>model\\n'
+        Without this the instruct model degenerates ('France is France is ...'); with it
+        plain greedy decoding is coherent.
+        """
+        return ([BOS_ID, TURN_START, ROLE_USER, NL_ID]
+                + self.encode(user_text)
+                + [TURN_END, NL_ID, TURN_START, ROLE_MODEL, NL_ID])
+
+    def decode(self, ids):
+        return self.tok.decode(ids)
+
+    # ---- host embeddings for one token ----
+    def embeds(self, token_id):
+        ie = (_bf16_row(self.embed, token_id) * EMB_SCALE).reshape(1, 1, H)
+        ple = (_bf16_row(self.ple, token_id) * PLE_SCALE).reshape(1, 1, NL, PLD)
+        return ie.astype(np.float32), ple.astype(np.float32)
+
+    # ---- masks (additive [1,1,1,CTX]) ----
+    def masks(self, pos):
+        j = np.arange(CTX)
+        full = np.where(j <= pos, 0.0, NEG).astype(np.float32).reshape(1, 1, 1, CTX)
+        slide = np.where((j <= pos) & (j > pos - 512), 0.0, NEG).astype(np.float32).reshape(1, 1, 1, CTX)
+        return full, slide
+
+    def masks_wgqa(self, pos):
+        """Masks for the WGQA graph: full stays [1,1,1,CTX], but the sliding mask is only
+        [1,1,1,WIN] because those layers attend over a WIN-entry ring buffer. Once pos has
+        filled the ring (pos >= WIN-1) every slot is valid, so the mask is all zeros."""
+        jf = np.arange(CTX)
+        full = np.where(jf <= pos, 0.0, NEG).astype(np.float32).reshape(1, 1, 1, CTX)
+        js = np.arange(WIN)
+        slide = (np.zeros(WIN, np.float32) if pos >= WIN - 1
+                 else np.where(js <= pos, 0.0, NEG).astype(np.float32)).reshape(1, 1, 1, WIN)
+        return full, slide
+
+    # ---- lm head (tied, unscaled) + softcap ----
+    def _embed_f32(self):
+        # Lazily materialize the tied word-embedding as float32 [V,H] for lm_head (~1.6GB).
+        if getattr(self, "_ef32", None) is None:
+            self._ef32 = (self.embed.astype(np.uint32) << 16).view(np.float32)
+        return self._ef32
+
+    def logits(self, hidden):
+        h = np.asarray(hidden, np.float32).reshape(H)
+        lg = self._embed_f32() @ h                       # [V,H] @ [H] -> [V]
+        lg = SOFTCAP * np.tanh(lg / SOFTCAP)
+        return lg
+
+    def argmax_next(self, hidden):
+        return int(self.logits(hidden).argmax())
+
+
+if __name__ == "__main__":
+    # smoke: load + embed a couple tokens, print shapes/norms
+    m = HostModel()
+    ids = m.encode("The capital of France is")
+    print("prompt ids:", ids, "->", repr(m.decode(ids)))
+    ie, ple = m.embeds(ids[0])
+    print("inputs_embeds", ie.shape, "norm", float(np.linalg.norm(ie)))
+    print("per_layer_inputs", ple.shape, "norm", float(np.linalg.norm(ple)))
+    f, s = m.masks(3)
+    print("full_mask nonneg count", int((f == 0).sum()), "sliding", int((s == 0).sum()))

runtime/run_gate.py

@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+"""Host orchestrator for the A16W8 v79 correctness gate.
+
+Runs the autoregressive decode loop by driving qnn-net-run on-device once per token.
+KV (~288MB) stays resident on device as files; only tiny per-step tensors cross adb.
+
+Prereqs on device (staged by push_gate.sh):
+  /data/local/tmp/gemma/{bin,lib,dsp,artifacts,step,kv,out}
+Host math from hostlib.py (gemma3n scaling, tied softcapped lm_head).
+
+Usage:
+  python run_gate.py --prompt "The capital of France is" --ntokens 15 [--adb-serial X]
+  Optionally --hf-check to compare against HF greedy on host (needs full model; heavy).
+"""
+import argparse, os, subprocess, sys, time, pathlib, numpy as np
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+import hostlib
+
+BASE = "/data/local/tmp/gemma"
+STEP = f"{BASE}/step"
+KV = f"{BASE}/kv"
+OUTR = f"{BASE}/out/Result_0"
+LOCAL_STEP = pathlib.Path("/tmp/gemma_step")
+CTX = hostlib.CTX
+H = hostlib.H
+KV_HD = hostlib.KV_HD
+
+def adb(args, serial=None, **kw):
+    cmd = ["adb"] + (["-s", serial] if serial else []) + args
+    return subprocess.run(cmd, capture_output=True, text=True, **kw)
+
+def adb_shell(script, serial=None, timeout=600):
+    return adb(["shell", script], serial=serial, timeout=timeout)
+
+def push(local, remote, serial=None):
+    r = adb(["push", str(local), remote], serial=serial, timeout=300)
+    if r.returncode != 0:
+        raise RuntimeError(f"push failed {local}->{remote}: {r.stderr}")
+
+def pull(remote, local, serial=None):
+    r = adb(["pull", remote, str(local)], serial=serial, timeout=300)
+    if r.returncode != 0:
+        raise RuntimeError(f"pull failed {remote}->{local}: {r.stderr}")
+
+def seed_kv(serial, wgqa=False):
+    """Zero the 15 past_k/v buffers on device.
+
+    Created ON DEVICE with dd rather than pushed: they are ~144MB of zeros, and pushing
+    them over the QDC tunnel took minutes and was the flakiest part of the run (a reset
+    mid-seed leaves a partial buffer and silently corrupts the whole generation).
+    WGQA uses a WIN-entry ring for sliding layers, so buffer depth is per-layer.
+    """
+    LOCAL_STEP.mkdir(exist_ok=True)
+    adb_shell(f"mkdir -p {KV} {STEP} {BASE}/out", serial=serial)
+    depths = hostlib.KV_BUF if wgqa else [CTX] * hostlib.NC
+    cmds = [f"rm -f {KV}/*.raw"]
+    for i in range(hostlib.NC):
+        nbytes = depths[i] * KV_HD[i] * 4          # [1,1,depth,hd] float32
+        for kind in ("k", "v"):
+            cmds.append(f"dd if=/dev/zero of={KV}/past_{kind}_{i}.raw bs=4096 "
+                        f"count={nbytes // 4096} 2>/dev/null")
+    r = adb_shell(" && ".join(cmds), serial=serial, timeout=600)
+    if r.returncode != 0:
+        raise RuntimeError(f"on-device KV seed failed: {r.stderr}")
+
+def write_step_inputs(m, token_id, pos, serial, wgqa=False):
+    ie, ple = m.embeds(token_id)
+    full, slide = m.masks_wgqa(pos) if wgqa else m.masks(pos)
+    files = {
+        "inputs_embeds": ie.astype(np.float32),
+        "per_layer_inputs": ple.astype(np.float32),
+        "position_ids": np.array([[pos]], np.int32),
+        "cache_position": np.array([pos], np.int32),
+        "full_mask": full.astype(np.float32),
+        "sliding_mask": slide.astype(np.float32),
+    }
+    for name, arr in files.items():
+        p = LOCAL_STEP / f"{name}.raw"
+        arr.tofile(p)
+        push(p, f"{STEP}/{name}.raw", serial=serial)
+
+def run_step(serial, script="gate_ondevice.sh"):
+    r = adb_shell(f"sh {BASE}/{script}", serial=serial, timeout=600)
+    if "STEP_OK" not in r.stdout:
+        raise RuntimeError(f"net-run step failed:\nSTDOUT:{r.stdout}\nSTDERR:{r.stderr}")
+    return r
+
+def fetch_hidden(serial):
+    pull(f"{OUTR}/hidden.raw", LOCAL_STEP / "hidden.raw", serial=serial)
+    return np.fromfile(LOCAL_STEP / "hidden.raw", dtype=np.float32).reshape(H)
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--prompt", default="The capital of France is")
+    ap.add_argument("--ntokens", type=int, default=15)
+    ap.add_argument("--adb-serial", default=os.environ.get("ADB_SERIAL"))
+    ap.add_argument("--hf-check", action="store_true")
+    ap.add_argument("--wgqa", action="store_true",
+                    help="target the windowed + broadcast-GQA decode graph (512-entry ring "
+                         "buffers on sliding layers, 512-wide sliding mask)")
+    ap.add_argument("--script", default=None,
+                    help="override the on-device step script (e.g. gate_ondevice_int8kv.sh)")
+    ap.add_argument("--chat", action="store_true",
+                    help="wrap the prompt in the Gemma-4 chat template (required for coherent "
+                         "output from the -it model; raw completion format degenerates)")
+    args = ap.parse_args()
+
+    print("loading host model (embeddings + tokenizer)...", flush=True)
+    m = hostlib.HostModel()
+    ids = m.encode_chat(args.prompt) if args.chat else m.encode(args.prompt)
+    print(f"prompt: {args.prompt!r}  (chat_template={args.chat})\nids: {ids}", flush=True)
+
+    print("seeding KV buffers on device...", flush=True)
+    seed_kv(args.adb_serial, wgqa=args.wgqa)
+
+    seq = ids[:]
+    gen_ids = []
+    pos = 0
+    t_steps = []
+    # prefill+decode: feed prompt tokens one-by-one (pos advances), then greedy-generate
+    total = len(seq) + args.ntokens
+    nxt = None
+    for step in range(total):
+        t = seq[step] if step < len(seq) else nxt
+        write_step_inputs(m, t, pos, args.adb_serial, wgqa=args.wgqa)
+        t0 = time.time()
+        run_step(args.adb_serial, (args.script or ("gate_ondevice_wgqa.sh" if args.wgqa else "gate_ondevice.sh")))
+        dt = time.time() - t0
+        t_steps.append(dt)
+        hidden = fetch_hidden(args.adb_serial)
+        pos += 1
+        if step >= len(seq) - 1:  # last prompt token onward -> predict next
+            nxt = m.argmax_next(hidden)
+            gen_ids.append(nxt)
+            print(f"  step {step:2d} pos {pos-1:2d}  {dt*1000:7.1f}ms  -> id {nxt:6d} {m.decode([nxt])!r}", flush=True)
+            if nxt in hostlib.STOP_IDS:
+                print("  (stop token reached)", flush=True)
+                break
+
+    text = m.decode([t for t in gen_ids if t not in hostlib.STOP_IDS])
+    print("\n=== GENERATION ===")
+    print("continuation:", repr(text))
+    print(f"per-step wall (incl adb+netrun init): mean {1000*np.mean(t_steps):.0f}ms  min {1000*min(t_steps):.0f}ms")
+    print("NOTE: this wall time is NOT throughput (net-run reloads context each step). Coherence/accuracy only.")
+
+    if args.hf_check:
+        hf_compare(m, ids, gen_ids)
+
+def hf_compare(m, ids, gen_ids):
+    print("\n=== HF greedy reference (host, CPU) ===", flush=True)
+    import torch
+    from transformers import AutoModelForCausalLM
+    tok = (pathlib.Path.home() / ".cache/huggingface/token").read_text().strip()
+    mdl = AutoModelForCausalLM.from_pretrained("google/gemma-4-E2B-it", token=tok,
+                                               torch_dtype=torch.float32, device_map="cpu").eval()
+    with torch.no_grad():
+        out = mdl.generate(torch.tensor([ids]), max_new_tokens=len(gen_ids), do_sample=False)
+    hf = out[0][len(ids):].tolist()
+    print("HF  :", repr(m.decode(hf)))
+    print("NPU :", repr(m.decode(gen_ids)))
+    match = sum(a == b for a, b in zip(hf, gen_ids))
+    print(f"token match: {match}/{len(gen_ids)}")
+
+if __name__ == "__main__":
+    main()

runtime/stage_device.sh

@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+# Stage this repo onto an adb-attached Snapdragon device.
+#
+# The QNN runtime (qnn-net-run + libQnnHtp*.so + the HTP skel/stub for your hexagon version)
+# is NOT in this repo -- it ships with Qualcomm's AI Engine Direct (QAIRT) SDK and is not
+# redistributable. Point QAIRT_DIR at your SDK install, or pre-stage those files yourself.
+#
+#   QAIRT_DIR=/path/to/qairt/2.45.0.xxxxxx ./stage_device.sh <serial> [v79|v81]
+#
+# Note the KV buffers are NOT pushed: run_gate.py creates them on device with dd, because
+# pushing ~144 MB of zeros over a slow adb link was the flakiest part of the pipeline.
+set -euo pipefail
+
+SERIAL="${1:?usage: stage_device.sh <adb-serial> [v79|v81]}"
+HTP="${2:-v79}"
+REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+BASE=/data/local/tmp/gemma
+ADB=(adb -s "$SERIAL")
+
+case "$HTP" in
+  v79) SOC_ARCH=v79 ;;
+  v81) SOC_ARCH=v81 ;;
+  *) echo "unknown HTP '$HTP' (expected v79 or v81)" >&2; exit 1 ;;
+esac
+
+echo ">> device: $("${ADB[@]}" shell getprop ro.soc.model | tr -d '\r') (staging for HTP $SOC_ARCH)"
+"${ADB[@]}" shell "mkdir -p $BASE/bin $BASE/lib $BASE/dsp $BASE/artifacts $BASE/step $BASE/kv $BASE/out $BASE/tstep $BASE/tout"
+
+# ---- QNN runtime (from your QAIRT SDK) ----------------------------------------------
+if [ -n "${QAIRT_DIR:-}" ]; then
+  echo ">> pushing QNN runtime from QAIRT_DIR=$QAIRT_DIR"
+  AA="$QAIRT_DIR/lib/aarch64-android"
+  HX="$QAIRT_DIR/lib/hexagon-$SOC_ARCH/unsigned"
+  [ -d "$AA" ] || { echo "missing $AA" >&2; exit 1; }
+  [ -d "$HX" ] || { echo "missing $HX -- your SDK may not include hexagon-$SOC_ARCH" >&2; exit 1; }
+  "${ADB[@]}" push "$QAIRT_DIR/bin/aarch64-android/qnn-net-run" "$BASE/bin/" >/dev/null
+  "${ADB[@]}" shell "chmod 755 $BASE/bin/qnn-net-run"
+  for f in libQnnHtp.so libQnnSystem.so libQnnHtpPrepare.so libQnnHtpNetRunExtensions.so \
+           "libQnnHtp${SOC_ARCH^^}Stub.so"; do
+    [ -f "$AA/$f" ] && "${ADB[@]}" push "$AA/$f" "$BASE/lib/" >/dev/null
+  done
+  for f in "$HX"/libQnnHtp*.so; do "${ADB[@]}" push "$f" "$BASE/dsp/" >/dev/null; done
+else
+  echo ">> QAIRT_DIR not set -- skipping QNN runtime."
+  echo "   You must stage these yourself under $BASE:"
+  echo "     bin/qnn-net-run"
+  echo "     lib/libQnnHtp.so libQnnSystem.so libQnnHtpPrepare.so libQnnHtpNetRunExtensions.so libQnnHtp${SOC_ARCH^^}Stub.so"
+  echo "     dsp/libQnnHtp${SOC_ARCH^^}.so libQnnHtp${SOC_ARCH^^}Skel.so"
+fi
+
+# ---- context binaries ---------------------------------------------------------------
+echo ">> pushing context binaries (~1.9 GB each, be patient)"
+for b in "$REPO"/*"_$HTP.bin"; do
+  [ -f "$b" ] || continue
+  echo "   $(basename "$b")"
+  "${ADB[@]}" push "$b" "$BASE/artifacts/" >/dev/null
+done
+
+# ---- on-device step scripts ---------------------------------------------------------
+for s in gate_ondevice_wgqa.sh gate_ondevice_int8kv.sh gate_ondevice_trunk.sh; do
+  "${ADB[@]}" push "$REPO/runtime/$s" "$BASE/" >/dev/null
+  "${ADB[@]}" shell "chmod 755 $BASE/$s"
+done
+
+echo ">> staged:"
+"${ADB[@]}" shell "ls -la $BASE/artifacts $BASE/bin $BASE/lib $BASE/dsp"
+echo ">> OK"

runtime/verify_trunk.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Verify the A16W8 trunk (prefill) context binary on real hardware.
+
+The trunk is stateless: one forward per prompt, no KV. So this pushes the per-prompt inputs,
+runs qnn-net-run once, pulls the hidden state, and compares against the float reference
+(trunk_ref.npz, produced on the AWS box by trunk_ref.py) on:
+  - cosine of the last-real-position hidden vs float
+  - the argmax next token through the tied softcapped lm_head
+
+Usage:
+  python verify_trunk.py --adb-serial <serial> --ref /path/to/trunk_ref.npz
+"""
+import argparse, pathlib, subprocess, sys, numpy as np
+
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+import hostlib
+
+BASE = "/data/local/tmp/gemma"
+STEP = f"{BASE}/tstep"
+OUTR = f"{BASE}/tout/Result_0"
+SEQ = 128
+NEG = hostlib.NEG
+L_ = pathlib.Path("/tmp/gemma_trunk"); L_.mkdir(exist_ok=True)
+
+
+def adb(a, serial, **kw):
+    return subprocess.run(["adb"] + (["-s", serial] if serial else []) + a,
+                          capture_output=True, text=True, **kw)
+
+
+def cmask(L):
+    j = np.arange(SEQ)[None, :]
+    i = np.arange(SEQ)[:, None]
+    return np.where((j <= i) & (j < L), 0.0, NEG).astype(np.float32).reshape(1, 1, SEQ, SEQ)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--adb-serial", required=True)
+    ap.add_argument("--ref", default="/home/azek/.claude/jobs/db9a6094/tmp/trunk_ref.npz")
+    args = ap.parse_args()
+
+    ref = np.load(args.ref)
+    n = int(ref["n"][0])
+    m = hostlib.HostModel()
+    adb(["shell", f"mkdir -p {STEP} {BASE}/tout"], args.adb_serial)
+
+    ok = 0
+    coss = []
+    for k in range(n):
+        ids = [int(x) for x in ref[f"ids_{k}"]]
+        L = len(ids)
+        hf = ref[f"h_{k}"]
+        tf = int(ref[f"tok_{k}"][0])
+
+        # host embeddings for the whole padded window
+        padded = ids + [0] * (SEQ - L)
+        ie = np.concatenate([m.embeds(t)[0].reshape(1, 1, hostlib.H) for t in padded], axis=1)
+        ple = np.concatenate([m.embeds(t)[1].reshape(1, 1, hostlib.NL, hostlib.PLD) for t in padded], axis=1)
+        mk = cmask(L)
+        files = {
+            "inputs_embeds": ie.astype(np.float32),
+            "per_layer_inputs": ple.astype(np.float32),
+            "position_ids": np.arange(SEQ, dtype=np.int32).reshape(1, SEQ),
+            "full_mask": mk,
+            "sliding_mask": mk.copy(),
+        }
+        for name, arr in files.items():
+            p = L_ / f"{name}.raw"
+            arr.tofile(p)
+            r = adb(["push", str(p), f"{STEP}/{name}.raw"], args.adb_serial, timeout=300)
+            if r.returncode != 0:
+                raise RuntimeError(f"push {name}: {r.stderr}")
+
+        r = adb(["shell", f"sh {BASE}/gate_ondevice_trunk.sh"], args.adb_serial, timeout=600)
+        if "TRUNK_OK" not in r.stdout:
+            raise RuntimeError(f"trunk net-run failed:\n{r.stdout}\n{r.stderr}")
+        adb(["pull", f"{OUTR}/hidden.raw", str(L_ / "hidden.raw")], args.adb_serial, timeout=300)
+
+        hd = np.fromfile(L_ / "hidden.raw", np.float32).reshape(SEQ, hostlib.H)[L - 1]
+        cos = float(hd @ hf / (np.linalg.norm(hd) * np.linalg.norm(hf) + 1e-9))
+        coss.append(cos)
+        td = int(m.logits(hd).argmax())
+        hit = (td == tf)
+        ok += hit
+        print(f"  [{k:2d}] cos={cos:.5f} |hw|={np.linalg.norm(hd):7.2f} |float|={np.linalg.norm(hf):7.2f} "
+              f"hw={td:6d} {m.decode([td])!r:16s} float={tf:6d} {m.decode([tf])!r:16s} "
+              f"{'OK' if hit else 'MISMATCH'}", flush=True)
+
+    print(f"\n  hidden cos: mean={np.mean(coss):.5f} min={np.min(coss):.5f}", flush=True)
+    print(f"  NEXT-TOKEN TOP-1 vs float on HARDWARE: {ok}/{n} ({100.0*ok/n:.1f}%)", flush=True)
+    print("TRUNK_VERIFY_DONE", flush=True)
+
+
+if __name__ == "__main__":
+    main()