docs: descriptive model card

README.md

@@ -1,64 +1,255 @@
 ---
+license: gemma
+license_link: https://ai.google.dev/gemma/terms
+base_model: google/gemma-4-E2B-it-qat-q4_0-unquantized
 library_name: coreai
-tags: [core-ai, aimodel, on-device, apple]
+pipeline_tag: text-generation
+tags:
+  - core-ai
+  - aimodel
+  - apple-silicon
+  - on-device
+  - quantized
+  - int4
+  - qat
+  - gemma4
 ---
 
-# gemma-4-E2B-CoreAI
+# Gemma 4 E2B — Core AI (.aimodel)
 
-Core AI `.aimodel` bundles converted for [Visible](https://visible.cx) on-device
-inference. **Derivative artifacts** — converted from the upstream checkpoints listed
-below; all original model licences apply and travel with these weights.
+`google/gemma-4-E2B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for
+Apple silicon by the [Visible](https://visible.cx) project. **Derivative artifacts**:
+Google's QAT-trained weights rounded onto the int4 grid they were trained for and
+re-expressed as a Core AI graph.
 
-| Bundle | Source checkpoint | Context | Functions | Producer | Created |
-|---|---|---|---|---|---|
-| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl` | google/gemma-4-E2B-it-qat-q4_0-unquantized | 4096 | main | coreai-core 1.0.0b2 | 20260817T104931Z |
-| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | google/gemma-4-E2B-it-qat-q4_0-unquantized | 4096 | main+prefill | coreai-core 1.0.0b2 | 20260817T110657Z |
+Gemma 4 E2B uses **Per-Layer Embeddings**, so these bundles take a large embedding gather
+table as a *static graph input* rather than carrying it in the graph. That table ships here
+in `ios-frontend/` and **the bundles do not load without it** — a missing table produces a
+bare input-arity error that says nothing about tables.
 
-## Provenance
-
-- Converted with the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo)
-  conversion recipes on the pinned `apple/coreai-models` base
-  `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the zoo python overlay.
-- Toolchain: `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0`.
-- Every bundle's inner `<name>.aimodel/metadata.json` reports
-  `producer: "coreai-core 1.0.0b2"`.
-- Gemma bundles: `eos_token` is set to `<turn|>` (Gemma 4 ends a *turn* with `<turn|>`;
-  stopping on the source's `<eos>` overruns the reply).
-- Bundles carry `runtime_env COREAI_CHUNK_THRESHOLD=1`.
-
-Upstream sources: `google/gemma-4-E2B-it-qat-q4_0-unquantized`
-
-
-## High-context bundles — ctx 8192
-
-Same weights, same graph, same recipe as the 4096 bundles above — the exporter's
-`--max-ctx` only widens the dynamic sequence bound and the manifest's
-`language.max_context_length`. KV cache stays fp16.
-
-| Bundle | Source checkpoint | Context | Functions | KV dtype | Producer | Created |
-|---|---|---|---|---|---|---|
-| `ctx8192/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | google/gemma-4-E2B-it-qat-q4_0-unquantized | 8192 | main+prefill | fp16 | coreai-core 1.0.0b2 | 20260817T203916Z |
+Of the Gemma 4 bundles in this catalog, this is **the only one worth considering for
+structured-output work**: 10/10 schema-clean at 29 tok/s on a 16 GB Mac. It is still 4× the
+cost per row of the LFM2.5 1.2B at 3× the peak memory, so it earns a slot only where
+Gemma's output specifically is wanted.
 
+## Provenance
 
-## High-context bundles — ctx 16384
-
-Same weights, same graph, same recipe as the 4096 bundles above — the exporter's
-`--max-ctx` only widens the dynamic sequence bound and the manifest's
-`language.max_context_length`. KV cache stays fp16.
-
-| Bundle | Source checkpoint | Context | Functions | KV dtype | Producer | Created |
-|---|---|---|---|---|---|---|
-| `ctx16384/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | google/gemma-4-E2B-it-qat-q4_0-unquantized | 16384 | main+prefill | fp16 | coreai-core 1.0.0b2 | 20260817T204052Z |
-
-
-## w4a8/ — int4 weights + INT8 activations (EXPERIMENTAL)
-
-> **EXPERIMENTAL — not qualified.** These bundles have not passed a Mac-side
-> oracle/parity gate or a device benchmark. Do not route production traffic to
-> them until they have.
+| | |
+|---|---|
+| Base checkpoint | `google/gemma-4-E2B-it-qat-q4_0-unquantized` (ungated) |
+| Recipe (decode) | `export_gemma4_decode_pipelined.py int4lin --tbl` — zoo recipe `gemma-4-e2b`, `status = "verified"` |
+| Recipe (pf64) | `export_gemma4_pf_pipelined.py --pf 64` with `--tbl` and `--raw-dir` pointed at the gather table below |
+| Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay |
+| Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` |
+| Producer fingerprint | `coreai-core 1.0.0b2` on every inner asset `metadata.json` |
+| Weight format | **int4, per-block-32** (`int4lin`, symmetric-with-clipping) — the ggml q4_0 grid the QAT checkpoint was trained on |
+| Vocab | 262,144 |
+| Conversion | 180 s (decode) / 105 s, 26.8 GB peak RSS (pf64); Linux x86_64 |
+
+"QAT-unquantized" means QAT-*trained*, stored at full width; the int4 rounding happens at
+export, onto the grid training already targeted.
+
+`_tbl` = the PLE gather table is bound as a static graph input. `_pf64` = a second
+entrypoint, `prefill`, with a static query width of 64
+(`function_map: {"main": ["main", "prefill"]}`).
+
+## Contents
+
+### Bundles
+
+| Path | Bytes | Context | Functions | Status |
+|---|---:|---|---|---|
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,973 | 4096 | main + prefill | the working bundle |
+| `ctx8192/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,964 | 8192 | main + prefill | same weights, wider manifest |
+| `ctx16384/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,964 | 16384 | main + prefill | same weights, wider manifest |
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl` | 2,121,479,596 | 4096 | main | superseded — no `prefill`, see below |
+| `w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64` | 2,122,679,604 | 16384 | main + prefill | **EXPERIMENTAL** |
+
+Each bundle folder holds `<name>.aimodel/` (`main.mlirb` ≈ 2.09 GB, `main.hash`, asset
+`metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
+32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
+18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl`
+(35,045 B).
+
+**The three `_tbl_pf64` context folders are the same weights.** `--max-ctx` does not change
+the exported graph — identical function signatures, identical state descriptors, identical
+export peak RSS at every value. It changes exactly one thing:
+`language.max_context_length` in the bundle manifest. This was proved with a same-config
+control re-export whose `main.mlirb` differed from its twin by *more* bytes (22,053) than a
+4096→8192 pair does (22,344); the context delta is inside the conversion noise floor.
+Corroborated by the upload itself: pushing a "new" ~2 GB ctx8192 bundle into a repo that
+already held the 4096 bundle transferred a few hundred kB of new data. Pick the folder whose
+manifest integer matches the window you intend to run; a context bump is a manifest edit,
+not a conversion.
+
+### The PLE gather-table sidecar — required, not optional
+
+| Path | Files | Bytes |
+|---|---|---:|
+| `ios-frontend/gemma4_qat_gather_raw/` | 7 | 2,808,612,071 |
+
+| File | Bytes |
+|---|---:|
+| `embed_per_layer.i8` | 2,348,810,240 |
+| `embed_tokens.i8` | 402,653,184 |
+| `proj.f32` | 55,050,240 |
+| `embed_per_layer.scale.f32` | 1,048,576 |
+| `embed_tokens.scale.f32` | 1,048,576 |
+| `proj_norm.f32` | 1,024 |
+| `meta.json` | 231 |
+
+`meta.json` records the shape and the dequant convention:
+`V 262144, D 1536, PLD 8960, L 35, ld 256, embed_scale_pl 16.0` (= √256, which is what the
+exporter assumes).
+
+Every `_tbl` bundle binds `ple_table` (from `embed_per_layer.i8`, 2.35 GB) and `ple_scale`
+(from `embed_per_layer.scale.f32`) as **static** graph inputs. Without them the bundle fails
+to load with a bare arity error naming `ple_table`/`ple_scale` — a diagnosis this project
+got wrong once already. The sidecar is published here so the repo is self-contained for
+both loading and a pf64 rebuild.
+
+## Requirements
+
+- **Apple silicon Mac**, Core AI runtime.
+- **Engine contract: 4 inputs** — `input_ids`, `position_ids`, plus static `ple_table` and
+  `ple_scale`. Two engines can take that:
+  - **Pipelined engine** — binds the statics zero-copy over the caller's buffer, but
+    **does not support logits**, so no grammar-constrained decoding.
+  - **Sequential engine** — the only logits-capable engine, and therefore the only one
+    guided decoding can run on. It accepts `>= 2` inputs and binds everything beyond
+    `input_ids`/`position_ids` from `EngineOptions.staticInputBuffers`; an extra input with
+    no buffer is rejected by name. It pays a **one-time copy of every constant input at
+    load — 2.19 GB for E2B** — because its submission path otherwise materialises a foreign
+    buffer-backed view on *every* forward pass. That was measured: binding the table as a
+    raw view cost ~0.30 s per pass (3.4 tok/s, 64 s/row); copying once at load took the same
+    bundle to 8.6 s/row, a 7.5× improvement.
+- **States:** `keyCache` / `valueCache` `Float16, 15 × 1 × 1 × ? × 512`. Dynamic sequence
+  dim → `GrowingKVCache` (initial 256, doubling), not a static allocation at the manifest
+  maximum.
+- **KV cost: 30,720 bytes per token** (fp16) — 126 MB at 4096, 252 MB at 8192, 503 MB at
+  16384.
+- **Memory envelope, 16 GB Mac** (Metal `recommendedMaxWorkingSetSize` ≈ 10.7 GB):
+
+  | | |
+  |---|---|
+  | Weights (`main.mlirb`) | 2.09 GB |
+  | PLE table bound as static input | 2.35 GB |
+  | Base | **4.44 GB** |
+  | Affordable context, fp16 KV | ~204k tokens → **131,072, the model cap, binds first** |
+
+  KV is not the ceiling at this tier; **the PLE table is**, at 2.35 GB against a 503 MB KV
+  cost even at 16384. Measured peak footprint on a guided run: **8.9 GB**.
+- Sliding-window note: E2B interleaves 12 sliding and 3 full attention layers across 35
+  layers collapsed to 15 KV slots. The export models the window in the *mask*, not in
+  memory — sliding layers occupy full-length slots and zero-pad head_dim 256→512, so 80% of
+  KV bytes sit in windowed slots and 40% is dead padding. A ring-buffered sliding cache
+  would be ~5× cheaper per token; that is model authoring, not a flag, and it is not needed
+  at this tier.
+- `runtime_env COREAI_CHUNK_THRESHOLD=1` travels with the recipe. (Both engines now read the
+  static query width off the graph, so the older host-side rules about warmup query length
+  and chunk threshold are retired.)
+
+## Measured performance
+
+Measured on a **16 GB M2 Pro Mac (macOS 27 beta)** with the out-of-process `coreai-repro`
+harness: 10 real enrichment rows (5 POST, 5 COMMENT), the app's live `app_litert` JSON
+schema, greedy decoding through `respondJSON(to:schema:)`, `reset()` per row, **sequential
+engine with the PLE tables bound as static inputs**, 128-token cap.
+
+| | `_tbl_pf64` | `_tbl` (no `prefill` function) |
+|---|---|---|
+| Load | 14.6 s | 29.0 s |
+| Guided JSON parse | **10/10** | 3/3 |
+| Enum-clean | **10/10** | 3/3 |
+| s/row (POST) | **7.06** | 44.69 |
+| s/row (COMMENT) | **4.27** | — |
+| Decode | 21.8–33.5 tok/s, mean **28.6** | 7.7–16.8 tok/s |
+| TTFT | 0.6–4.7 s | 29–37 s |
+| Peak footprint | 8.9 GB | — |
+| Outcome | completed all ten rows | **died at row 4** |
+
+**`pf64` is worth 7.7× on time-to-first-token.** Head to head on the same 942-token prompt
+and the same weights, differing only in whether the S=64 `prefill` function was exported:
+TTFT 35.74 s → 4.65 s, s/row 45.04 → 8.83. Decode is untouched — the prefill function only
+serves full-width chunks. The non-`pf64` bundle is also not durable: its footprint ballooned
+past 37 GB across ~2,800 single-token passes and the campaign died partway through.
+
+**Enum conformance is the grammar's doing, and it is decisive.** Unguided, Gemma 4 invents
+an off-schema `motivation` value on essentially every row (`advocacy` for `advocate`,
+`informative` — a valid *tone* — used as a *motivation*). Guided, all ten are correct, not
+because the model improved but because an off-enum token is unsamplable. Decode throughput
+is essentially unchanged by the constraint; the ~25% extra cost of a guided row is the
+sequential engine's step-synchronous prefill, not the grammar.
+
+**Context for the cost.** On the same harness and machine, LFM2.5-1.2B runs 1.77/1.30 s/row
+at 3.06 GB peak, and Gemma 4 E4B runs 21.89/12.36 s/row at 10.7 GB peak. This bundle is
+about **4× the 1.2B's s/row for 3× its peak memory**. For reference, upstream measured E2B
+at 77.0/87.1 tok/s on an M4 Max, and an unguided pipelined run on this 16 GB Mac reached
+44.3 tok/s — the 28.6 tok/s above is the guided, sequential-engine figure.
+
+Nothing was measured above 4096 tokens. Quality at depth is a genuine open question: beyond
+~4k the long-range signal rides on 3 full-attention layers, and widening the manifest does
+not widen the sliding layers' 512-token window.
+
+## Qualification status
+
+| Artifact | Status |
+|---|---|
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED — measured.** 10/10 parse, 10/10 enum-clean, 7.06/4.27 s/row, 28.6 tok/s, 8.9 GB peak. The best Gemma in this catalog for structured output, and still not a default over the LFM tier. |
+| `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** — same weights, same graph, wider manifest integer; no run above 4096 tokens. |
+| `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** — as above. |
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl` | **DEPRECATED — superseded by its `_pf64` sibling.** Same weights, no `prefill` function: 8× the TTFT and it did not survive a ten-row campaign. Kept published because it is an immutable artifact that earlier revisions pin; there is no reason to ship it. |
+| `w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64` | **EXPERIMENTAL — built, unmeasured.** Gate: a Mac-side oracle/parity check plus a device benchmark, neither of which has been run. Do not route production traffic to it. |
+
+### About the `w4a8` bundle
+
+Same QAT int4 per-block-32 weights and the same graph as
+`gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64`; the difference is an int8
+quantize/dequantize pair on the inputs of every `F.linear`, calibrated on 128 synthetic
+social-media-style samples (the corpus ships in the folder). It was built as a prefill/TTFT
+lever.
+
+The caveat is load-bearing and comes from the compression toolchain's own documentation:
+the runtime fast path is `W_INT8 × A_INT8`, and a float weight path executes in floating
+point *regardless* of activation quantization. These weights use int4 scale-multiply
+dequant, not an int8 LUT, so the expected TTFT gain may be **zero or negative**. This bundle
+exists to be measured, not because a win is predicted.
+
+## Verification
+
+- **Producer fingerprint:** every inner `<name>.aimodel/metadata.json` reads
+  `producer: "coreai-core 1.0.0b2"`. `coreai-torch 0.4.0` / `coreai-core 1.0.0b1` produce
+  ODIE-poisoned bundles that abort **in-process** at load, taking the host application down
+  uncatchably. The field lives in the **inner asset** metadata; the bundle manifest never
+  carries a `producer` field, for good and bad bundles alike, so a check against the
+  manifest condemns every good bundle.
+- **Stop token, verified per folder:** every bundle here declares `eos_token = "<turn|>"`.
+  Gemma 4 ends a *turn* with `<turn|>` (id 106); the raw upstream `<eos>` ends the whole
+  sequence, and a host stopping on it overruns every reply. The exporter copies the upstream
+  tokenizer config verbatim and does **not** apply this correction — it is a post-export
+  patch, and it was checked on these bundles rather than assumed.
+  `generation_config.json` independently lists `eos_token_id: [1, 106, 50]`.
+- **Determinism:** `.aimodel` conversion is not byte-reproducible. `main.mlirb` differs by
+  ~22 kB between two runs of an identical command on the same box — which is why the
+  context-folder comparison above needed a same-config control to be meaningful at all.
+  Integrity rests on per-file SHA-256 of the exact published artifact, never on rebuilding.
+- **No oracle gate has been run on this model.** The 10/10 results are behavioural (parse
+  rate, enum conformance, clean stop) on Visible's own task. A decode oracle against an fp32
+  reference — the gate that would catch an int4 or kernel-level numerics error — has not been
+  run on any Gemma 4 bundle in this catalog.
+
+## License
+
+The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma
+model and **the Gemma terms apply downstream**, which is why this repo declares
+`license: gemma`. Use is subject to the
+[Gemma Terms of Use](https://ai.google.dev/gemma/terms), the
+[Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy) and the
+[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
+travel with any redistribution of these bundles, including the gather-table sidecar, which
+is derived from the same weights. Visible's contribution is the conversion, not the weights.
 
-Same QAT int4 per-block-32 weights and same graph as gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64; the only difference is an int8 quantize/dequantize pair on the inputs of every F.linear, calibrated on 128 synthetic social-media-style samples (calibration_corpus.jsonl ships in the bundle). Intended as a prefill/TTFT lever. READ THE CAVEAT: coreai-opt documents the runtime fast path as W_INT8-A_INT8, and states that a float weight path executes in floating point regardless of activation quantization. These weights use int4 scale-multiply dequant, not an int8 LUT, so the expected TTFT gain may be zero or negative. This bundle exists to be MEASURED, not because a win is predicted. ctx 16384. Unlike the 4096 folders, this one does include chat_template.jinja.
+---
 
-| Bundle | Source checkpoint | Context | Functions | Bytes | Producer | Created |
-|---|---|---|---|---|---|---|
-| `w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64` | google/gemma-4-E2B-it-qat-q4_0-unquantized | 16384 | main+prefill | 2,122,679,604 | coreai-core 1.0.0b2 | 20260817T222458Z |
+*Additional variants may land in this repo after this card was written; the tables above
+describe its contents as listed at the time of writing.*