docs: descriptive model card

README.md

@@ -1,64 +1,257 @@
 ---
+license: gemma
+license_link: https://ai.google.dev/gemma/terms
+base_model: google/gemma-4-E4B-it-qat-q4_0-unquantized
 library_name: coreai
-tags: [core-ai, aimodel, on-device, apple]
+pipeline_tag: text-generation
+tags:
+  - core-ai
+  - aimodel
+  - apple-silicon
+  - on-device
+  - quantized
+  - int4
+  - qat
+  - gemma4
 ---
 
-# gemma-4-E4B-CoreAI
+# Gemma 4 E4B — Core AI (.aimodel)
 
-Core AI `.aimodel` bundles converted for [Visible](https://visible.cx) on-device
-inference. **Derivative artifacts** — converted from the upstream checkpoints listed
-below; all original model licences apply and travel with these weights.
+`google/gemma-4-E4B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for
+Apple silicon by the [Visible](https://visible.cx) project. **Derivative artifacts**:
+Google's QAT-trained weights rounded onto the int4 grid they were trained for and
+re-expressed as a Core AI graph.
 
-| Bundle | Source checkpoint | Context | Functions | Producer | Created |
-|---|---|---|---|---|---|
-| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | google/gemma-4-E4B-it-qat-q4_0-unquantized | 4096 | main | coreai-core 1.0.0b2 | 20260817T105544Z |
-| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | google/gemma-4-E4B-it-qat-q4_0-unquantized | 4096 | main+prefill | coreai-core 1.0.0b2 | 20260817T111528Z |
+Gemma 4 E4B uses **Per-Layer Embeddings**, so the working bundles here take a large
+embedding gather table as a *static graph input* rather than carrying it in the graph. That
+table ships in this repo (`ios-frontend/`) and **the bundles do not run without it** — see
+Contents and Requirements, because a missing table produces a bare input-arity error that
+mentions nothing about tables.
+
+This model does grammar-constrained structured output correctly (10/10 enum-clean on
+Visible's schema) and it is **slow and memory-bound on a 16 GB Mac**. Both facts are
+measured below.
 
 ## Provenance
 
-- Converted with the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo)
-  conversion recipes on the pinned `apple/coreai-models` base
-  `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the zoo python overlay.
-- Toolchain: `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0`.
-- Every bundle's inner `<name>.aimodel/metadata.json` reports
-  `producer: "coreai-core 1.0.0b2"`.
-- Gemma bundles: `eos_token` is set to `<turn|>` (Gemma 4 ends a *turn* with `<turn|>`;
-  stopping on the source's `<eos>` overruns the reply).
-- Bundles carry `runtime_env COREAI_CHUNK_THRESHOLD=1`.
+| | |
+|---|---|
+| Base checkpoint | `google/gemma-4-E4B-it-qat-q4_0-unquantized` (ungated) |
+| Recipe (decode) | `export_gemma4_decode_pipelined.py int4lin` — zoo recipe `gemma-4-e4b`, `status = "verified"` |
+| Recipe (pf64) | `export_gemma4_pf_pipelined.py --pf 64` with `--tbl` and `--raw-dir` pointed at the gather table below |
+| Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay |
+| Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` |
+| Producer fingerprint | `coreai-core 1.0.0b2` on every inner asset `metadata.json` |
+| Weight format | **int4, per-block-32** (`int4lin`, symmetric-with-clipping) — the ggml q4_0 grid the QAT checkpoint was trained on |
+| Vocab | 262,144 |
+| Conversion | 241 s / 44.0 GB peak RSS (decode); 196 s / 42.7 GB (pf64); Linux x86_64 |
+
+"QAT-unquantized" means QAT-*trained*, stored full width; the int4 rounding happens at
+export onto the grid training already targeted.
+
+`_tbl` = the PLE gather table is bound as a static graph input. `_pf64` = a second
+entrypoint, `prefill`, with a static query width of 64
+(`function_map: {"main": ["main", "prefill"]}`).
+
+## Contents
+
+### Bundles
+
+| Path | Bytes | Context | Functions | Status |
+|---|---:|---|---|---|
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,858 | 4096 | main + prefill | the working bundle |
+| `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,830 | 8192 | main + prefill | same weights, wider manifest |
+| `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,844 | 16384 | main + prefill | same weights, wider manifest |
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | 3,989,205,604 | 4096 | main | **BLOCKED — see below** |
+
+Each folder holds `<name>.aimodel/` (`main.mlirb` ≈ 3.96 GB, `main.hash`, asset
+`metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
+32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
+18,569 B).
+
+**The three `_tbl_pf64` folders are the same weights.** `--max-ctx` does not change the
+exported graph — identical function signatures, identical state descriptors, identical
+export peak RSS at every value. It changes exactly one thing:
+`language.max_context_length` in the bundle manifest. This was proved with a same-config
+control re-export whose `main.mlirb` differed from its twin by *more* bytes than a
+4096→8192 pair does. Pick the folder whose manifest integer matches the window you intend
+to run; a future context bump is a manifest edit, not a conversion.
+
+### The PLE gather-table sidecar — required, not optional
+
+| Path | Files | Bytes |
+|---|---|---:|
+| `ios-frontend/gemma4_e4b_qat_gather_raw/` | 7 | 3,601,859,815 |
+
+| File | Bytes |
+|---|---:|
+| `embed_per_layer.i8` | 2,818,572,288 |
+| `embed_tokens.i8` | 671,088,640 |
+| `proj.f32` | 110,100,480 |
+| `embed_per_layer.scale.f32` | 1,048,576 |
+| `embed_tokens.scale.f32` | 1,048,576 |
+| `proj_norm.f32` | 1,024 |
+| `meta.json` | 231 |
+
+Every `_tbl` bundle binds `ple_table` (from `embed_per_layer.i8`) and `ple_scale` (from
+`embed_per_layer.scale.f32`) as **static** graph inputs. Without them the bundle fails to
+load with a bare input-arity error naming `ple_table`/`ple_scale` and saying nothing about
+tables — a diagnosis this project got wrong once already. The sidecar is published here so
+the repo is self-contained for both loading and a pf64 rebuild.
 
-Upstream sources: `google/gemma-4-E4B-it-qat-q4_0-unquantized`
+## Requirements
 
+- **Apple silicon Mac**, Core AI runtime.
+- **Engine contract: 4 inputs** — `input_ids`, `position_ids`, plus the static `ple_table`
+  and `ple_scale`. Two engines can take that:
+  - **Pipelined engine** — binds the static inputs zero-copy over the caller's buffer, but
+    **does not support logits**, so no grammar-constrained decoding.
+  - **Sequential engine** — the only logits-capable engine, and therefore the only one
+    guided decoding can run on. It requires the static-input binding that accepts `>= 2`
+    inputs and binds everything beyond `input_ids`/`position_ids` from
+    `EngineOptions.staticInputBuffers`; an extra input with no buffer is rejected by name.
+    It pays a **one-time copy of every constant input at load — 2.69 GB for E4B** — because
+    its submission path materialises a foreign buffer-backed view on every pass otherwise
+    (measured: binding the table as a raw view drove peak footprint to 14.9 GB and
+    segfaulted the runtime mid-prefill).
+- **States:** `keyCache` / `valueCache` `Float16, 24 × 1 × 2 × ? × 512`. Dynamic sequence
+  dim → `GrowingKVCache` (initial 256, doubling).
+- **KV cost: 98,304 bytes per token** (fp16) — 403 MB at 4096, 805 MB at 8192, 1.61 GB at
+  16384.
+- **Memory envelope, 16 GB Mac** (Metal `recommendedMaxWorkingSetSize` ≈ 10.7 GB):
 
-## High-context bundles — ctx 8192
+  | | |
+  |---|---|
+  | Weights (`main.mlirb`) | 3.96 GB |
+  | PLE gather table (static input) | 2.82 GB |
+  | Base | **6.78 GB** |
+  | Affordable context, fp16 KV | **~39,900 tokens** |
 
-Same weights, same graph, same recipe as the 4096 bundles above — the exporter's
-`--max-ctx` only widens the dynamic sequence bound and the manifest's
-`language.max_context_length`. KV cache stays fp16.
+  So KV was never the ceiling at this tier — 16384 costs 1.61 GB against 3.92 GB of
+  headroom. **The real constraint is the PLE table**, which dwarfs the KV cache at any
+  context worth shipping. Measured peak footprint on a guided run was **10.7 GB**, i.e.
+  right at the working-set edge.
+- **Run one E4B campaign per process.** Wired memory from a Gemma campaign is not reclaimed
+  until the process exits; a second full E4B run in the same session drove wired memory to
+  13.2 GB and left the machine with 63 MB free.
+- Sliding-window note: E4B interleaves 20 sliding and 4 full attention layers across 42
+  layers collapsed to 24 KV slots. The export models the window in the *mask*, not in
+  memory — sliding layers ride full-length slots and zero-pad head_dim 256→512, so 83.3% of
+  KV bytes sit in windowed slots and 41.7% is dead padding. A ring-buffered sliding cache
+  would be ~6× cheaper per token. That is model authoring, not a flag, and it is not needed
+  at this tier.
+- `runtime_env COREAI_CHUNK_THRESHOLD=1` travels with the recipe. (Both engines now derive
+  the static query width from the graph, so the older host-side rules about warmup length
+  and chunk threshold are retired.)
 
-| Bundle | Source checkpoint | Context | Functions | KV dtype | Producer | Created |
-|---|---|---|---|---|---|---|
-| `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | google/gemma-4-E4B-it-qat-q4_0-unquantized | 8192 | main+prefill | fp16 | coreai-core 1.0.0b2 | 20260817T204459Z |
+## Measured performance
 
+Measured on a **16 GB M2 Pro Mac (macOS 27 beta)** with the out-of-process `coreai-repro`
+harness: 10 real enrichment rows (5 POST, 5 COMMENT), the app's live `app_litert` JSON
+schema, greedy decoding through `respondJSON(to:schema:)`, `reset()` per row, **sequential
+engine with the PLE tables bound as static inputs**, 128-token cap.
 
-## High-context bundles — ctx 16384
+`gemma4_e4b_qat_decode_int4lin_tbl_pf64`:
 
-Same weights, same graph, same recipe as the 4096 bundles above — the exporter's
-`--max-ctx` only widens the dynamic sequence bound and the manifest's
-`language.max_context_length`. KV cache stays fp16.
+| | |
+|---|---|
+| Load | 26.1 s |
+| Guided JSON parse | **10/10** |
+| Enum-clean | **10/10** |
+| s/row | **21.89** (POST) / **12.36** (COMMENT) |
+| Decode | 6.7–15.9 tok/s, mean **10.2** |
+| TTFT | 2.8–9.7 s |
+| Peak footprint | 10.7 GB |
+| Stop token | `<turn\|>`, clean self-stop on every row |
 
-| Bundle | Source checkpoint | Context | Functions | KV dtype | Producer | Created |
-|---|---|---|---|---|---|---|
-| `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | google/gemma-4-E4B-it-qat-q4_0-unquantized | 16384 | main+prefill | fp16 | coreai-core 1.0.0b2 | 20260817T204711Z |
+**The grammar is close to free; prefill is the cost.** Decode throughput is essentially
+unchanged from unguided pipelined measurements (9.3–11.0 → 10.2 tok/s). Guided rows cost
+~25% more than unguided ones because the sequential engine's prefill is step-synchronous,
+not because of the constraint.
 
+**Enum conformance is entirely the grammar's doing, and it is decisive.** Unguided, this
+model invented an off-schema `motivation` value in **all ten** rows (`empowerment`,
+`advocacy`, `informative`…, none of them schema members). Guided, all ten are correct — not
+because the model improved, but because an off-enum token is unsamplable.
 
-## w4a8/ — int4 weights + INT8 activations (EXPERIMENTAL)
+**Context for the cost.** On the same harness and machine, LFM2.5-1.2B runs 1.77/1.30 s/row
+and Gemma 4 E2B runs 7.06/4.27. This bundle is roughly **12× the 1.2B's cost per row** and
+is memory-bound at 10.7 GB peak. Published Mac figures of ~55.8 tok/s for E4B decode should
+be read as bigger-Mac figures.
 
-> **EXPERIMENTAL — not qualified.** These bundles have not passed a Mac-side
-> oracle/parity gate or a device benchmark. Do not route production traffic to
-> them until they have.
+Nothing was measured above 4096 tokens. The `ctx8192`/`ctx16384` folders carry the 4096
+graph, so throughput should be unchanged, but quality at depth is a real open question:
+beyond ~4k the long-range signal rides on 4 full-attention layers, and widening the manifest
+does not widen the sliding layers' 512-token window.
 
-Same QAT int4 per-block-32 weights and same graph as gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64; the only difference is an int8 quantize/dequantize pair on the inputs of every F.linear, calibrated on 128 synthetic social-media-style samples (calibration_corpus.jsonl ships in the bundle). Intended as a prefill/TTFT lever. READ THE CAVEAT: coreai-opt documents the runtime fast path as W_INT8-A_INT8 and states a float weight path executes in floating point regardless of activation quantization. These weights use int4 scale-multiply dequant, not an int8 LUT, so the expected TTFT gain may be zero or negative. This bundle exists to be MEASURED. ctx 16384; includes chat_template.jinja, which the 4096 folders do not.
+## Qualification status
+
+| Artifact | Status |
+|---|---|
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED — measured, viable, not a default.** 10/10 parse, 10/10 enum-clean on the harness above. Slow and memory-bound at the 16 GB tier; it earns a slot only where Gemma's output specifically is wanted. |
+| `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** — same weights, same graph, wider manifest integer. No run above 4096 tokens. |
+| `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** — as above. |
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | **BLOCKED — does not load on any current code path.** |
+
+### Why the non-`tbl` decode bundle is blocked
+
+It declares a **different third input** from the `_tbl` family:
+
+```
+LOAD FAILED: invalidInputType("Expected 2 inputs, got 3:
+  [\"input_ids\", \"position_ids\", \"ple_tokens\"]")
+```
+
+`ple_tokens`, not `ple_table`/`ple_scale`. The Gemma architecture description in the kit
+knows only the pair (`ple_table` → `embed_per_layer.i8`, `ple_scale` →
+`embed_per_layer.scale.f32`), so **no code path binds `ple_tokens`** and the table sidecar
+in this repo does not help. It is unusable as exported. The fix is a re-export with `--tbl`
+so E4B decode and prefill are the same graph family, not an extension to the kit; until
+then, use the `_tbl_pf64` bundle, which is a strict superset (it carries both `main` and
+`prefill`).
+
+It is kept published rather than deleted because it is a fingerprinted, immutable artifact
+that earlier revisions pin, and because the failure it demonstrates is worth documenting.
+
+## Verification
+
+- **Producer fingerprint:** every inner `<name>.aimodel/metadata.json` reads
+  `producer: "coreai-core 1.0.0b2"`. `coreai-torch 0.4.0` / `coreai-core 1.0.0b1` produce
+  ODIE-poisoned bundles that abort **in-process** at load, taking the host application down
+  uncatchably. The field is in the **inner asset** metadata; the bundle manifest never
+  carries a `producer` field, so checking the manifest condemns every good bundle.
+- **Stop token, verified per folder:** all four declare `eos_token = "<turn|>"`. Gemma 4
+  ends a *turn* with `<turn|>` (id 106); the raw upstream `<eos>` ends the whole sequence,
+  and a host stopping on it overruns every reply. The exporter copies the upstream tokenizer
+  config verbatim and does **not** apply this correction — it is a post-export patch, and it
+  was checked on these bundles rather than assumed. `generation_config.json` independently
+  lists `eos_token_id: [1, 106, 50]`.
+- **`pf64` is worth 7.7× on time-to-first-token, measured.** The A/B was run on the E2B
+  sibling (same prompt, same weights, differing only in whether the `prefill` function was
+  exported): TTFT 35.74 s → 4.65 s, s/row 45.04 → 8.83. The non-`pf64` variant also failed
+  partway through a ten-row campaign with the runtime's footprint ballooning past 37 GB
+  across ~2,800 single-token passes — it is not merely slower, it is not durable. Prefer
+  `_pf64` everywhere; that is why no plain `_tbl` E4B bundle exists here.
+- **Determinism:** `.aimodel` conversion is not byte-reproducible; the same command twice on
+  one box gives different `main.mlirb` bytes. Integrity rests on per-file SHA-256 of the
+  exact published artifact, never on rebuilding.
+- **No oracle gate has been run on this model.** The 10/10 results above are behavioural
+  (parse rate, enum conformance, clean stop) on Visible's own task. A decode oracle against
+  an fp32 reference — the gate that would catch an int4 or kernel-level numerics error — has
+  not been run on any Gemma 4 bundle in this catalog.
+
+## License
+
+The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma
+model and **the Gemma terms apply downstream**, which is why this repo declares
+`license: gemma`. Use is subject to the
+[Gemma Terms of Use](https://ai.google.dev/gemma/terms), the
+[Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy) and the
+[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
+travel with any redistribution of these bundles, including the gather-table sidecar, which
+is derived from the same weights. Visible's contribution is the conversion, not the weights.
+
+---
 
-| Bundle | Source checkpoint | Context | Functions | Bytes | Producer | Created |
-|---|---|---|---|---|---|---|
-| `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | google/gemma-4-E4B-it-qat-q4_0-unquantized | 16384 | main+prefill | 3,990,690,638 | coreai-core 1.0.0b2 | 20260817T223600Z |
+*Additional variants may land in this repo after this card was written; the tables above
+describe its contents as listed at the time of writing.*