docs: refresh measured performance (local retest 2026-08-18)

README.md

@@ -18,19 +18,24 @@ tags:
 # Gemma 4 E2B — Core AI (.aimodel)
 
 `google/gemma-4-E2B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for
-Apple silicon by the [Visible](https://visible.cx) project. **Derivative artifacts**:
-Google's QAT-trained weights rounded onto the int4 grid they were trained for and
-re-expressed as a Core AI graph.
+Apple silicon by [visible-cx](https://huggingface.co/visible-cx). These are derivative
+artifacts: Google's QAT-trained weights rounded onto the int4 grid they were trained for
+and re-expressed as a Core AI graph. They load through Core AI on macOS and are not usable
+by PyTorch, GGUF or MLX.
 
 Gemma 4 E2B uses **Per-Layer Embeddings**, so these bundles take a large embedding gather
-table as a *static graph input* rather than carrying it in the graph. That table ships here
-in `ios-frontend/` and **the bundles do not load without it** — a missing table produces a
-bare input-arity error that says nothing about tables.
-
-Of the Gemma 4 bundles in this catalog, this is **the only one worth considering for
-structured-output work**: 10/10 schema-clean at 29 tok/s on a 16 GB Mac. It is still 4× the
-cost per row of the LFM2.5 1.2B at 3× the peak memory, so it earns a slot only where
-Gemma's output specifically is wanted.
+table as a *static graph input* rather than carrying it in the graph. That table ships in
+`ios-frontend/` and **the bundles do not load without it**; a missing table produces a bare
+input-arity error naming `ple_table`/`ple_scale`.
+
+> ⚠️ **Known issue — unguided generation.** On the current reference engine
+> implementation, free-form (unguided) decoding exhibits unbounded Metal memory growth:
+> approximately **80 MB of process `phys_footprint` per generated token**, on both the
+> sequential and pipelined engines, scaling with prompt length. Long generations exhaust
+> memory and the process is terminated. **Constrained/guided decoding is unaffected** and
+> plateaus at 8–10 GB. A fix is in progress. Until it lands, use these bundles for
+> guided/constrained generation only. A fixed token cap is not a reliable mitigation,
+> because the safe cap shrinks as the prompt grows.
 
 ## Provenance
 
@@ -44,7 +49,7 @@ Gemma's output specifically is wanted.
 | Producer fingerprint | `coreai-core 1.0.0b2` on every inner asset `metadata.json` |
 | Weight format | **int4, per-block-32** (`int4lin`, symmetric-with-clipping) — the ggml q4_0 grid the QAT checkpoint was trained on |
 | Vocab | 262,144 |
-| Conversion | 180 s (decode) / 105 s, 26.8 GB peak RSS (pf64); Linux x86_64 |
+| Export functions | `main` (S=1 decode) and, in `_pf64` bundles, `prefill` (S=64 chunked prefill) |
 
 "QAT-unquantized" means QAT-*trained*, stored at full width; the int4 rounding happens at
 export, onto the grid training already targeted.
@@ -57,30 +62,30 @@ entrypoint, `prefill`, with a static query width of 64
 
 ### Bundles
 
-| Path | Bytes | Context | Functions | Status |
-|---|---:|---|---|---|
-| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,973 | 4096 | main + prefill | the working bundle |
-| `ctx8192/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,964 | 8192 | main + prefill | same weights, wider manifest |
-| `ctx16384/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,964 | 16384 | main + prefill | same weights, wider manifest |
-| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl` | 2,121,479,596 | 4096 | main | superseded — no `prefill`, see below |
-| `w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64` | 2,122,679,604 | 16384 | main + prefill | **EXPERIMENTAL** |
+| Path | Bytes | Context | Functions |
+|---|---:|---|---|
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,973 | 4096 | main + prefill |
+| `ctx8192/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,964 | 8192 | main + prefill |
+| `ctx16384/gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` | 2,122,089,964 | 16384 | main + prefill |
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl` | 2,121,479,596 | 4096 | main |
+| `w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64` | 2,122,679,604 | 16384 | main + prefill |
 
 Each bundle folder holds `<name>.aimodel/` (`main.mlirb` ≈ 2.09 GB, `main.hash`, asset
 `metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
 32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
-18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl`
-(35,045 B).
-
-**The three `_tbl_pf64` context folders are the same weights.** `--max-ctx` does not change
-the exported graph — identical function signatures, identical state descriptors, identical
-export peak RSS at every value. It changes exactly one thing:
-`language.max_context_length` in the bundle manifest. This was proved with a same-config
-control re-export whose `main.mlirb` differed from its twin by *more* bytes (22,053) than a
-4096→8192 pair does (22,344); the context delta is inside the conversion noise floor.
-Corroborated by the upload itself: pushing a "new" ~2 GB ctx8192 bundle into a repo that
-already held the 4096 bundle transferred a few hundred kB of new data. Pick the folder whose
-manifest integer matches the window you intend to run; a context bump is a manifest edit,
-not a conversion.
+18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl` (35,045 B).
+
+**The three `_tbl_pf64` context folders hold the same weights and the same graph.**
+`--max-ctx` changes exactly one thing: `language.max_context_length` in the bundle
+manifest. Function signatures, state descriptors and export peak RSS are identical at every
+value, and the byte differences between the folders are conversion nondeterminism rather
+than content. Pick the folder whose manifest integer matches the window you intend to run;
+a context bump is a manifest edit, not a conversion.
+
+**Stop token:** every bundle declares `eos_token = "<turn|>"` (id 106), which is the
+turn terminator Gemma 4 emits. `generation_config.json` independently lists
+`eos_token_id: [1, 106, 50]`. A host that stops on the raw upstream `<eos>` instead will
+overrun every reply.
 
 ### The PLE gather-table sidecar — required, not optional
 
@@ -103,26 +108,22 @@ not a conversion.
 exporter assumes).
 
 Every `_tbl` bundle binds `ple_table` (from `embed_per_layer.i8`, 2.35 GB) and `ple_scale`
-(from `embed_per_layer.scale.f32`) as **static** graph inputs. Without them the bundle fails
-to load with a bare arity error naming `ple_table`/`ple_scale` — a diagnosis this project
-got wrong once already. The sidecar is published here so the repo is self-contained for
-both loading and a pf64 rebuild.
+(from `embed_per_layer.scale.f32`) as **static** graph inputs.
 
 ## Requirements
 
 - **Apple silicon Mac**, Core AI runtime.
 - **Engine contract: 4 inputs** — `input_ids`, `position_ids`, plus static `ple_table` and
-  `ple_scale`. Two engines can take that:
-  - **Pipelined engine** — binds the statics zero-copy over the caller's buffer, but
-    **does not support logits**, so no grammar-constrained decoding.
-  - **Sequential engine** — the only logits-capable engine, and therefore the only one
-    guided decoding can run on. It accepts `>= 2` inputs and binds everything beyond
+  `ple_scale`. Two engines accept that:
+  - **Pipelined engine** — binds the statics zero-copy over the caller's buffer, but does
+    not support logits, so no grammar-constrained decoding.
+  - **Sequential engine** — the only logits-capable engine, and therefore the only path for
+    guided decoding. It accepts `>= 2` inputs and binds everything beyond
     `input_ids`/`position_ids` from `EngineOptions.staticInputBuffers`; an extra input with
     no buffer is rejected by name. It pays a **one-time copy of every constant input at
-    load — 2.19 GB for E2B** — because its submission path otherwise materialises a foreign
-    buffer-backed view on *every* forward pass. That was measured: binding the table as a
-    raw view cost ~0.30 s per pass (3.4 tok/s, 64 s/row); copying once at load took the same
-    bundle to 8.6 s/row, a 7.5× improvement.
+    load — 2.19 GB for E2B**, because its submission path otherwise materialises a foreign
+    buffer-backed view on every forward pass. Binding the table as a raw view instead costs
+    ~0.30 s per pass (3.4 tok/s); the load-time copy takes the same bundle to 8.6 s/row.
 - **States:** `keyCache` / `valueCache` `Float16, 15 × 1 × 1 × ? × 512`. Dynamic sequence
   dim → `GrowingKVCache` (initial 256, doubling), not a static allocation at the manifest
   maximum.
@@ -137,106 +138,93 @@ both loading and a pf64 rebuild.
   | Base | **4.44 GB** |
   | Affordable context, fp16 KV | ~204k tokens → **131,072, the model cap, binds first** |
 
-  KV is not the ceiling at this tier; **the PLE table is**, at 2.35 GB against a 503 MB KV
-  cost even at 16384. Measured peak footprint on a guided run: **8.9 GB**.
+  KV is not the ceiling at this tier; the PLE table is. Measured peak footprint on a guided
+  run: **8.26 GB**, max RSS 5.40 GB. **Minimum practical machine memory: 16 GB**, for
+  guided decoding only. Free-form generation does not fit this envelope on any machine —
+  see the known issue above.
 - Sliding-window note: E2B interleaves 12 sliding and 3 full attention layers across 35
   layers collapsed to 15 KV slots. The export models the window in the *mask*, not in
   memory — sliding layers occupy full-length slots and zero-pad head_dim 256→512, so 80% of
-  KV bytes sit in windowed slots and 40% is dead padding. A ring-buffered sliding cache
-  would be ~5× cheaper per token; that is model authoring, not a flag, and it is not needed
-  at this tier.
-- `runtime_env COREAI_CHUNK_THRESHOLD=1` travels with the recipe. (Both engines now read the
-  static query width off the graph, so the older host-side rules about warmup query length
-  and chunk threshold are retired.)
+  KV bytes sit in windowed slots and 40% is dead padding.
+- The bundle manifest declares `runtime_env COREAI_CHUNK_THRESHOLD=1`. Both engines read
+  the static query width off the graph.
 
-## Measured performance
+## Performance
 
-Measured on a **16 GB M2 Pro Mac (macOS 27 beta)** with the out-of-process `coreai-repro`
-harness: 10 real enrichment rows (5 POST, 5 COMMENT), the app's live `app_litert` JSON
-schema, greedy decoding through `respondJSON(to:schema:)`, `reset()` per row, **sequential
-engine with the PLE tables bound as static inputs**, 128-token cap.
+Measured on a **16 GB Apple silicon Mac (M2 Pro, macOS 27 beta)**: 10-sample harness,
+guided JSON-constrained decoding against a fixed schema, greedy, sequential engine with the
+PLE tables bound as static inputs, `reset()` between samples, 128-token cap. Load excluded
+from `s/row`; sample 1 excluded as a cache-warm outlier.
 
 | | `_tbl_pf64` | `_tbl` (no `prefill` function) |
 |---|---|---|
-| Load | 14.6 s | 29.0 s |
+| Load | 12.1 s | 29.0 s |
 | Guided JSON parse | **10/10** | 3/3 |
 | Enum-clean | **10/10** | 3/3 |
-| s/row (POST) | **7.06** | 44.69 |
-| s/row (COMMENT) | **4.27** | — |
-| Decode | 21.8–33.5 tok/s, mean **28.6** | 7.7–16.8 tok/s |
-| TTFT | 0.6–4.7 s | 29–37 s |
-| Peak footprint | 8.9 GB | — |
-| Outcome | completed all ten rows | **died at row 4** |
+| s/row (long samples) | **6.12** | 44.69 |
+| s/row (short samples) | **4.23** | — |
+| Decode | **22.7–32.7 tok/s** | 7.7–16.8 tok/s |
+| TTFT | **0.59–4.20 s** | 29–37 s |
+| Peak footprint | 8.26 GB | — |
+| Max RSS | 5.40 GB | — |
+| Outcome | completed all ten samples | did not complete |
 
 **`pf64` is worth 7.7× on time-to-first-token.** Head to head on the same 942-token prompt
 and the same weights, differing only in whether the S=64 `prefill` function was exported:
 TTFT 35.74 s → 4.65 s, s/row 45.04 → 8.83. Decode is untouched — the prefill function only
-serves full-width chunks. The non-`pf64` bundle is also not durable: its footprint ballooned
-past 37 GB across ~2,800 single-token passes and the campaign died partway through.
+serves full-width chunks.
+
+**Enum conformance is the grammar's doing.** Unguided, the model emits an off-schema enum
+value on essentially every sample. Guided, all ten are correct, because an off-enum token
+is unsamplable. Decode throughput is essentially unchanged by the constraint; the ~25%
+extra cost of a guided sample is the sequential engine's step-synchronous prefill.
+
+For reference, upstream measured E2B at 77.0/87.1 tok/s on an M4 Max, and an unguided
+pipelined run on the 16 GB machine reached 44.3 tok/s. The 22.7–32.7 tok/s above is the
+guided, sequential-engine figure.
+
+### Unguided workload — memory-capped
 
-**Enum conformance is the grammar's doing, and it is decisive.** Unguided, Gemma 4 invents
-an off-schema `motivation` value on essentially every row (`advocacy` for `advocate`,
-`informative` — a valid *tone* — used as a *motivation*). Guided, all ten are correct, not
-because the model improved but because an off-enum token is unsamplable. Decode throughput
-is essentially unchanged by the constraint; the ~25% extra cost of a guided row is the
-sequential engine's step-synchronous prefill, not the grammar.
+Free-form generation from a fixed prompt. **Every generation length below is a memory cap,
+not a model stop** — see the known issue.
 
-**Context for the cost.** On the same harness and machine, LFM2.5-1.2B runs 1.77/1.30 s/row
-at 3.06 GB peak, and Gemma 4 E4B runs 21.89/12.36 s/row at 10.7 GB peak. This bundle is
-about **4× the 1.2B's s/row for 3× its peak memory**. For reference, upstream measured E2B
-at 77.0/87.1 tok/s on an M4 Max, and an unguided pipelined run on this 16 GB Mac reached
-44.3 tok/s — the 28.6 tok/s above is the guided, sequential-engine figure.
+| depth | prompt tokens | TTFT | decode | generated | wall | peak footprint |
+|---|---|---|---|---|---|---|
+| 3.4k | 3314 | 12.21 s | 7.2 tok/s | 128 (capped) | 30.4 s | **19.97 GB** |
+| 8k | 7972 | 54.01 s | 2.1 tok/s | 64 (capped) | 84.3 s | **36.71 GB** |
 
-Nothing was measured above 4096 tokens. Quality at depth is a genuine open question: beyond
-~4k the long-range signal rides on 3 full-attention layers, and widening the manifest does
-not widen the sliding layers' 512-token window.
+Output quality up to the cap is sound; the limit is memory, not capability.
 
-## Qualification status
+**Needle-in-haystack recall: 3/3 verbatim at 8k**, within a 64-token cap (three distinctive
+facts planted at 10/50/90% of the filler, strict scoring). 15k was not attempted: 8k
+already cost 36.71 GB of footprint.
+
+## Status
 
 | Artifact | Status |
 |---|---|
-| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED — measured.** 10/10 parse, 10/10 enum-clean, 7.06/4.27 s/row, 28.6 tok/s, 8.9 GB peak. The best Gemma in this catalog for structured output, and still not a default over the LFM tier. |
-| `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** — same weights, same graph, wider manifest integer; no run above 4096 tokens. |
-| `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** ��� as above. |
-| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl` | **DEPRECATED — superseded by its `_pf64` sibling.** Same weights, no `prefill` function: 8× the TTFT and it did not survive a ten-row campaign. Kept published because it is an immutable artifact that earlier revisions pin; there is no reason to ship it. |
-| `w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64` | **EXPERIMENTAL — built, unmeasured.** Gate: a Mac-side oracle/parity check plus a device benchmark, neither of which has been run. Do not route production traffic to it. |
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED FOR GUIDED GENERATION** — measured: 10/10 parse, 10/10 enum-clean, 6.12/4.23 s/row, 22.7–32.7 tok/s, 8.26 GB peak footprint. Not qualified for free-form generation on the current reference engine implementation. |
+| `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE** — same weights and graph, wider manifest integer. Free-form at this depth is measured-unsafe (36.71 GB under a 64-token cap). |
+| `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE** — as above; free-form at this depth was not attempted. |
+| `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl` | **DEPRECATED** — same weights, no `prefill` function: 8× the TTFT and it did not complete a ten-sample run. Superseded by its `_pf64` sibling. |
+| `w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64` | **EXPERIMENTAL** — built, unmeasured. Gate: a Mac-side oracle/parity check plus a device benchmark. |
 
 ### About the `w4a8` bundle
 
 Same QAT int4 per-block-32 weights and the same graph as
 `gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64`; the difference is an int8
 quantize/dequantize pair on the inputs of every `F.linear`, calibrated on 128 synthetic
-social-media-style samples (the corpus ships in the folder). It was built as a prefill/TTFT
-lever.
-
-The caveat is load-bearing and comes from the compression toolchain's own documentation:
-the runtime fast path is `W_INT8 × A_INT8`, and a float weight path executes in floating
-point *regardless* of activation quantization. These weights use int4 scale-multiply
-dequant, not an int8 LUT, so the expected TTFT gain may be **zero or negative**. This bundle
-exists to be measured, not because a win is predicted.
-
-## Verification
-
-- **Producer fingerprint:** every inner `<name>.aimodel/metadata.json` reads
-  `producer: "coreai-core 1.0.0b2"`. `coreai-torch 0.4.0` / `coreai-core 1.0.0b1` produce
-  ODIE-poisoned bundles that abort **in-process** at load, taking the host application down
-  uncatchably. The field lives in the **inner asset** metadata; the bundle manifest never
-  carries a `producer` field, for good and bad bundles alike, so a check against the
-  manifest condemns every good bundle.
-- **Stop token, verified per folder:** every bundle here declares `eos_token = "<turn|>"`.
-  Gemma 4 ends a *turn* with `<turn|>` (id 106); the raw upstream `<eos>` ends the whole
-  sequence, and a host stopping on it overruns every reply. The exporter copies the upstream
-  tokenizer config verbatim and does **not** apply this correction — it is a post-export
-  patch, and it was checked on these bundles rather than assumed.
-  `generation_config.json` independently lists `eos_token_id: [1, 106, 50]`.
-- **Determinism:** `.aimodel` conversion is not byte-reproducible. `main.mlirb` differs by
-  ~22 kB between two runs of an identical command on the same box — which is why the
-  context-folder comparison above needed a same-config control to be meaningful at all.
-  Integrity rests on per-file SHA-256 of the exact published artifact, never on rebuilding.
-- **No oracle gate has been run on this model.** The 10/10 results are behavioural (parse
-  rate, enum conformance, clean stop) on Visible's own task. A decode oracle against an fp32
-  reference — the gate that would catch an int4 or kernel-level numerics error — has not been
-  run on any Gemma 4 bundle in this catalog.
+samples (the corpus ships in the folder). It was built as a prefill/TTFT lever.
+
+The caveat comes from the compression toolchain's own documentation: the runtime fast path
+is `W_INT8 × A_INT8`, and a float weight path executes in floating point *regardless* of
+activation quantization. These weights use int4 scale-multiply dequant, not an int8 LUT, so
+the expected TTFT gain may be zero or negative. This bundle exists to be measured, not
+because a win is predicted.
+
+**No numerics gate has been run on any bundle in this repo.** The 10/10 results are
+behavioural (parse rate, enum conformance, clean stop); a decode oracle against an fp32
+reference has not been run.
 
 ## License
 
@@ -247,9 +235,4 @@ model and **the Gemma terms apply downstream**, which is why this repo declares
 [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy) and the
 [Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
 travel with any redistribution of these bundles, including the gather-table sidecar, which
-is derived from the same weights. Visible's contribution is the conversion, not the weights.
-
----
-
-*Additional variants may land in this repo after this card was written; the tables above
-describe its contents as listed at the time of writing.*
+is derived from the same weights. The contribution here is the conversion, not the weights.