docs: refresh measured performance (local retest 2026-08-18)

README.md

@@ -18,19 +18,25 @@ tags:
 # Gemma 4 E4B — Core AI (.aimodel)
 
 `google/gemma-4-E4B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for
-Apple silicon by the [Visible](https://visible.cx) project. **Derivative artifacts**:
-Google's QAT-trained weights rounded onto the int4 grid they were trained for and
-re-expressed as a Core AI graph.
-
-Gemma 4 E4B uses **Per-Layer Embeddings**, so the working bundles here take a large
-embedding gather table as a *static graph input* rather than carrying it in the graph. That
-table ships in this repo (`ios-frontend/`) and **the bundles do not run without it** — see
-Contents and Requirements, because a missing table produces a bare input-arity error that
-mentions nothing about tables.
-
-This model does grammar-constrained structured output correctly (10/10 enum-clean on
-Visible's schema) and it is **slow and memory-bound on a 16 GB Mac**. Both facts are
-measured below.
+Apple silicon by [visible-cx](https://huggingface.co/visible-cx). These are derivative
+artifacts: Google's QAT-trained weights rounded onto the int4 grid they were trained for
+and re-expressed as a Core AI graph. They load through Core AI on macOS and are not usable
+by PyTorch, GGUF or MLX.
+
+Gemma 4 E4B uses **Per-Layer Embeddings**, so the working bundles take a large embedding
+gather table as a *static graph input* rather than carrying it in the graph. That table
+ships in `ios-frontend/` and **the bundles do not load without it**; a missing table
+produces a bare input-arity error naming `ple_table`/`ple_scale`.
+
+> ⚠️ **Known issue — unguided generation.** On the current reference engine
+> implementation, free-form (unguided) decoding exhibits unbounded Metal memory growth:
+> approximately **80 MB of process `phys_footprint` per generated token**, on both the
+> sequential and pipelined engines, scaling with prompt length. E4B is the more severe of
+> the two Gemma variants: 16.91 GB after 64 generated tokens at 3.4k of context, and
+> 31.33 GB after 32 tokens at 8k. **Constrained/guided decoding is unaffected** and
+> plateaus at 8–10 GB. A fix is in progress. Until it lands, use these bundles for
+> guided/constrained generation only. A fixed token cap is not a reliable mitigation,
+> because the safe cap shrinks as the prompt grows.
 
 ## Provenance
 
@@ -44,7 +50,7 @@ measured below.
 | Producer fingerprint | `coreai-core 1.0.0b2` on every inner asset `metadata.json` |
 | Weight format | **int4, per-block-32** (`int4lin`, symmetric-with-clipping) — the ggml q4_0 grid the QAT checkpoint was trained on |
 | Vocab | 262,144 |
-| Conversion | 241 s / 44.0 GB peak RSS (decode); 196 s / 42.7 GB (pf64); Linux x86_64 |
+| Export functions | `main` (S=1 decode) and, in `_pf64` bundles, `prefill` (S=64 chunked prefill) |
 
 "QAT-unquantized" means QAT-*trained*, stored full width; the int4 rounding happens at
 export onto the grid training already targeted.
@@ -57,27 +63,29 @@ entrypoint, `prefill`, with a static query width of 64
 
 ### Bundles
 
-| Path | Bytes | Context | Functions | Status |
-|---|---:|---|---|---|
-| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,858 | 4096 | main + prefill | the working bundle |
-| `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,830 | 8192 | main + prefill | same weights, wider manifest |
-| `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,844 | 16384 | main + prefill | same weights, wider manifest |
-| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | 3,989,205,604 | 4096 | main | **BLOCKED — see below** |
-| `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | 3,990,690,638 | 16384 | main + prefill | **EXPERIMENTAL — see below** |
+| Path | Bytes | Context | Functions |
+|---|---:|---|---|
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,858 | 4096 | main + prefill |
+| `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,830 | 8192 | main + prefill |
+| `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,844 | 16384 | main + prefill |
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | 3,989,205,604 | 4096 | main — **does not load**, see Status |
+| `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | 3,990,690,638 | 16384 | main + prefill |
 
 Each folder holds `<name>.aimodel/` (`main.mlirb` ≈ 3.96 GB, `main.hash`, asset
 `metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
 32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
-18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl`
-(35,045 B).
+18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl` (35,045 B).
+
+**The three `_tbl_pf64` folders hold the same weights and the same graph.** `--max-ctx`
+changes exactly one thing: `language.max_context_length` in the bundle manifest. Function
+signatures, state descriptors and export peak RSS are identical at every value. Pick the
+folder whose manifest integer matches the window you intend to run; a context bump is a
+manifest edit, not a conversion.
 
-**The three `_tbl_pf64` folders are the same weights.** `--max-ctx` does not change the
-exported graph — identical function signatures, identical state descriptors, identical
-export peak RSS at every value. It changes exactly one thing:
-`language.max_context_length` in the bundle manifest. This was proved with a same-config
-control re-export whose `main.mlirb` differed from its twin by *more* bytes than a
-4096→8192 pair does. Pick the folder whose manifest integer matches the window you intend
-to run; a future context bump is a manifest edit, not a conversion.
+**Stop token:** every bundle declares `eos_token = "<turn|>"` (id 106), which is the turn
+terminator Gemma 4 emits. `generation_config.json` independently lists
+`eos_token_id: [1, 106, 50]`. A host that stops on the raw upstream `<eos>` instead will
+overrun every reply.
 
 ### The PLE gather-table sidecar — required, not optional
 
@@ -96,26 +104,22 @@ to run; a future context bump is a manifest edit, not a conversion.
 | `meta.json` | 231 |
 
 Every `_tbl` bundle binds `ple_table` (from `embed_per_layer.i8`) and `ple_scale` (from
-`embed_per_layer.scale.f32`) as **static** graph inputs. Without them the bundle fails to
-load with a bare input-arity error naming `ple_table`/`ple_scale` and saying nothing about
-tables — a diagnosis this project got wrong once already. The sidecar is published here so
-the repo is self-contained for both loading and a pf64 rebuild.
+`embed_per_layer.scale.f32`) as **static** graph inputs.
 
 ## Requirements
 
 - **Apple silicon Mac**, Core AI runtime.
 - **Engine contract: 4 inputs** — `input_ids`, `position_ids`, plus the static `ple_table`
-  and `ple_scale`. Two engines can take that:
+  and `ple_scale`. Two engines accept that:
   - **Pipelined engine** — binds the static inputs zero-copy over the caller's buffer, but
-    **does not support logits**, so no grammar-constrained decoding.
-  - **Sequential engine** — the only logits-capable engine, and therefore the only one
-    guided decoding can run on. It requires the static-input binding that accepts `>= 2`
-    inputs and binds everything beyond `input_ids`/`position_ids` from
-    `EngineOptions.staticInputBuffers`; an extra input with no buffer is rejected by name.
-    It pays a **one-time copy of every constant input at load — 2.69 GB for E4B** — because
-    its submission path materialises a foreign buffer-backed view on every pass otherwise
-    (measured: binding the table as a raw view drove peak footprint to 14.9 GB and
-    segfaulted the runtime mid-prefill).
+    does not support logits, so no grammar-constrained decoding.
+  - **Sequential engine** — the only logits-capable engine, and therefore the only path for
+    guided decoding. It accepts `>= 2` inputs and binds everything beyond
+    `input_ids`/`position_ids` from `EngineOptions.staticInputBuffers`; an extra input with
+    no buffer is rejected by name. It pays a **one-time copy of every constant input at
+    load — 2.69 GB for E4B**, because its submission path otherwise materialises a foreign
+    buffer-backed view on every pass. Binding the table as a raw view instead drives peak
+    footprint to 14.9 GB and faults the runtime mid-prefill.
 - **States:** `keyCache` / `valueCache` `Float16, 24 × 1 × 2 × ? × 512`. Dynamic sequence
   dim → `GrowingKVCache` (initial 256, doubling).
 - **KV cost: 98,304 bytes per token** (fp16) — 403 MB at 4096, 805 MB at 8192, 1.61 GB at
@@ -129,86 +133,96 @@ the repo is self-contained for both loading and a pf64 rebuild.
   | Base | **6.78 GB** |
   | Affordable context, fp16 KV | **~39,900 tokens** |
 
-  So KV was never the ceiling at this tier — 16384 costs 1.61 GB against 3.92 GB of
-  headroom. **The real constraint is the PLE table**, which dwarfs the KV cache at any
-  context worth shipping. Measured peak footprint on a guided run was **10.7 GB**, i.e.
-  right at the working-set edge.
-- **Run one E4B campaign per process.** Wired memory from a Gemma campaign is not reclaimed
-  until the process exits; a second full E4B run in the same session drove wired memory to
-  13.2 GB and left the machine with 63 MB free.
+  KV is not the ceiling at this tier; the PLE table is. Measured peak footprint on a guided
+  run: **9.99 GB**, max RSS 9.63 GB — right at the working-set edge. **Minimum practical
+  machine memory: 16 GB with nothing else resident**, for guided decoding only; 24 GB or
+  more is the comfortable tier. Free-form generation does not fit this envelope on any
+  machine — see the known issue above.
+- **Run one E4B session per process.** Wired memory is not reclaimed until the process
+  exits; a second full run in the same process drives wired memory to 13.2 GB on a 16 GB
+  machine.
 - Sliding-window note: E4B interleaves 20 sliding and 4 full attention layers across 42
   layers collapsed to 24 KV slots. The export models the window in the *mask*, not in
   memory — sliding layers ride full-length slots and zero-pad head_dim 256���512, so 83.3% of
-  KV bytes sit in windowed slots and 41.7% is dead padding. A ring-buffered sliding cache
-  would be ~6× cheaper per token. That is model authoring, not a flag, and it is not needed
-  at this tier.
-- `runtime_env COREAI_CHUNK_THRESHOLD=1` travels with the recipe. (Both engines now derive
-  the static query width from the graph, so the older host-side rules about warmup length
-  and chunk threshold are retired.)
+  KV bytes sit in windowed slots and 41.7% is dead padding.
+- The bundle manifest declares `runtime_env COREAI_CHUNK_THRESHOLD=1`. Both engines derive
+  the static query width from the graph.
 
-## Measured performance
+## Performance
 
-Measured on a **16 GB M2 Pro Mac (macOS 27 beta)** with the out-of-process `coreai-repro`
-harness: 10 real enrichment rows (5 POST, 5 COMMENT), the app's live `app_litert` JSON
-schema, greedy decoding through `respondJSON(to:schema:)`, `reset()` per row, **sequential
-engine with the PLE tables bound as static inputs**, 128-token cap.
+Measured on a **16 GB Apple silicon Mac (M2 Pro, macOS 27 beta)**: 10-sample harness,
+guided JSON-constrained decoding against a fixed schema, greedy, sequential engine with the
+PLE tables bound as static inputs, `reset()` between samples, 128-token cap. Load excluded
+from `s/row`; sample 1 excluded as a cache-warm outlier.
 
 `gemma4_e4b_qat_decode_int4lin_tbl_pf64`:
 
 | | |
 |---|---|
-| Load | 26.1 s |
+| Load | 16.7 s |
 | Guided JSON parse | **10/10** |
 | Enum-clean | **10/10** |
-| s/row | **21.89** (POST) / **12.36** (COMMENT) |
-| Decode | 6.7–15.9 tok/s, mean **10.2** |
-| TTFT | 2.8–9.7 s |
-| Peak footprint | 10.7 GB |
-| Stop token | `<turn\|>`, clean self-stop on every row |
+| s/row (long samples) | **14.70** |
+| s/row (short samples) | **5.93** |
+| Decode | **9.5–26.4 tok/s** (mean 17.8) |
+| TTFT | **1.23–6.51 s** |
+| Peak footprint | 9.99 GB |
+| Max RSS | 9.63 GB |
+| Stop token | `<turn\|>`, clean self-stop on every sample |
+
+E4B is memory-bound at the 16 GB tier, and its per-sample cost is sensitive to what else is
+resident on the machine — expect roughly 2× these figures when the model has to share.
 
 **The grammar is close to free; prefill is the cost.** Decode throughput is essentially
-unchanged from unguided pipelined measurements (9.3–11.0 → 10.2 tok/s). Guided rows cost
-~25% more than unguided ones because the sequential engine's prefill is step-synchronous,
-not because of the constraint.
+unchanged from unguided pipelined measurements. Guided samples cost ~25% more than unguided
+ones because the sequential engine's prefill is step-synchronous, not because of the
+constraint.
+
+**Enum conformance is entirely the grammar's doing.** Unguided, this model emits an
+off-schema enum value in all ten samples. Guided, all ten are correct, because an off-enum
+token is unsamplable.
+
+Published Mac figures of ~55.8 tok/s for E4B decode should be read as larger-machine
+figures.
 
-**Enum conformance is entirely the grammar's doing, and it is decisive.** Unguided, this
-model invented an off-schema `motivation` value in **all ten** rows (`empowerment`,
-`advocacy`, `informative`…, none of them schema members). Guided, all ten are correct — not
-because the model improved, but because an off-enum token is unsamplable.
+### Unguided workload — memory-capped
 
-**Context for the cost.** On the same harness and machine, LFM2.5-1.2B runs 1.77/1.30 s/row
-and Gemma 4 E2B runs 7.06/4.27. This bundle is roughly **12× the 1.2B's cost per row** and
-is memory-bound at 10.7 GB peak. Published Mac figures of ~55.8 tok/s for E4B decode should
-be read as bigger-Mac figures.
+Free-form generation from a fixed prompt. **Every generation length below is a memory cap,
+not a model stop** — see the known issue.
 
-Nothing was measured above 4096 tokens. The `ctx8192`/`ctx16384` folders carry the 4096
-graph, so throughput should be unchanged, but quality at depth is a real open question:
-beyond ~4k the long-range signal rides on 4 full-attention layers, and widening the manifest
-does not widen the sliding layers' 512-token window.
+| depth | prompt tokens | TTFT | decode | generated | wall | peak footprint |
+|---|---|---|---|---|---|---|
+| 3.4k | 3314 | 68.46 s | 0.78 tok/s | 64 (capped) | 151.6 s | **16.91 GB** |
+| 8k | 7907 | 159.1 s | 0.52 tok/s | 32 (capped) | 218.8 s | **31.33 GB** |
 
-## Qualification status
+Output quality up to the cap is sound; the limit is memory, not capability.
+
+**Needle-in-haystack recall at 8k: ≥2/3.** Three distinctive facts planted at 10/50/90% of
+the filler; the 32-token memory cap truncated the answer mid-fact-2, so fact 3 was never
+reachable. 15k was not attempted.
+
+## Status
 
 | Artifact | Status |
 |---|---|
-| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED — measured, viable, not a default.** 10/10 parse, 10/10 enum-clean on the harness above. Slow and memory-bound at the 16 GB tier; it earns a slot only where Gemma's output specifically is wanted. |
-| `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** — same weights, same graph, wider manifest integer. No run above 4096 tokens. |
-| `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** — as above. |
-| `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | **EXPERIMENTAL — built, unmeasured.** Gate: a Mac-side oracle/parity check plus a device benchmark, neither of which has been run. Do not route production traffic to it. |
-| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | **BLOCKED — does not load on any current code path.** |
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED FOR GUIDED GENERATION** — measured: 10/10 parse, 10/10 enum-clean, 14.70/5.93 s/row, 9.5–26.4 tok/s, 9.99 GB peak footprint. Not qualified for free-form generation on the current reference engine implementation. |
+| `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE** — same weights and graph, wider manifest integer. Free-form at this depth is measured-unsafe (31.33 GB under a 32-token cap). |
+| `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE** — as above; free-form at this depth was not attempted. |
+| `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | **EXPERIMENTAL** — built, unmeasured. Gate: a Mac-side oracle/parity check plus a device benchmark. |
+| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | **BLOCKED** — does not load on any current code path. |
 
 ### About the `w4a8` bundle
 
 Same QAT int4 per-block-32 weights and the same graph as
 `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64`; the difference is an int8
 quantize/dequantize pair on the inputs of every `F.linear`, calibrated on 128 synthetic
-social-media-style samples (the corpus ships in the folder). It was built as a prefill/TTFT
-lever, and its manifest context is 16384.
+samples (the corpus ships in the folder). Manifest context 16384.
 
-The caveat is load-bearing and comes from the compression toolchain's own documentation: the
-runtime fast path is `W_INT8 × A_INT8`, and a float weight path executes in floating point
-*regardless* of activation quantization. These weights use int4 scale-multiply dequant, not
-an int8 LUT, so the expected TTFT gain may be **zero or negative**. This bundle exists to be
-measured, not because a win is predicted.
+The caveat comes from the compression toolchain's own documentation: the runtime fast path
+is `W_INT8 × A_INT8`, and a float weight path executes in floating point *regardless* of
+activation quantization. These weights use int4 scale-multiply dequant, not an int8 LUT, so
+the expected TTFT gain may be zero or negative. This bundle exists to be measured, not
+because a win is predicted.
 
 ### Why the non-`tbl` decode bundle is blocked
 
@@ -219,43 +233,16 @@ LOAD FAILED: invalidInputType("Expected 2 inputs, got 3:
   [\"input_ids\", \"position_ids\", \"ple_tokens\"]")
 ```
 
-`ple_tokens`, not `ple_table`/`ple_scale`. The Gemma architecture description in the kit
-knows only the pair (`ple_table` → `embed_per_layer.i8`, `ple_scale` →
-`embed_per_layer.scale.f32`), so **no code path binds `ple_tokens`** and the table sidecar
-in this repo does not help. It is unusable as exported. The fix is a re-export with `--tbl`
-so E4B decode and prefill are the same graph family, not an extension to the kit; until
-then, use the `_tbl_pf64` bundle, which is a strict superset (it carries both `main` and
-`prefill`).
-
-It is kept published rather than deleted because it is a fingerprinted, immutable artifact
-that earlier revisions pin, and because the failure it demonstrates is worth documenting.
-
-## Verification
-
-- **Producer fingerprint:** every inner `<name>.aimodel/metadata.json` reads
-  `producer: "coreai-core 1.0.0b2"`. `coreai-torch 0.4.0` / `coreai-core 1.0.0b1` produce
-  ODIE-poisoned bundles that abort **in-process** at load, taking the host application down
-  uncatchably. The field is in the **inner asset** metadata; the bundle manifest never
-  carries a `producer` field, so checking the manifest condemns every good bundle.
-- **Stop token, verified per folder:** all four declare `eos_token = "<turn|>"`. Gemma 4
-  ends a *turn* with `<turn|>` (id 106); the raw upstream `<eos>` ends the whole sequence,
-  and a host stopping on it overruns every reply. The exporter copies the upstream tokenizer
-  config verbatim and does **not** apply this correction — it is a post-export patch, and it
-  was checked on these bundles rather than assumed. `generation_config.json` independently
-  lists `eos_token_id: [1, 106, 50]`.
-- **`pf64` is worth 7.7× on time-to-first-token, measured.** The A/B was run on the E2B
-  sibling (same prompt, same weights, differing only in whether the `prefill` function was
-  exported): TTFT 35.74 s → 4.65 s, s/row 45.04 → 8.83. The non-`pf64` variant also failed
-  partway through a ten-row campaign with the runtime's footprint ballooning past 37 GB
-  across ~2,800 single-token passes — it is not merely slower, it is not durable. Prefer
-  `_pf64` everywhere; that is why no plain `_tbl` E4B bundle exists here.
-- **Determinism:** `.aimodel` conversion is not byte-reproducible; the same command twice on
-  one box gives different `main.mlirb` bytes. Integrity rests on per-file SHA-256 of the
-  exact published artifact, never on rebuilding.
-- **No oracle gate has been run on this model.** The 10/10 results above are behavioural
-  (parse rate, enum conformance, clean stop) on Visible's own task. A decode oracle against
-  an fp32 reference — the gate that would catch an int4 or kernel-level numerics error — has
-  not been run on any Gemma 4 bundle in this catalog.
+`ple_tokens`, not `ple_table`/`ple_scale`. The Gemma architecture description in the
+runtime knows only the pair (`ple_table` → `embed_per_layer.i8`, `ple_scale` →
+`embed_per_layer.scale.f32`), so no code path binds `ple_tokens` and the table sidecar does
+not help. Use the `_tbl_pf64` bundle, which is a strict superset (it carries both `main`
+and `prefill`). The bundle is retained because it is an immutable artifact that earlier
+revisions pin.
+
+**No numerics gate has been run on any bundle in this repo.** The 10/10 results are
+behavioural (parse rate, enum conformance, clean stop); a decode oracle against an fp32
+reference has not been run.
 
 ## License
 
@@ -266,9 +253,4 @@ model and **the Gemma terms apply downstream**, which is why this repo declares
 [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy) and the
 [Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
 travel with any redistribution of these bundles, including the gather-table sidecar, which
-is derived from the same weights. Visible's contribution is the conversion, not the weights.
-
----
-
-*Additional variants may land in this repo after this card was written; the tables above
-describe its contents as listed at the time of writing.*
+is derived from the same weights. The contribution here is the conversion, not the weights.