--- license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license base_model: google/gemma-4-E4B-it-qat-q4_0-unquantized base_model_relation: quantized library_name: coreai pipeline_tag: text-generation tags: - core-ai - aimodel - apple-silicon - on-device - coreai-kit - quantized - int4 - qat - gemma4 --- # Gemma 4 E4B — Core AI (.aimodel) `google/gemma-4-E4B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for Apple silicon by [visible-cx](https://huggingface.co/visible-cx). These are derivative artifacts: Google's QAT-trained weights rounded onto the int4 grid they were trained for and re-expressed as a Core AI graph. They load through Core AI on macOS and are not usable by PyTorch, GGUF or MLX. Gemma 4 E4B uses **Per-Layer Embeddings**, so the working bundles take a large embedding gather table as a *static graph input* rather than carrying it in the graph. That table ships in `ios-frontend/` and **the bundles do not load without it**; a missing table produces a bare input-arity error naming `ple_table`/`ple_scale`. It is 3.60 GB and it is part of the model's memory cost, not a sidecar you can ignore. > ⚠️ **This model does not fit a 16 GB Mac under an honest memory gate.** Charged for both > artifacts it loads, at the coefficient measured on its E2B sibling, E4B asks **8.99 GiB** — > and on the reference 16 GB machine the check refused it, short by 3.20 GiB. It has never been > run under a watchdog. See [Measurements](#measurements). > ⚠️ **Known issue — memory growth per generated token.** The Core AI runtime caches one > compiled specialization per input-shape signature, and this export makes sequence length > load-bearing: every generated token mints a new signature, retaining roughly **81 MB of GPU > allocations per generated token** until the process exits. **Guided decoding does not protect > you — shape reuse does.** Repeating a length you have already run replays from cache at no > cost, so a fixed-schema, fixed-cap extraction workload stays bounded; long or variable-length > generation grows without bound, and it is not fixable below a re-export. `stable/`, `ring/` > and `ring3/` in this repo are those re-exports. ## Contents ### Production bundles | Path | Bytes | Context | Functions | Status | |---|---:|---|---|---| | `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,858 | 4096 | main + prefill | **QUALIFIED (guided / bounded shapes)** | | `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | 3,990,690,638 | 16384 | main + prefill | EXPERIMENTAL | Each folder holds `.aimodel/` (`main.mlirb` ≈ 3.96 GB, `main.hash`, asset `metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json` 32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja` 18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl` (35,045 B). ### The PLE gather-table sidecar — required, not optional | Path | Files | Bytes | |---|---|---:| | `ios-frontend/gemma4_e4b_qat_gather_raw/` | 7 | 3,601,859,815 | | File | Bytes | |---|---:| | `embed_per_layer.i8` | 2,818,572,288 | | `embed_tokens.i8` | 671,088,640 | | `proj.f32` | 110,100,480 | | `embed_per_layer.scale.f32` | 1,048,576 | | `embed_tokens.scale.f32` | 1,048,576 | | `proj_norm.f32` | 1,024 | | `meta.json` | 231 | Every `_tbl` bundle binds `ple_table` (from `embed_per_layer.i8`) and `ple_scale` (from `embed_per_layer.scale.f32`) as **static** graph inputs. **A QAT bundle must be paired with the QAT tables.** ### Shape-stable re-exports | Path | Bundle | Bytes | Context | |---|---|---:|---| | `stable/gpu-pipelined/` | `…_tbl_pf64_stable_c16384` | 3,989,939,812 | 16384 | | `ring/gpu-pipelined/` | `…_tbl_pf64_ring_c16384` | 3,989,979,750 | 16384 | | `ring3/gpu-pipelined/` | `…_tbl_pf64_ring_c4096_gqafull` | 3,989,929,508 | 4096 | **Stop token:** every bundle declares `eos_token = ""` (id 106), the turn terminator Gemma 4 emits. `generation_config.json` independently lists `eos_token_id: [1, 106, 50]`. A host that stops on the raw upstream `` will overrun every reply. ## Provenance | | | |---|---| | Base checkpoint | `google/gemma-4-E4B-it-qat-q4_0-unquantized` (ungated) | | Zoo recipe | `gemma-4-e4b`, `status = "verified"` — `int4lin` | | Recipe (pf64) | `export_gemma4_pf_pipelined.py --pf 64` with `--tbl` and `--raw-dir` pointed at the gather table above | | Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay | | Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` | | Producer fingerprint | `coreai-core 1.0.0b2` on every inner asset `metadata.json` | | Weight format | **int4, per-block-32** (`int4lin`, symmetric-with-clipping) — the ggml q4_0 grid the QAT checkpoint was trained on | | Vocab | 262,144 | | Export functions | `main` (S=1 decode) and, in `_pf64` bundles, `prefill` (S=64 chunked prefill) | "QAT-unquantized" means QAT-*trained*, stored full width; the int4 rounding happens at export onto the grid training already targeted. `_tbl` = the PLE gather table is bound as a static graph input. `_pf64` = a second entrypoint, `prefill`, with a static query width of 64 (`function_map: {"main": ["main", "prefill"]}`). The `gpu-pipelined/` bundle mirrors the zoo's verified `gemma-4-e4b` recipe with the `pf64` multifunction addition; the gather table follows the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) / [mlboydaisuke](https://huggingface.co/mlboydaisuke) artifact layout. ## Requirements - **Apple silicon Mac**, Core AI runtime. Practically, **24 GB or more** — see the arithmetic below. - **Engine contract: 4 inputs** — `input_ids`, `position_ids`, plus static `ple_table` and `ple_scale`: - **Pipelined engine** — binds the statics zero-copy over the caller's buffer, but does not expose logits, so no grammar-constrained decoding. This is CoreAIKit's default for `_tbl` bundles. - **Sequential engine** — the only logits-capable engine, and therefore the only path for guided decoding. It binds everything beyond `input_ids`/`position_ids` from `EngineOptions.staticInputBuffers`, and **a caller that does not supply those buffers gets a load failure naming them**, not a fallback. It pays a **one-time copy of every constant input at load — 2.69 GB for E4B**; binding the table as a raw view instead drives peak footprint to 14.9 GB and faults the runtime mid-prefill. - **States:** `keyCache` / `valueCache` `Float16, 24 × 1 × 2 × ? × 512`. Dynamic sequence dim → `GrowingKVCache` (initial 256, doubling). - **KV cost: 98,304 bytes per token** (fp16) — 403 MB at 4096, 1.61 GB at 16384. **KV is not the ceiling at this tier; the PLE table is.** - **Run one E4B session per process.** Wired memory is not reclaimed until the process exits unless the host releases the inference function; a second full run in the same process drove wired memory to 13.2 GB on a 16 GB machine. - Sliding-window note: E4B interleaves sliding and full attention layers across 42 layers collapsed to 24 KV slots. The export models the window in the *mask*, not in memory — sliding layers ride full-length slots and zero-pad head_dim 256→512, so 83.3% of KV bytes sit in windowed slots and 41.7% is dead padding. - The bundle manifest declares `runtime_env COREAI_CHUNK_THRESHOLD=1`. Both engines derive the static query width from the graph. ## Measurements Measured on a **16 GB Apple silicon Mac (M2 Pro, macOS 27 beta)** unless stated. ### The memory verdict — arithmetic, not a run **E4B has never been run under a watchdog.** What exists is its artifact inventory charged at the coefficient measured on its E2B sibling (n=2 completed runs, `wired ÷ (blob + PLE tables)` = **1.101**): ``` compiled blob 3,955,446,640 B = 3.6838 GiB PLE tables 3,601,859,584 B = 3.3545 GiB ------------------------------------------------ artifacts = 7.0383 GiB x 1.10 (measured on E2B) = 7.742 GiB + 1.25 GiB in-flight floor = 8.99 GiB required reclaimable at the check = 5.79 GiB -> REFUSED, short by 3.20 GiB ``` (The PLE figure is the binary payload the runtime binds; the published folder is 231 bytes larger because it also carries `meta.json`.) This is a **3.67 GiB correction against the gate this project previously shipped**, which charged `blob × 1.106 + 1.25` = 5.324 GiB and would have admitted the model. The cause is the same one-line error in both directions across the E-series: **the runtime loads two artifacts and the law was only ever told about the first.** (E2B carried the same defect at 2.87 GiB.) Stated honestly: **the refusal is arithmetic on E2B's coefficient, not a measurement of E4B.** For scale, the compile-only readings that do exist for this bundle: bundle 3.716 GiB, compiled blob **3.684 GiB**, graph constant **−0.002 GiB** (Gemma's compiled artifact is very slightly *smaller* than its bundle, unlike the LFMs at 1.18–1.34×). ### Guided structured output 10-sample harness, guided JSON-constrained decoding against a fixed schema, greedy, **sequential engine** with the PLE tables bound as static inputs, `reset()` between samples, 128-token cap. | | `gpu-pipelined/…_tbl_pf64` | |---|---| | Load | 16.7 s | | Guided JSON parse | **10/10** | | Enum-clean | **10/10** | | s/row (long samples) | **14.70** | | s/row (short samples) | **5.93** | | Decode | **9.5–26.4 tok/s** (mean 17.8) | | TTFT | **1.23–6.51 s** | | Peak footprint | 9.99 GB | | Max RSS | 9.63 GB | | Stop | ``, clean self-stop on every sample | E4B is memory-bound at the 16 GB tier and its per-sample cost is sensitive to what else is resident — expect roughly 2× these figures when the model has to share. Note the spread in the decode column: that variance *is* the memory pressure. **The grammar is close to free; prefill is the cost.** Decode throughput is essentially unchanged from unguided pipelined measurements; guided samples cost ~25% more because the sequential engine's prefill is step-synchronous, not because of the constraint. **Enum conformance is entirely the grammar's doing.** Unguided, this model emits an off-schema enum value in all ten samples. Guided, all ten are correct, because an off-enum token is unsamplable. Published Mac figures of ~55.8 tok/s for E4B decode should be read as larger-machine figures. ### Unguided workload — memory-capped Free-form generation from a fixed prompt. **Every generation length below is a memory cap, not a model stop** — see the known issue. | depth | prompt tokens | TTFT | decode | generated | wall | peak footprint | |---|---|---|---|---|---|---| | 3.4k | 3,314 | 68.46 s | 0.78 tok/s | 64 (capped) | 151.6 s | **16.91 GB** | | 8k | 7,907 | 159.1 s | 0.52 tok/s | 32 (capped) | 218.8 s | **31.33 GB** | Output quality up to the cap is sound; the limit is memory, not capability. **Needle-in-haystack recall at 8k: ≥2/3.** Three distinctive facts planted at 10/50/90% of the filler; the 32-token memory cap truncated the answer mid-fact-2, so fact 3 was never reachable. 15k was not attempted. ### What this model is and is not, on this stack The shape-signature growth sets a ceiling no cap works around: at ~81 MB/token over a ~9.6 GB intercept, a 900-token report costs ~54 GB and a 600-token insight card ~58 GB. The caps that do survive are shorter than a single card. **Gemma 4 is an enrichment model on this stack — bounded, repeating shapes — and it is not a report, RAG or long-insight model. The blocker is the export, not the weights.** ## Usage Swift Package Manager, via [CoreAIKit](https://github.com/john-rocky/coreai-kit) — a community package, not affiliated with Apple, requiring macOS 27 beta: ```swift .package(url: "https://github.com/john-rocky/coreai-kit", branch: "main") // target dependency: .product(name: "CoreAIKit", package: "coreai-kit") ``` An E-series bundle is **two downloads**, the decoder and its paired PLE tables, addressed as two paths inside this repo: ```swift import CoreAIKit let store = ModelStore.default let decoderURL = try await store.download( ModelID("visible-cx/gemma-4-E4B-CoreAI", path: "gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64")) let tablesURL = try await store.download( ModelID("visible-cx/gemma-4-E4B-CoreAI", path: "ios-frontend/gemma4_e4b_qat_gather_raw")) // engineVariant defaults to .pipelined here, which is the supported path for _tbl bundles. let runtime = try await GemmaRuntime( decoderBundleAt: decoderURL, tablesAt: tablesURL, arch: .gemma4) ``` Notes that are not optional: - **Do not pair a QAT bundle with non-QAT tables.** - **Do not enable chunked prefill.** The `…_tbl` graph is S=1 on `main`; `GemmaRuntime` sets `COREAI_CHUNK_THRESHOLD=1` for you if it is unset. Leave it. - **Guided decoding needs the sequential engine**, and the sequential engine needs `ple_table`/`ple_scale` supplied through `EngineOptions.staticInputBuffers` — it will refuse by name otherwise. - **Budget 7.04 GiB of artifacts plus the in-flight floor before you start**, and release the inference function between sessions. - Pass `revision:` a Hub commit hash to pin immutable bytes. ## Integrity Core AI `.aimodel` bundles are **not byte-reproducible**: the exporter is not deterministic even against itself. Verify by digesting the exact published bytes rather than by rebuilding. Every bundle carries `main.hash`, the raw 32 bytes of `sha256(main.mlirb)`; on the Hub the same value is recoverable from the LFS oid without fetching the file. ## Status | Artifact | Status | |---|---| | `gpu-pipelined/…_tbl_pf64` (ctx 4096) | **QUALIFIED FOR BOUNDED-SHAPE GENERATION, ON A MACHINE THAT FITS IT** — measured: 10/10 parse, 10/10 enum-clean, 14.70/5.93 s/row, 9.5–26.4 tok/s, 9.99 GB peak footprint. Not qualified for free-form generation. Under the corrected two-artifact gate it is refused on 16 GB; the guided figures above were taken before that gate existed and are a real run on a machine that was over-committed. | | `w4a8/…_a8_tbl_pf64` | **EXPERIMENTAL** — built, unmeasured. Same int4 weights and graph; the difference is an int8 quantize/dequantize pair on the inputs of every `F.linear`, calibrated on 128 synthetic samples (corpus ships in the folder). The compression toolchain's own documentation says the runtime fast path is `W_INT8 × A_INT8` and a float weight path executes in floating point regardless of activation quantization; these weights use int4 scale-multiply dequant, so the expected TTFT gain may be zero or negative. It exists to be measured, not because a win is predicted. | | `stable/…_stable_c16384` | **EXPERIMENTAL** — shape-stable decode contract, gated in torch, never run on a Mac. On the E2B sibling the memory fix is confirmed and decode falls to ~1.0 tok/s from the full-capacity cache read; the same capacity cost applies here. A working proof of the contract, not a deployable bundle. | | `ring/…_ring_c16384`, `ring3/…_ring_c4096_gqafull` | **EXPERIMENTAL** — gated in torch, no on-device numbers. `ring/`'s E4B was never measured on a Mac. | **No numerics gate has been run on device for any bundle in this repo.** The 10/10 results are behavioural (parse rate, enum conformance, clean stop); a decode oracle against an fp32 reference has not been run. ## The shape-stable family, in detail All three re-export the same weights and the same quantization onto a contract in which **no input shape moves between steps**, removing the per-generated-token growth by design. They need a host that feeds `position_ids` as the **absolute positions of the S new tokens only**; a host that feeds the growing `0..N` prefix will write the KV cache at the wrong offset. **Contract** (identical across all three): ``` main IN input_ids Int32 1x1 | position_ids Int32 1x1 | ple_table Int8 V x (L*ld) | ple_scale Float32 V prefill IN input_ids Int32 1x64 | position_ids Int32 1x64 | (same statics) ST keyCache / valueCache Float16, literal extents OUT logits Float16 1 x S x 262144 ``` `position_ids[0,0]` is also the cache slot the K/V for those tokens is written at. The mask — causality, unwritten slots and the sliding window — is derived inside the graph from those positions, so there is no mask input to bind. The context ceiling is **encoded in the graph**: a different window needs a different export, not a manifest edit. `stable/` gave every layer a full 16,384-slot cache and read all of it every step, which is where the ~1 tok/s comes from. `ring/` keeps the contract byte for byte and changes what sits behind it: sliding layers read a **576-slot ring** (the 512-token window plus one 64-token prefill chunk), and the grouped-query head expansion is folded into the query rather than materialised. `ring3/` folds that expansion **only in the full-attention layers**, where the key length is long enough for the trade to pay, and keeps the stock gathered form in the short-key sliding layers — E4B runs 35 sliding and 7 full attention layers, so the split matters more here than on E2B. | | E4B `stable/` | E4B `ring/` | E4B `ring3/` (c4096) | |---|---|---|---| | KV state shape | `[…, 16384, 512]` ×2 | `[1, 1, 2, 77056, 512]` ×2 | `[1, 1, 2, 27904, 512]` ×2 | | KV bytes, both states | 1.61 GB | **315.6 MB** | **114.3 MB** | | cache slots read per decode step | 393,216 | 77,056 | — | | sliding / full layers | — | 20 × 576 + 4 × 16,384 | 20 × 576 + 4 × 4,096 | | dynamic dimensions | none | none | none | **One host precondition new in `ring/` and `ring3/`:** a `prefill` call's first position must be a multiple of 64. The ring write is a fixed-width store at `p0 % 576`, and 576 is nine 64-token chunks, so a 64-aligned chunk can never straddle the wrap. `stable/` tolerated an unaligned chunk; the ring bundles do not. **Equivalence gates**, all run in eager torch against the shipped graph before conversion: `stable/` — greedy argmax identical on all tested decode steps including past the sliding window, worst logits cosine 0.99999 (fp16), prefill bit-identical in fp32. `ring*/` — 1,600 prompt tokens as 25 chunks of 64 then 32 greedy decode steps, final position 1,631 so the ring wraps about three times, fp32, 57 comparisons per run: **0 argmax mismatches**. No dynamic dimension appears in either entrypoint of any of them. ## License Google publishes the upstream QAT checkpoint under **Apache-2.0** with a `license_link` to the [Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license), and this repo mirrors that declaration. Use is governed by those terms and by the [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy); the obligations travel with any redistribution of these bundles, **including the gather-table sidecar**, which is derived from the same weights. The contribution here is the conversion, not the weights. ## levered2/ — gather-first embedding, int4 PLE, fp16 head The Speed-tier build. Shape-stable windowed KV, ring sliding cache, length-bounded Metal SDPA, cap 131,072 — plus three changes to what the vocab path costs, each measured on the compiled graph rather than assumed. **The embedding is gathered before it is dequantized.** The previous build compiled to mps.dequantize(si4) -> tensor<262144x2560xf16> 1.342 GB, every pass mps.gather_nd(that, ids) -> tensor<1x1x2560xf16> ONE ROW so the whole table was reconstructed in fp16 to fetch 2,560 values and the int4 saving evaporated at run time. This revision packs two 4-bit codes per byte and gathers the PACKED row. Verified on the exported graph — every 262144-row gather is now over an integer table: gather_along_axis(262144x1280xsi8) -> 1x1280xsi8 embedding gather_along_axis(262144x80xf16) -> 1x80xf16 its block scales gather_along_axis(262144x5376xsi8) -> 1x5376xsi8 PLE table gather_along_axis(262144x336xf16) -> 1x336xf16 PLE scales **The PLE table is 4-bit**, per-block-32, and gated: 42 layers, 256 teacher-forced positions, against the checkpoint's own bf16 rows — **32.936 dB, 255/256 top-1, reference greedy token in the top-5 on every position**, for 1.4766 GiB against int8's 2.6260. **The head is fp16.** The int4 head reconstructed a full `[262144, 2560]` fp16 table into its matmul. Element accounting across the two compiled assets: `Int4` −691,200,000 elements, `Float16` **+671,088,640 = 262,144 x 2,560 exactly**, and `blockwise_shift_scale` **688 -> 686** — one dequantize removed per entrypoint. The matmul now reads the constant out of the blob. The embedding's own lineage gate, same method: **37.215 dB, 256/256 top-1**. blob 3,957,723,854 + PLE 1,585,446,912 = 5,543,170,766 artifacts ## PAIR IT WITH THE TABLES IN THIS FOLDER `levered2/ios-frontend/gemma4_e4b_qat_gather_raw/` — `embed_per_layer.i4` (packed nibbles, `+8` biased) and `embed_per_layer.scale.f16` (per-32-block scales, rows stored 352 wide for the engine's 32-element row stride, the trailing 16 unread). **The repo's int8 `ios-frontend/` tables will not work**: the 4-bit unpack is compiled into this graph. **Not yet run on a Mac.** No token generated, no decode speed claimed. Requires a kit with `coreai-kit-gemma-int4-ple.patch` and an engine with the stable-position-contract and sequential n-state patches. ### `levered2/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64_ple4_pe4_hf16_msdpa_g8_wkv_stable_c131072` — 3,989,920,339 bytes export_report.json 2,324 B 96b34dcca50696b8b2526e99dae7afea0936dad3ecdceb24f4a3d0bb883f6f26 gemma4_e4b_qat_decode_int4lin_tbl_pf64_ple4_pe4_hf16_msdpa_g8_wkv_stable_c131072.aimodel/main.hash 32 B 568b0927d32636eeff95623c62a9b6bb207dc71b35c1f1c72302cd68cb5f01f3 gemma4_e4b_qat_decode_int4lin_tbl_pf64_ple4_pe4_hf16_msdpa_g8_wkv_stable_c131072.aimodel/main.mlirb 3,957,723,854 B c60c98a02f57790a8c7a527df6b91ba6c1b625d14d7ebd78640b52280cc52447 gemma4_e4b_qat_decode_int4lin_tbl_pf64_ple4_pe4_hf16_msdpa_g8_wkv_stable_c131072.aimodel/metadata.json 105 B c8e32c43e985105d978b858cf336cf8d121d8d052657bbed8f1036df259f4365 metadata.json 1,897 B f374ad6d7c79dd1941769a7d9271b6b05175e7616eb03f490fa2f2fa8b377aaf tokenizer/chat_template.jinja 18,569 B 0a2c8073c878ab1da004bee933a998606537bbb62016310352c7285c3f01c5b5 tokenizer/generation_config.json 203 B b69207f9be617e982d13cc273cce6fd88c98dda99a4bdc5e2d52ffe0a0d9f0a9 tokenizer/tokenizer.json 32,169,626 B cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f tokenizer/tokenizer_config.json 3,729 B 3ab5c7b94dc97d65ca7064496fa69b88ff875378e1cb7ee3e43070c3a8170999 ### `levered2/ios-frontend/gemma4_e4b_qat_gather_raw` — 1,593,838,213 bytes embed_per_layer.i4 1,409,286,144 B 5651f14b2a4cead5619da219aafceaa96e78752af9c8bb2eb11e2230310839fa embed_per_layer.scale.f16 184,549,376 B a4b621325c55f5dcc60097178491262539ccd2458ca5ae8cb8caef398806e6ba meta.json 721 B 9f225f51dc810bbb5ec91af8c28482ecbd56229a4979ca0bd1d6ad1189909b84 quant_report.json 1,972 B bf0ca04abebe20e729fede1c7c34e4fba2b2489c2f1204b784609fbb0b8767a6