visible-cx
/

gemma4-26b-a4b-CoreAI

@@ -1,7 +1,8 @@
 ---
-license: gemma
-license_link: https://ai.google.dev/gemma/terms
 base_model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
 library_name: coreai
 pipeline_tag: text-generation
 tags:
@@ -9,6 +10,7 @@ tags:
   - aimodel
   - apple-silicon
   - on-device
   - quantized
   - int4
   - qat
@@ -19,30 +21,61 @@ tags:
 # Gemma 4 26B-A4B (MoE) — Core AI (.aimodel)
-`google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` converted to a Core AI `.aimodel` bundle
-for Apple silicon by [visible-cx](https://huggingface.co/visible-cx). This is a derivative
-artifact: Google's QAT-trained weights rounded onto int4 and re-expressed as a Core AI
-graph, with the 128-expert sparse branch of every layer lowered onto a Metal gather kernel.
 It is a **sparse mixture-of-experts** model — 26.5B total parameters, roughly 4B active per
-token.
-> ⚠️ **UNQUALIFIED.** This bundle has never produced a token, on any machine. The graph
-> exports and the producer fingerprint passes; the block wiring is verified against the
-> reference implementation (below). Nothing else is established.
 >
 > ⚠️ **The 16 GB tier is excluded.** Weights are ~17.6 GB resident against a 16 GB Mac's
-> ~10.7 GB Metal working set. **Minimum practical machine memory: 24–32 GB.**
-> ⚠️ **Expected known issue — long unguided generations (unverified on this model).**
-> On the sibling E2B/E4B exports, the Core AI runtime retains one compiled
-> specialization per sequence-length signature, costing roughly 80 MB of GPU
-> allocations per generated token until process exit; the decode graphs in this
-> repository share the length-dependent structure that causes it. Until a
-> shape-stable re-export exists for this model, treat these bundles as suitable for
-> guided/constrained and short generations; budget memory for long free-form runs
-> accordingly. Measured details: the gemma-4-E2B-CoreAI card's known-issue section.
 ## Provenance
@@ -54,16 +87,16 @@ token.
 | Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay, plus three new modules (below) |
 | Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` |
 | Producer fingerprint | `coreai-core 1.0.0b2` — verified on the inner asset `metadata.json` |
-| Asset creation date | `20260817T215629Z` |
 | Vocab | 262,144 |
-| Export functions | `main` (S=1 decode) only |
-"QAT-unquantized" means QAT-*trained*, stored bf16; the int4 rounding happens at export
-onto the ggml q4_0 grid the training already targeted. There is no separate pre-quantised
-int4 artifact to prefer.
-**To reproduce this conversion you need a ≥128 GB host**: the export peaks at
-**102.25 GB RSS** (`VmHWM`) on Linux x86_64.
 ### Architecture
@@ -79,11 +112,10 @@ int4 artifact to prefer.
 | `head_dim` / `global_head_dim` | **256 / 512** | 256 / 512 | 256 / 512 |
 | `sliding_window` | **1024** | 1024 | 512 |
 | Per-Layer Embeddings | **none** | none | 256 |
-| KV-shared layers | **0** | 0 | 18 |
-Same attention family as the dense 12B/31B — same dual head_dim, same `attention_k_eq_v`
-(full layers carry no `v_proj`), same dual RoPE, same softcap. There are no Per-Layer
-Embeddings and therefore no gather-table sidecar in this repo.
 Every layer carries a dense MLP **and** a sparse branch, in parallel:
@@ -98,11 +130,11 @@ x  = residual + post_feedforward_layernorm(h1 + h2)
 x  = x * layer_scalar
 ```
-Two details are load-bearing: **the router reads the un-normalised residual**, and **the
-experts are GELU-gated** (`gelu_pytorch_tanh`), where the `SwitchGLU` primitive's default
-is SiLU. The router itself: scale-free RMSNorm → `× scale × hidden**-0.5` → linear to 128 →
-fp32 softmax over all experts → top-8 → renormalise to sum 1 → multiply by a learned
-`per_expert_scale` gathered at the selected ids.
 ### Compression recipe
@@ -113,271 +145,227 @@ fp32 softmax over all experts → top-8 → renormalise to sum 1 → multiply by
 | **Router** (`proj`, `scale`, `per_expert_scale`) | **fp16**, excluded by name | routers are the quantization-sensitive part of an MoE and the whole set is 0.02 GB |
 | `embed_tokens` (in-graph) | fp16 | the gather stays exact; the head is untied and quantized separately |
-`sym8` on the experts would put them at 22.8 GB before anything else, so int4 is not
-optional. `aff4` was chosen over a k-means palette (`km4`) because this checkpoint was
-QAT-trained on per-block-32 q4_0 (`w = (q-8)·d`, 16 uniform levels), and an affine int4
-block-32 grid represents that **exactly** (`scale = d`, `bias = -8d`). A palette fits one
-16-entry codebook per 32 output rows across the whole K axis and structurally cannot follow
-per-block QAT scales. The price is aux bytes: `aff4` stores an fp16 scale *and* bias per
-32-element K block, i.e. 5 bits/param effective against `km4`'s ~4.03 — about 3 GB of the
-bundle.
-The `down` projection's K of 704 is padded to 768 by the kernel's automatic `_kpad256`
-treatment — 64 zero columns, ~0.5% wasted expert bytes. `gate`/`up` need no padding.
-**The gather kernel is load-bearing.** Without `MetalSwitchGLU`, the sparse branch lowers
-to a dense matmul that reads all 128 experts every token — a 16× over-read.
 ### New source modules
-Three files were authored for this port; nothing in the shared toolchain was modified:
 | File | Contents |
 |---|---|
 | `coreai_models/models/macos/gemma4_moe_text.py` | `Gemma4MoeConfig`, `Gemma4MoeRouter`, `GeluGLU`, `Gemma4MoeDecoderLayer`, `Gemma4MoeForCausalLM`, and a loader that splits the checkpoint's fused `gate_up_proj [E, 2*704, 2816]` into the `SwitchGLU`'s separate stacks |
 | `coreai_models/models/macos/gemma4_moe_pipelined.py` | `Gemma4MoePipelinedForCausalLM` — subclasses the dense pipelined core and overrides one method; the attention half is inherited unchanged |
 | `coreai-model-zoo/conversion/export_gemma4_moe_decode_pipelined.py` | the recipe |
-## Contents
-| Path | Bytes | Files | Manifest context | Functions |
-|---|---:|---|---|---|
-| `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8` | 17,580,059,414 | 8 | 16384 | main (decode only) |
-| File | Bytes |
-|---|---:|
-| `…aimodel/main.mlirb` | 17,547,866,346 |
-| `…aimodel/main.hash` | 32 |
-| `…aimodel/metadata.json` | 105 |
-| `metadata.json` (bundle manifest) | 689 |
-| `tokenizer/tokenizer.json` | 32,169,626 |
-| `tokenizer/chat_template.jinja` | 18,683 |
-| `tokenizer/tokenizer_config.json` | 3,730 |
-| `tokenizer/generation_config.json` | 203 |
-Manifest context is 16384. `--max-ctx` sets `language.max_context_length` and nothing else;
-lower it if the machine cannot afford the KV — that is a metadata edit, not a re-export.
-**Stop token:** `eos_token = "<turn|>"` (id 106), the turn terminator Gemma 4 emits, applied
-by the export script itself. A host that stops on the raw upstream `<eos>` will overrun
-every reply.
-### Decode only, and a prefill variant is not deliverable by a flag
-There is no `prefill` function. Two independent kernel blockers:
-1. `--metal-sdpa`'s flash-decode kernel is structurally q=1 with no causal mask. The full
-   attention layers need that kernel for the same scratch-heap reason the dense 12B/31B do.
-2. `MetalSwitchGLU.forward` is itself decode-only — it asserts a token batch of 1 and
-   expands the single activation row across the routed slots. A `BatchedMetalSwitchGLU`
-   (sort-by-expert grouped GEMM) exists for q>1, so this half is reachable, but it has
-   never been exported for this model.
-A prefill variant therefore needs a new flash-**prefill** Metal kernel plus the first
-export of the batched MoE kernel, plus a numerics gate for each. Prompts are prefilled one
-token at a time.
-## Requirements
-- **Apple silicon Mac with ≥24–32 GB unified memory**, Core AI runtime.
-- **Engine contract: 2 inputs** (`input_ids`, `position_ids`) → logits, one growing KV pair,
-  no static inputs and no per-step mask.
-- **States:** `keyCache` / `valueCache` `Float16, 30 × 1 × 8 × ? × 512` — one growing pair,
-  30 slots (no KV sharing). The dynamic sequence dim means the runtime resolves
-  `GrowingKVCache`, not a static allocation at the manifest maximum.
-- **KV cost: 491,520 bytes per token of context** (fp16):
-  | Context | KV |
-  |---|---|
-  | 4096 | 2.01 GB |
-  | 8192 | 4.03 GB |
-  | **16384** | **8.05 GB** |
-  | 32768 | 16.1 GB |
-- **Resident weights, ~17.6 GB:**
-  | | bytes |
-  |---|---|
-  | Routed experts, `aff4` (incl. `_kpad256` zero columns) | ~14.7 GB |
-  | Attention + dense MLP + untied head, int4 block-32 | ~1.4 GB |
-  | `embed_tokens`, fp16 in-graph | ~1.5 GB |
-  | Router, fp16 | ~0.02 GB |
-  | **Total** | **~17.6 GB** |
-  | + KV at 4096 | ~19.6 GB |
-  | + KV at 16384 | ~25.7 GB |
-  | Tier | Verdict |
-  |---|---|
-  | 16 GB | **no** — weights alone are ~1.6× the entire ~10.7 GB Metal working set |
-  | 24–32 GB | yes at ctx ≤ 8192 (~21.6 GB); 16384 is tight |
-  | 64 GB | yes, at any context |
-  This table is arithmetic from state shapes and weight formats — a prediction to check,
-  not a measured tier claim. The residency mechanism for MoE bundles on macOS (mmap'd vs
-  wired expert slabs) has not been measured.
-  25 of the 30 layers have `head_dim` 256 zero-padded to 512, and all 30 slots grow linearly
-  even though 25 of them only ever attend a 1024-token window. A ring-buffered sliding cache
-  would be roughly 5× cheaper per token; that is model authoring, not a flag.
-- The bundle manifest declares `runtime_env COREAI_CHUNK_THRESHOLD=1`.
-## Performance
-**No measurement exists, on any machine.** Every figure in this card is either a byte count,
-an export-host measurement, or derived from state shapes and weight formats. No token has
-been generated from this bundle.
-The intended argument for this model is throughput at comparable size against the dense 31B
-— 17.6 GB vs 20 GB, with ~4B active parameters per token read through a gather kernel — and
-quality per token against the 12B at ~2.1× the footprint. The nearest measured analogue is
-LFM2.5-8B-A1B at 140 tok/s through its gather kernel versus 39 tok/s for the same weights
-read as a dense over-read. Neither claim is measured for this bundle.
 ## Status
 | Artifact | Status |
 |---|---|
-| `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8` | **UNQUALIFIED** — never executed on target hardware. Established: the checkpoint is the QAT source, the graph exports, the producer fingerprint passes, and the MoE block wiring matches the reference implementation to 3.5e-7 relative error on real layer-0 weights (below). |
 A qualifying run would need, in priority order:
-1. **A decode oracle** — generate and compare against an fp32 HF reference. This is the gate
-   that catches an int4 or kernel-level error which the authoring gate below cannot see.
-2. **Isolate the two Metal kernels.** `gather_qmm` (the `aff4` variant) and the flash-decode
-   SDPA have never been in the same graph, on any model. If decode produces garbage,
-   re-export without `--metal-sdpa` first — it is an optimisation, the MoE kernel is not.
-3. **Measure residency**, both dirty footprint and maximum resident including mmap. No MoE
-   bundle has been measured this way.
-4. **Compare tok/s against the dense 12B and 31B** on the same machine.
-5. **Router sanity at depth.** 128 experts / top-8 with a learned `per_expert_scale` is far
-   sparser routing than the LFM2.5-8B-A1B's 32/top-4. Check expert-utilisation spread on a
-   real prompt set before trusting long-form output.
-### Verification
-**MoE block wiring verified against the reference implementation.** `transformers` 4.57.6
-(the pinned export environment) has no `gemma4`, so the gate transcribes
-`Gemma4TextDecoderLayer.forward` (the MoE-block half), `Gemma4TextRouter.forward` and
-`Gemma4TextExperts.forward` from upstream and runs both implementations on the real layer-0
-weights in fp32:
-```
-T=1: max|dFFN|=7.629e-05  rel=3.453e-07  |  same top-8 ids: True  max|dw|=0.000e+00
-T=4: max|dFFN|=9.537e-05  rel=2.735e-07  |  same top-8 ids: True  max|dw|=0.000e+00
-```
-Identical expert selection, bit-identical routing weights, ~3.5e-7 relative error on the
-block output. **This gates the authoring** — the wiring, the fused-weight split, the
-activation choice, the norm placement, the router maths. It does **not** gate the exported
-graph, the int4 rounding, or either Metal kernel. The attention half was not re-gated: it
-is the dense `Attention` class already used in the 12B and 31B bundles.
 ## License
-The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma
-model and **the Gemma terms apply downstream**, which is why this repo declares
-`license: gemma`. Use of these weights is subject to the
-[Gemma Terms of Use](https://ai.google.dev/gemma/terms), the
-[Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy) and the
-[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
-travel with any redistribution of this bundle. The contribution here is the port and the
-recipe, not the weights.
-## `hybrid-mf64/` bundles
-> **EXPERIMENTAL — not qualified.** These bundles have not passed a Mac-side
-> oracle/parity gate or a device benchmark. Do not route production traffic to
-> them until they have.
-**What this is.** The 26B-A4B with a `prefill` entrypoint, from one copy of the weights
-the published `gpu-pipelined/` decode bundle already ships. Two things had to be
-per-entrypoint at once, and both are:
-* **Experts.** `BatchedMetalSwitchGLU` sorts the `S*k` (token, expert) pairs of a 64-token
-  chunk by expert id so each expert slab is read once, and falls through to
-  `MetalSwitchGLU.forward` verbatim whenever `b*s == 1`. One metalization, both
-  entrypoints, **decode arithmetic unchanged** — `main` traces the identical q=1 gather
-  matvec the published decode bundle carries.
-* **Attention.** The 26B inherits the dense 12B/31B full-attention layers, so it needs the
-  flash-decode Metal kernel for the scratch-heap reason, and that kernel is structurally
-  q=1. `SplitSDPA` branches on `query.shape[2]`, which is a concrete int in each trace, so
-  `main` gets the kernel and `prefill` gets the MPSGraph composite (query axis +
-  `is_causal=True` intra-chunk mask). Same technique as the 31B `hybrid-pf64/` bundle.
-Both kernels are registered on one converter before either `add_pytorch_module`.
-**This is `mf64`, not `mf64-tp`, and it cannot be `-tp` today.** The token-tiled prefill
-kernel (`moe_metal_tiled`, the 8B-A1B `mf64-tp` experiment) is **sym8-only** —
-`TiledBatchedMetalSwitchGLU.__init__` raises for any other scheme. The 26B ships **aff4**
-experts, and that is not incidental: the checkpoint is QAT-trained on the ggml q4_0
-per-block-32 grid, which an affine int4 block-32 grid represents exactly. An aff4 tiled
-kernel is future work, gated on two things — the 8B `mf64-tp` Mac verdict proving the
-tiling principle at all, and then an aff4 equality proof of the kind `moek_tilegate.py`
-gives the sym8 kernel. What this bundle uses instead is the **shipped** batched path,
-the one behind the 8B `mf64` bundle that has a Mac measurement.
-**Measured on the conversion box (Linux x86_64, no GPU).** Export wall 305 s; peak RSS
-**87.5 GB** (well under the 12B's 68.7 → 31B's 148.9 band this was budgeted against, and
-never near the 75% kill line); bundle 17,580,916,612 B; producer `coreai-core 1.0.0b2`;
-ctx 16384; `function_map {"main": ["main", "prefill"]}`. Trace accounting: all 5 full
-layers took the metal branch at S=1 **and** the composite branch at S=64, 5/5 both ways —
-the export fails if either count is 0. 30 MoE layers metalized. Graph bytecode declares
-`input_ids`, `position_ids`, `keyCache`, `valueCache`, `logits`, `prefill`, 90
-`gather_qmm_int4aff` call sites (30 layers x gate/up/down) and
-`gemma4_dense_full_sdpa_occ8` — **2 inputs, no `mask`**, which is the arity
-`CoreAISequentialEngine` requires.
-**Numerics gate (box-side, `gate_hybrid_sdpa.json`).** The 26B is a *third* distinct
-block-GQA mapping and was gated as its own case: 16 query heads, **2** global KV heads
-replicated across 8 cache slots, so the kernel's `kv = h / (H / slots)` = `h//2` must land
-on real head `(h//2)//4 = h//8`. Kernel reference vs the composite at those exact shapes:
-max abs 2.50e-2 (S=137) and 1.17e-2 (S=1024) — the composite's own fp16 error. Against an
-fp32 ground truth the kernel is the more accurate of the two: **9.74e-4 / 74.9 dB** at
-S=137 and 9.76e-4 / 75.1 dB at S=1024, versus the composite's 53.0 and 57.9 dB. The 12B
-(1 global head, rep 8) and 31B (4 global heads, rep 4 over 16 slots) rows are in the same
-file as controls; a mapping bug that cancelled in those would show here. `MetalDenseFullSDPA`
-also evaluates in eager at 26B dims and matches its own reference exactly (max abs 0.0).
-This gates the mapping and the scale, not the Metal source (no GPU on the box).
-**Tier — read the arithmetic, not the bundle size.** KV is `2 x 30 layers x 8 KV slots x
-512 head_dim x fp16` = **491,520 B/token**:
-| Context | KV | + 17.6 GB weights | Verdict |
-|---|---|---|---|
-| 4096 | 2.01 GB | 19.6 GB | 32 GB machine |
-| 8192 | 4.03 GB | 21.6 GB | 32 GB machine, marginal against a ~21-24 GB working set |
-| 16384 (this manifest) | 8.05 GB | 25.6 GB | **64 GB machine** |
-So: a 32 GB Mac can run it at shallow context, the manifest's full 16384 wants 64 GB, and
-no 16 GB machine runs it at any context.
-**EXPERIMENTAL — unqualified until a big-machine measurement exists.** Nothing here has
-produced a token. In priority order:
-1. *Does it load and decode?* Decode should be indistinguishable from the published
-   `gpu-pipelined/` bundle: same weights, same q=1 gather kernel, same flash-decode SDPA.
-   A decode regression against that bundle would mean the batched wrapper is not falling
-   through at `b*s == 1` as its source says it does.
-2. *Prefill throughput.* The **falsifiable prediction**: the batched kernel's anchor is the
-   8B-A1B `mf64` bundle's measured **6.3 ms/prompt-token**, and this bundle should land in
-   that mechanism's class rather than the 21.5 ms/token of a decode-only bundle walking
-   the prompt one token at a time. Do not read 6.3 ms as the expected number for this
-   model: the 26B activates ~4B parameters per token against the 8B-A1B's ~1B, and its
-   dense MLP runs in parallel with the expert branch, so a figure several times 6.3 ms
-   would still confirm the mechanism. What would falsify it is prefill landing at or above
-   the per-token cost of decode — that would mean the sorted grouped-GEMM path is not
-   engaging and the chunk is being walked serially.
-3. *Chunk parity.* A prompt walked in 64-token chunks must give the same logits as the same
-   prompt walked token by token. Two independent reasons to check it here rather than
-   assume: the entrypoints use different attention implementations, and the expert path
-   does a sort/unsort round trip that the q=1 path does not.
-4. *Compare against the dense 31B on the same machine.* That is the whole case for this
-   model — 17.6 GB and ~4B active per token against 20 GB and 31B dense.
-Nothing already published was touched. `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8`
-is unchanged and remains the only 26B artifact anyone has run.
-| Bundle | Source checkpoint | Context | Functions | Bytes | Producer | Created |
-|---|---|---|---|---|---|---|
-| `hybrid-mf64/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8_pf64` | google/gemma-4-26B-A4B-it-qat-q4_0-unquantized | 16384 | main+prefill | 17,580,916,612 | coreai-core 1.0.0b2 | 20260818T141434Z |

 ---
+license: apache-2.0
+license_link: https://ai.google.dev/gemma/docs/gemma_4_license
 base_model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
+base_model_relation: quantized
 library_name: coreai
 pipeline_tag: text-generation
 tags:
   - aimodel
   - apple-silicon
   - on-device
+  - coreai-kit
   - quantized
   - int4
   - qat
 # Gemma 4 26B-A4B (MoE) — Core AI (.aimodel)
+`google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for
+Apple silicon by [visible-cx](https://huggingface.co/visible-cx). These are derivative
+artifacts: Google's QAT-trained weights rounded onto int4 and re-expressed as a Core AI graph,
+with the 128-expert sparse branch of every layer lowered onto a Metal gather kernel. They load
+through Core AI on macOS and are not usable by PyTorch, GGUF or MLX.
 It is a **sparse mixture-of-experts** model — 26.5B total parameters, roughly 4B active per
+token. This is a **new port**, not a reproduction of a published zoo bundle: three source
+modules were authored for it (below), and nothing in the shared toolchain was modified.
+> ⚠️ **UNQUALIFIED. Neither bundle has ever produced a token, on any machine.** The graphs
+> export, the producer fingerprints pass, and the MoE block wiring is verified against the
+> reference implementation to 3.5e-7 relative error. Nothing else is established.
 >
 > ⚠️ **The 16 GB tier is excluded.** Weights are ~17.6 GB resident against a 16 GB Mac's
+> ~10.7 GB Metal working set. **Minimum practical machine memory: 32 GB at shallow context,
+> 64 GB at the manifest's 16384.**
+> ⚠️ **Expected known issue — long unguided generations (unverified on this model).** On the
+> sibling E2B/E4B exports the Core AI runtime retains one compiled specialization per
+> sequence-length signature, costing roughly 81 MB of GPU allocations per generated token until
+> process exit, and the decode graphs here share the length-dependent structure that causes it.
+> Guided decoding is not a defence — **shape reuse is**. Measured details: the
+> [gemma-4-E2B-CoreAI](https://huggingface.co/visible-cx/gemma-4-E2B-CoreAI) card.
+## Contents
+| Path | Bytes | Files | Context | Functions | Status |
+|---|---:|---|---|---|---|
+| `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8` | 17,580,059,414 | 8 | 16384 | main (decode only) | UNQUALIFIED |
+| `hybrid-mf64/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8_pf64` | 17,580,916,612 | 8 | 16384 | main + prefill | EXPERIMENTAL |
+`gpu-pipelined/`, file by file:
+| File | Bytes |
+|---|---:|
+| `…aimodel/main.mlirb` | 17,547,866,346 |
+| `…aimodel/main.hash` | 32 |
+| `…aimodel/metadata.json` | 105 |
+| `metadata.json` (bundle manifest) | 689 |
+| `tokenizer/tokenizer.json` | 32,169,626 |
+| `tokenizer/chat_template.jinja` | 18,683 |
+| `tokenizer/tokenizer_config.json` | 3,730 |
+| `tokenizer/generation_config.json` | 203 |
+`hybrid-mf64/` is the same layout with `main.mlirb` at 17,548,723,515 B — **857,169 B more than
+the decode-only build**, which is the whole cost of a second entrypoint over deduplicated
+weights.
+Manifest context is 16384. `--max-ctx` sets `language.max_context_length` and nothing else;
+lower it if the machine cannot afford the KV — that is a metadata edit, not a re-export.
+**Stop token:** `eos_token = "<turn|>"` (id 106), the turn terminator Gemma 4 emits, applied by
+the export script itself. A host that stops on the raw upstream `<eos>` will overrun every
+reply.
 ## Provenance
 | Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay, plus three new modules (below) |
 | Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` |
 | Producer fingerprint | `coreai-core 1.0.0b2` — verified on the inner asset `metadata.json` |
+| Asset creation date | `20260817T215629Z` (`gpu-pipelined/`), `20260818T141434Z` (`hybrid-mf64/`) |
 | Vocab | 262,144 |
+"QAT-unquantized" means QAT-*trained*, stored bf16; the int4 rounding happens at export onto the
+ggml q4_0 grid the training already targeted. There is no separate pre-quantised int4 artifact
+to prefer.
+**To reproduce this conversion you need a ≥128 GB host**: the decode-only export peaks at
+**102.25 GB RSS** (`VmHWM`) on Linux x86_64; the `hybrid-mf64/` export peaked at **87.5 GB** in
+305 s.
 ### Architecture
 | `head_dim` / `global_head_dim` | **256 / 512** | 256 / 512 | 256 / 512 |
 | `sliding_window` | **1024** | 1024 | 512 |
 | Per-Layer Embeddings | **none** | none | 256 |
+Same attention family as the dense 12B/31B — same dual head_dim, same `attention_k_eq_v` (full
+layers carry no `v_proj`), same dual RoPE, same softcap. **There are no Per-Layer Embeddings and
+therefore no gather-table sidecar in this repo**, unlike E2B/E4B.
 Every layer carries a dense MLP **and** a sparse branch, in parallel:
 x  = x * layer_scalar
 ```
+Two details are load-bearing: **the router reads the un-normalised residual**, and **the experts
+are GELU-gated** (`gelu_pytorch_tanh`), where the `SwitchGLU` primitive's default is SiLU. The
+router itself: scale-free RMSNorm → `× scale × hidden**-0.5` → linear to 128 → fp32 softmax over
+all experts → top-8 → renormalise to sum 1 → multiply by a learned `per_expert_scale` gathered
+at the selected ids.
 ### Compression recipe
 | **Router** (`proj`, `scale`, `per_expert_scale`) | **fp16**, excluded by name | routers are the quantization-sensitive part of an MoE and the whole set is 0.02 GB |
 | `embed_tokens` (in-graph) | fp16 | the gather stays exact; the head is untied and quantized separately |
+`sym8` on the experts would put them at 22.8 GB before anything else, so int4 is not optional.
+`aff4` was chosen over a k-means palette (`km4`) because this checkpoint was QAT-trained on
+per-block-32 q4_0 (`w = (q−8)·d`, 16 uniform levels), and an affine int4 block-32 grid represents
+that **exactly** (`scale = d`, `bias = −8d`). A palette fits one 16-entry codebook per 32 output
+rows across the whole K axis and structurally cannot follow per-block QAT scales. The price is
+aux bytes: `aff4` stores an fp16 scale *and* bias per 32-element K block, ~5 bits/param effective
+against `km4`'s ~4.03 — about 3 GB of the bundle.
+The `down` projection's K of 704 is padded to 768 by the kernel's automatic `_kpad256` treatment
+— 64 zero columns, ~0.5% wasted expert bytes. `gate`/`up` need no padding.
+**The gather kernel is load-bearing.** Without `MetalSwitchGLU`, the sparse branch lowers to a
+dense matmul that reads all 128 experts every token — a 16× over-read.
 ### New source modules
 | File | Contents |
 |---|---|
 | `coreai_models/models/macos/gemma4_moe_text.py` | `Gemma4MoeConfig`, `Gemma4MoeRouter`, `GeluGLU`, `Gemma4MoeDecoderLayer`, `Gemma4MoeForCausalLM`, and a loader that splits the checkpoint's fused `gate_up_proj [E, 2*704, 2816]` into the `SwitchGLU`'s separate stacks |
 | `coreai_models/models/macos/gemma4_moe_pipelined.py` | `Gemma4MoePipelinedForCausalLM` — subclasses the dense pipelined core and overrides one method; the attention half is inherited unchanged |
 | `coreai-model-zoo/conversion/export_gemma4_moe_decode_pipelined.py` | the recipe |
+## Requirements
+- **Apple silicon Mac with ≥32 GB unified memory**, Core AI runtime.
+- **Engine contract: 2 inputs** (`input_ids`, `position_ids`) → logits, one growing KV pair, no
+  static inputs and no per-step mask. Verified by reading the graph bytecode of `hybrid-mf64/`:
+  `input_ids`, `position_ids`, `keyCache`, `valueCache`, `logits`, `prefill`, 90
+  `gather_qmm_int4aff` call sites (30 layers × gate/up/down) and `gemma4_dense_full_sdpa_occ8`
+  — **2 inputs, no `mask`**, which is the arity the sequential engine requires.
+- **States:** `keyCache` / `valueCache` `Float16, 30 × 1 × 8 × ? × 512` — one growing pair, 30
+  slots (no KV sharing). Dynamic sequence dim → `GrowingKVCache`.
+- **KV cost: 491,520 bytes per token of context** (fp16) — 2.01 GB at 4096, 8.05 GB at 16384.
+- **Resident weights, ~17.6 GB:**
+  | | bytes |
+  |---|---|
+  | Routed experts, `aff4` (incl. `_kpad256` zero columns) | ~14.7 GB |
+  | Attention + dense MLP + untied head, int4 block-32 | ~1.4 GB |
+  | `embed_tokens`, fp16 in-graph | ~1.5 GB |
+  | Router, fp16 | ~0.02 GB |
+  | **Total** | **~17.6 GB** |
+  | Context | KV | + weights | Verdict |
+  |---|---|---|---|
+  | 4096 | 2.01 GB | 19.6 GB | 32 GB machine |
+  | 8192 | 4.03 GB | 21.6 GB | 32 GB machine, marginal against a ~21–24 GB working set |
+  | 16384 (this manifest) | 8.05 GB | 25.6 GB | **64 GB machine** |
+  | 16 GB machine | — | — | **no, at any context** |
+  **This table is arithmetic from state shapes and weight formats — a prediction to check, not a
+  measured tier claim.** The residency mechanism for MoE bundles on macOS has never been
+  measured, and this project's one check of a bundle-bytes proxy against a real compile found it
+  **1.35 GiB optimistic** on a large bundle.
+  25 of the 30 layers have `head_dim` 256 zero-padded to 512, and all 30 slots grow linearly even
+  though 25 of them only ever attend a 1024-token window. A ring-buffered sliding cache would be
+  roughly 5× cheaper per token; that is model authoring, not a flag.
+- The bundle manifest declares `runtime_env COREAI_CHUNK_THRESHOLD=1`.
+## Measurements
+**None. No measurement of any kind exists, on any machine, for either bundle.** Every figure in
+this card is a byte count, an export-host reading, a box-side torch comparison, or arithmetic
+from state shapes and weight formats. No token has been generated.
+The intended argument for this model is throughput at comparable size against the dense 31B —
+17.6 GB vs 20 GB, with ~4B active parameters per token read through a gather kernel — and
+quality per token against the 12B at ~2.1× the footprint. The nearest *measured* analogue is
+LFM2.5-8B-A1B at 140 tok/s through its gather kernel against 39 tok/s for the same weights read
+as a dense over-read. **Neither claim is measured for this bundle.**
+One datapoint about the *checkpoint*, clearly scoped because it is not about these bundles: the
+same model served remotely (Cloudflare's `@cf/google/gemma-4-26b-a4b-it`) answered eleven real
+report prompts 11/11 schema-valid under a tuned recipe and passed two grounding probes that the
+local 8B-A1B fails — it identified a vendor-versus-subject confusion correctly and attributed an
+opinion rather than asserting it as fact. That is evidence about the weights' grounding, on a
+completely different serving path, and says nothing about this Core AI conversion.
+### What *has* been gated, box-side
+**MoE block wiring, against the reference implementation.** `transformers` 4.57.6 (the pinned
+export environment) has no `gemma4`, so the gate transcribes `Gemma4TextDecoderLayer.forward`
+(the MoE-block half), `Gemma4TextRouter.forward` and `Gemma4TextExperts.forward` from upstream
+and runs both implementations on the real layer-0 weights in fp32:
+```
+T=1: max|dFFN|=7.629e-05  rel=3.453e-07  |  same top-8 ids: True  max|dw|=0.000e+00
+T=4: max|dFFN|=9.537e-05  rel=2.735e-07  |  same top-8 ids: True  max|dw|=0.000e+00
+```
+Identical expert selection, bit-identical routing weights, ~3.5e-7 relative error on the block
+output. **This gates the authoring** — the wiring, the fused-weight split, the activation choice,
+the norm placement, the router maths. It does **not** gate the exported graph, the int4 rounding,
+or either Metal kernel.
+**Attention kernel numerics** (`hybrid-mf64/`). The 26B is a *third* distinct block-GQA mapping
+and was gated as its own case: 16 query heads, **2** global KV heads replicated across 8 cache
+slots, so the kernel's `kv = h / (H / slots)` = `h//2` must land on real head `(h//2)//4 = h//8`.
+| | vs the MPSGraph composite | vs an fp32 ground truth |
+|---|---|---|
+| max abs, S=137 | 2.50e-2 | **9.74e-4 — PSNR 74.9 dB** |
+| max abs, S=1024 | 1.17e-2 | 9.76e-4 — 75.1 dB |
+| the composite itself, vs fp32 | — | 53.0 and 57.9 dB |
+The divergence from the composite is the composite's own fp16 error; against fp32 the kernel is
+the more accurate of the two. The 12B (1 global head, rep 8) and 31B (4 global heads, rep 4 over
+16 slots) rows are in the same file as controls — a mapping bug that cancelled in those would
+show here. This gates the mapping and the scale, **not the Metal source** (no GPU on the box).
+**Trace accounting** (`hybrid-mf64/`): all 5 full layers took the metal branch at S=1 **and** the
+composite branch at S=64, 5/5 both ways — the export fails if either count is 0. 30 MoE layers
+metalized.
+## `hybrid-mf64/` — decode and prefill from one copy of the weights
+The 26B-A4B with a `prefill` entrypoint. Two things had to be per-entrypoint at once, and both
+are:
+- **Experts.** `BatchedMetalSwitchGLU` sorts the `S*k` (token, expert) pairs of a 64-token chunk
+  by expert id so each expert slab is read once, and falls through to `MetalSwitchGLU.forward`
+  verbatim whenever `b*s == 1`. One metalization, both entrypoints, **decode arithmetic
+  unchanged** — `main` traces the identical q=1 gather matvec the published decode bundle
+  carries.
+- **Attention.** The 26B inherits the dense 12B/31B full-attention layers, so it needs the
+  flash-decode Metal kernel for the scratch-heap reason, and that kernel is structurally q=1.
+  `SplitSDPA` branches on `query.shape[2]`, a concrete int in each trace, so `main` gets the
+  kernel and `prefill` gets the MPSGraph composite (query axis + `is_causal=True` intra-chunk
+  mask). Same technique as the 31B `hybrid-pf64/` bundle.
+**This is `mf64`, not `mf64-tp`, and it cannot be `-tp` today.** The token-tiled prefill kernel
+is **sym8-only** — `TiledBatchedMetalSwitchGLU.__init__` raises for any other scheme — and the
+26B ships `aff4` experts for the QAT-grid reason above. An `aff4` tiled kernel is future work,
+gated on the 8B `mf64-tp` Mac verdict (**which came back negative**: that kernel failed its gate
+on divergence and delivered 1.17× rather than the predicted 6.5×) and then an `aff4` equality
+proof. What this bundle uses instead is the **shipped** batched path, the one behind the 8B
+`mf64` bundle that has a Mac measurement.
+**The falsifiable prediction, for whoever runs it first.** The batched kernel's anchor is the
+8B-A1B `mf64` bundle's measured **6.3 ms/prompt-token**, and this bundle should land in that
+mechanism's class rather than the ~21.5 ms/token of a decode-only bundle walking the prompt one
+token at a time. Do not read 6.3 ms as the expected number: the 26B activates ~4B parameters per
+token against the 8B-A1B's ~1B, and its dense MLP runs in parallel with the expert branch, so
+several times 6.3 ms would still confirm the mechanism. **What would falsify it** is prefill
+landing at or above the per-token cost of decode — that would mean the sorted grouped-GEMM path
+is not engaging and the chunk is being walked serially.
+### Decode-only, and a prefill variant is not deliverable by a flag
+`gpu-pipelined/` has no `prefill` function, for two independent kernel reasons:
+1. `--metal-sdpa`'s flash-decode kernel is structurally q=1 with no causal mask. The full
+   attention layers need that kernel for the same scratch-heap reason the dense 12B/31B do.
+2. `MetalSwitchGLU.forward` is itself decode-only — it asserts a token batch of 1 and expands the
+   single activation row across the routed slots.
+`hybrid-mf64/` is the bundle that solves both at once.
+## Usage
+Swift Package Manager, via [CoreAIKit](https://github.com/john-rocky/coreai-kit) — a community
+package, not affiliated with Apple, requiring macOS 27 beta:
+```swift
+.package(url: "https://github.com/john-rocky/coreai-kit", branch: "main")
+// target dependency: .product(name: "CoreAIKit", package: "coreai-kit")
+```
+```swift
+import CoreAIKit
+let model = ModelID(
+    "visible-cx/gemma4-26b-a4b-CoreAI",
+    path: "hybrid-mf64/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8_pf64")
+let chat = try await ChatSession(model: model)   // ~17.6 GB download on first use
+```
+These are plain 2-input graphs — **no PLE table sidecar, no static input buffers** — so either
+engine will load them, and the sequential engine is available for grammar-constrained decoding
+(`configuration.engineVariant = .sequential`). Budget disk for the compiled graph on top of the
+17.6 GB bundle, and expect a long cold compile.
+## Integrity
+Core AI `.aimodel` bundles are **not byte-reproducible**: the exporter is not deterministic even
+against itself. Verify by digesting the exact published bytes rather than by rebuilding. Every
+bundle carries `main.hash`, the raw 32 bytes of `sha256(main.mlirb)`; on the Hub the same value
+is recoverable from the LFS oid without fetching the 17.6 GB file.
 ## Status
 | Artifact | Status |
 |---|---|
+| `gpu-pipelined/…_moeaff4_msdpa_g8` | **UNQUALIFIED** — never executed on target hardware. Established: the checkpoint is the QAT source, the graph exports, the producer fingerprint passes, and the MoE block wiring matches the reference implementation to 3.5e-7 relative error. |
+| `hybrid-mf64/…_msdpa_g8_pf64` | **EXPERIMENTAL — not qualified.** Adds an S=64 `prefill` entrypoint over the same weights, with box-side gates on the block-GQA mapping and the trace accounting. No Mac-side oracle, no device benchmark. Do not route production traffic to it. |
 A qualifying run would need, in priority order:
+1. **A decode oracle** — generate and compare against an fp32 HF reference. This is the gate that
+   catches an int4 or kernel-level error which the authoring gate cannot see.
+2. **Isolate the two Metal kernels.** `gather_qmm` (the `aff4` variant) and the flash-decode SDPA
+   have never been in the same graph, on any model. If decode produces garbage, re-export without
+   `--metal-sdpa` first — it is an optimisation, the MoE kernel is not.
+3. **Chunk parity** on `hybrid-mf64/`. Two independent reasons to check rather than assume: the
+   entrypoints use different attention implementations, and the expert path does a sort/unsort
+   round trip the q=1 path does not.
+4. **Measure residency**, both dirty footprint and wired growth. No MoE Gemma bundle has been
+   measured this way, and the ~17.6 GB figure above is arithmetic.
+5. **Compare tok/s against the dense 12B and 31B** on the same machine. That is the whole case
+   for this model.
+6. **Router sanity at depth.** 128 experts / top-8 with a learned `per_expert_scale` is far
+   sparser routing than the LFM2.5-8B-A1B's 32/top-4. Check expert-utilisation spread on a real
+   prompt set before trusting long-form output.
 ## License
+Google publishes the upstream QAT checkpoint under **Apache-2.0** with a `license_link` to the
+[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license), and this repo mirrors that
+declaration. Use is governed by those terms and by the
+[Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy); the
+obligations travel with any redistribution of these bundles. The contribution here is the port
+and the recipe, not the weights.