README.md

---
license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
base_model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
base_model_relation: quantized
library_name: coreai
pipeline_tag: text-generation
tags:
  - core-ai
  - aimodel
  - apple-silicon
  - on-device
  - coreai-kit
  - quantized
  - int4
  - qat
  - moe
  - mixture-of-experts
  - gemma4
---

# Gemma 4 26B-A4B (MoE) — Core AI (.aimodel)

`google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for
Apple silicon by [visible-cx](https://huggingface.co/visible-cx). These are derivative
artifacts: Google's QAT-trained weights rounded onto int4 and re-expressed as a Core AI graph,
with the 128-expert sparse branch of every layer lowered onto a Metal gather kernel. They load
through Core AI on macOS and are not usable by PyTorch, GGUF or MLX.

It is a **sparse mixture-of-experts** model — 26.5B total parameters, roughly 4B active per
token. This is a **new port**, not a reproduction of a published zoo bundle: three source
modules were authored for it (below), and nothing in the shared toolchain was modified.

> ⚠️ **UNQUALIFIED. Neither bundle has ever produced a token, on any machine.** The graphs
> export, the producer fingerprints pass, and the MoE block wiring is verified against the
> reference implementation to 3.5e-7 relative error. Nothing else is established.
>
> ⚠️ **The 16 GB tier is excluded.** Weights are ~17.6 GB resident against a 16 GB Mac's
> ~10.7 GB Metal working set. **Minimum practical machine memory: 32 GB at shallow context,
> 64 GB at the manifest's 16384.**

> ⚠️ **Expected known issue — long unguided generations (unverified on this model).** On the
> sibling E2B/E4B exports the Core AI runtime retains one compiled specialization per
> sequence-length signature, costing roughly 81 MB of GPU allocations per generated token until
> process exit, and the decode graphs here share the length-dependent structure that causes it.
> Guided decoding is not a defence — **shape reuse is**. Measured details: the
> [gemma-4-E2B-CoreAI](https://huggingface.co/visible-cx/gemma-4-E2B-CoreAI) card.

## Contents

| Path | Bytes | Files | Context | Functions | Status |
|---|---:|---|---|---|---|
| `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8` | 17,580,059,414 | 8 | 16384 | main (decode only) | UNQUALIFIED |
| `hybrid-mf64/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8_pf64` | 17,580,916,612 | 8 | 16384 | main + prefill | EXPERIMENTAL |

`gpu-pipelined/`, file by file:

| File | Bytes |
|---|---:|
| `…aimodel/main.mlirb` | 17,547,866,346 |
| `…aimodel/main.hash` | 32 |
| `…aimodel/metadata.json` | 105 |
| `metadata.json` (bundle manifest) | 689 |
| `tokenizer/tokenizer.json` | 32,169,626 |
| `tokenizer/chat_template.jinja` | 18,683 |
| `tokenizer/tokenizer_config.json` | 3,730 |
| `tokenizer/generation_config.json` | 203 |

`hybrid-mf64/` is the same layout with `main.mlirb` at 17,548,723,515 B — **857,169 B more than
the decode-only build**, which is the whole cost of a second entrypoint over deduplicated
weights.

Manifest context is 16384. `--max-ctx` sets `language.max_context_length` and nothing else;
lower it if the machine cannot afford the KV — that is a metadata edit, not a re-export.

**Stop token:** `eos_token = "<turn|>"` (id 106), the turn terminator Gemma 4 emits, applied by
the export script itself. A host that stops on the raw upstream `<eos>` will overrun every
reply.

## Provenance

| | |
|---|---|
| Base checkpoint | `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` @ `f1e06dc520982d9b9edd76859fdb7ab209449949` |
| Checkpoint size | 51,644,341,801 B, bf16, 1,013 tensors across 2 shards; ungated |
| Recipe | `export_gemma4_moe_decode_pipelined.py int4lin --lin-sym --metal-sdpa --max-ctx 16384` |
| Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay, plus three new modules (below) |
| Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` |
| Producer fingerprint | `coreai-core 1.0.0b2` — verified on the inner asset `metadata.json` |
| Asset creation date | `20260817T215629Z` (`gpu-pipelined/`), `20260818T141434Z` (`hybrid-mf64/`) |
| Vocab | 262,144 |

"QAT-unquantized" means QAT-*trained*, stored bf16; the int4 rounding happens at export onto the
ggml q4_0 grid the training already targeted. There is no separate pre-quantised int4 artifact
to prefer.

**To reproduce this conversion you need a ≥128 GB host**: the decode-only export peaks at
**102.25 GB RSS** (`VmHWM`) on Linux x86_64; the `hybrid-mf64/` export peaked at **87.5 GB** in
305 s.

### Architecture

| | 26B-A4B | 31B | E4B |
|---|---|---|---|
| `hidden_size` | **2816** | 5376 | 2560 |
| layers | **30** (25 sliding / 5 full) | 60 | 42 |
| dense MLP `intermediate_size` | **2112** | 21504 | 10240 |
| `num_experts` / `top_k` | **128 / 8** | — | — |
| `moe_intermediate_size` | **704** | — | — |
| attention heads | **16** | 32 | 8 |
| KV heads (sliding / full) | **8 / 2** | 16 / 4 | 2 / — |
| `head_dim` / `global_head_dim` | **256 / 512** | 256 / 512 | 256 / 512 |
| `sliding_window` | **1024** | 1024 | 512 |
| Per-Layer Embeddings | **none** | none | 256 |

Same attention family as the dense 12B/31B — same dual head_dim, same `attention_k_eq_v` (full
layers carry no `v_proj`), same dual RoPE, same softcap. **There are no Per-Layer Embeddings and
therefore no gather-table sidecar in this repo**, unlike E2B/E4B.

Every layer carries a dense MLP **and** a sparse branch, in parallel:

```python
residual = x                                              # post-attention hidden
h  = mlp(pre_feedforward_layernorm(x))                    # dense branch
h1 = post_feedforward_layernorm_1(h)
w, idx = router(residual)                                 # routes on the RAW residual
h2 = experts(pre_feedforward_layernorm_2(residual), idx)  # sparse branch
h2 = post_feedforward_layernorm_2(sum_k w_k * h2_k)
x  = residual + post_feedforward_layernorm(h1 + h2)
x  = x * layer_scalar
```

Two details are load-bearing: **the router reads the un-normalised residual**, and **the experts
are GELU-gated** (`gelu_pytorch_tanh`), where the `SwitchGLU` primitive's default is SiLU. The
router itself: scale-free RMSNorm → `× scale × hidden**-0.5` → linear to 128 → fp32 softmax over
all experts → top-8 → renormalise to sum 1 → multiply by a learned `per_expert_scale` gathered
at the selected ids.

### Compression recipe

| Weights | Scheme | Rationale |
|---|---|---|
| **Routed experts** (128/layer, 22.84 G params) | **affine int4** (`aff4`), per-(output row, K-block-32) scale + bias, MSE-optimal clip over 6 candidates | see below |
| Attention, dense MLP, untied LM head | linear **int4** per-block-32, plain absmax (`--lin-sym`) | the ggml q4_0 grid the QAT checkpoint was trained on — the 12B/31B recipe verbatim |
| **Router** (`proj`, `scale`, `per_expert_scale`) | **fp16**, excluded by name | routers are the quantization-sensitive part of an MoE and the whole set is 0.02 GB |
| `embed_tokens` (in-graph) | fp16 | the gather stays exact; the head is untied and quantized separately |

`sym8` on the experts would put them at 22.8 GB before anything else, so int4 is not optional.
`aff4` was chosen over a k-means palette (`km4`) because this checkpoint was QAT-trained on
per-block-32 q4_0 (`w = (q−8)·d`, 16 uniform levels), and an affine int4 block-32 grid represents
that **exactly** (`scale = d`, `bias = −8d`). A palette fits one 16-entry codebook per 32 output
rows across the whole K axis and structurally cannot follow per-block QAT scales. The price is
aux bytes: `aff4` stores an fp16 scale *and* bias per 32-element K block, ~5 bits/param effective
against `km4`'s ~4.03 — about 3 GB of the bundle.

The `down` projection's K of 704 is padded to 768 by the kernel's automatic `_kpad256` treatment
— 64 zero columns, ~0.5% wasted expert bytes. `gate`/`up` need no padding.

**The gather kernel is load-bearing.** Without `MetalSwitchGLU`, the sparse branch lowers to a
dense matmul that reads all 128 experts every token — a 16× over-read.

### New source modules

| File | Contents |
|---|---|
| `coreai_models/models/macos/gemma4_moe_text.py` | `Gemma4MoeConfig`, `Gemma4MoeRouter`, `GeluGLU`, `Gemma4MoeDecoderLayer`, `Gemma4MoeForCausalLM`, and a loader that splits the checkpoint's fused `gate_up_proj [E, 2*704, 2816]` into the `SwitchGLU`'s separate stacks |
| `coreai_models/models/macos/gemma4_moe_pipelined.py` | `Gemma4MoePipelinedForCausalLM` — subclasses the dense pipelined core and overrides one method; the attention half is inherited unchanged |
| `coreai-model-zoo/conversion/export_gemma4_moe_decode_pipelined.py` | the recipe |

## Requirements

- **Apple silicon Mac with ≥32 GB unified memory**, Core AI runtime.
- **Engine contract: 2 inputs** (`input_ids`, `position_ids`) → logits, one growing KV pair, no
  static inputs and no per-step mask. Verified by reading the graph bytecode of `hybrid-mf64/`:
  `input_ids`, `position_ids`, `keyCache`, `valueCache`, `logits`, `prefill`, 90
  `gather_qmm_int4aff` call sites (30 layers × gate/up/down) and `gemma4_dense_full_sdpa_occ8`
  — **2 inputs, no `mask`**, which is the arity the sequential engine requires.
- **States:** `keyCache` / `valueCache` `Float16, 30 × 1 × 8 × ? × 512` — one growing pair, 30
  slots (no KV sharing). Dynamic sequence dim → `GrowingKVCache`.
- **KV cost: 491,520 bytes per token of context** (fp16) — 2.01 GB at 4096, 8.05 GB at 16384.
- **Resident weights, ~17.6 GB:**

  | | bytes |
  |---|---|
  | Routed experts, `aff4` (incl. `_kpad256` zero columns) | ~14.7 GB |
  | Attention + dense MLP + untied head, int4 block-32 | ~1.4 GB |
  | `embed_tokens`, fp16 in-graph | ~1.5 GB |
  | Router, fp16 | ~0.02 GB |
  | **Total** | **~17.6 GB** |

  | Context | KV | + weights | Verdict |
  |---|---|---|---|
  | 4096 | 2.01 GB | 19.6 GB | 32 GB machine |
  | 8192 | 4.03 GB | 21.6 GB | 32 GB machine, marginal against a ~21–24 GB working set |
  | 16384 (this manifest) | 8.05 GB | 25.6 GB | **64 GB machine** |
  | 16 GB machine | — | — | **no, at any context** |

  **This table is arithmetic from state shapes and weight formats — a prediction to check, not a
  measured tier claim.** The residency mechanism for MoE bundles on macOS has never been
  measured, and this project's one check of a bundle-bytes proxy against a real compile found it
  **1.35 GiB optimistic** on a large bundle.

  25 of the 30 layers have `head_dim` 256 zero-padded to 512, and all 30 slots grow linearly even
  though 25 of them only ever attend a 1024-token window. A ring-buffered sliding cache would be
  roughly 5× cheaper per token; that is model authoring, not a flag.
- The bundle manifest declares `runtime_env COREAI_CHUNK_THRESHOLD=1`.

## Measurements

**None. No measurement of any kind exists, on any machine, for either bundle.** Every figure in
this card is a byte count, an export-host reading, a box-side torch comparison, or arithmetic
from state shapes and weight formats. No token has been generated.

The intended argument for this model is throughput at comparable size against the dense 31B —
17.6 GB vs 20 GB, with ~4B active parameters per token read through a gather kernel — and
quality per token against the 12B at ~2.1× the footprint. The nearest *measured* analogue is
LFM2.5-8B-A1B at 140 tok/s through its gather kernel against 39 tok/s for the same weights read
as a dense over-read. **Neither claim is measured for this bundle.**

One datapoint about the *checkpoint*, clearly scoped because it is not about these bundles: the
same model served remotely (Cloudflare's `@cf/google/gemma-4-26b-a4b-it`) answered eleven real
report prompts 11/11 schema-valid under a tuned recipe and passed two grounding probes that the
local 8B-A1B fails — it identified a vendor-versus-subject confusion correctly and attributed an
opinion rather than asserting it as fact. That is evidence about the weights' grounding, on a
completely different serving path, and says nothing about this Core AI conversion.

### What *has* been gated, box-side

**MoE block wiring, against the reference implementation.** `transformers` 4.57.6 (the pinned
export environment) has no `gemma4`, so the gate transcribes `Gemma4TextDecoderLayer.forward`
(the MoE-block half), `Gemma4TextRouter.forward` and `Gemma4TextExperts.forward` from upstream
and runs both implementations on the real layer-0 weights in fp32:

```
T=1: max|dFFN|=7.629e-05  rel=3.453e-07  |  same top-8 ids: True  max|dw|=0.000e+00
T=4: max|dFFN|=9.537e-05  rel=2.735e-07  |  same top-8 ids: True  max|dw|=0.000e+00
```

Identical expert selection, bit-identical routing weights, ~3.5e-7 relative error on the block
output. **This gates the authoring** — the wiring, the fused-weight split, the activation choice,
the norm placement, the router maths. It does **not** gate the exported graph, the int4 rounding,
or either Metal kernel.

**Attention kernel numerics** (`hybrid-mf64/`). The 26B is a *third* distinct block-GQA mapping
and was gated as its own case: 16 query heads, **2** global KV heads replicated across 8 cache
slots, so the kernel's `kv = h / (H / slots)` = `h//2` must land on real head `(h//2)//4 = h//8`.

| | vs the MPSGraph composite | vs an fp32 ground truth |
|---|---|---|
| max abs, S=137 | 2.50e-2 | **9.74e-4 — PSNR 74.9 dB** |
| max abs, S=1024 | 1.17e-2 | 9.76e-4 — 75.1 dB |
| the composite itself, vs fp32 | — | 53.0 and 57.9 dB |

The divergence from the composite is the composite's own fp16 error; against fp32 the kernel is
the more accurate of the two. The 12B (1 global head, rep 8) and 31B (4 global heads, rep 4 over
16 slots) rows are in the same file as controls — a mapping bug that cancelled in those would
show here. This gates the mapping and the scale, **not the Metal source** (no GPU on the box).

**Trace accounting** (`hybrid-mf64/`): all 5 full layers took the metal branch at S=1 **and** the
composite branch at S=64, 5/5 both ways — the export fails if either count is 0. 30 MoE layers
metalized.

## `hybrid-mf64/` — decode and prefill from one copy of the weights

The 26B-A4B with a `prefill` entrypoint. Two things had to be per-entrypoint at once, and both
are:

- **Experts.** `BatchedMetalSwitchGLU` sorts the `S*k` (token, expert) pairs of a 64-token chunk
  by expert id so each expert slab is read once, and falls through to `MetalSwitchGLU.forward`
  verbatim whenever `b*s == 1`. One metalization, both entrypoints, **decode arithmetic
  unchanged** — `main` traces the identical q=1 gather matvec the published decode bundle
  carries.
- **Attention.** The 26B inherits the dense 12B/31B full-attention layers, so it needs the
  flash-decode Metal kernel for the scratch-heap reason, and that kernel is structurally q=1.
  `SplitSDPA` branches on `query.shape[2]`, a concrete int in each trace, so `main` gets the
  kernel and `prefill` gets the MPSGraph composite (query axis + `is_causal=True` intra-chunk
  mask). Same technique as the 31B `hybrid-pf64/` bundle.

**This is `mf64`, not `mf64-tp`, and it cannot be `-tp` today.** The token-tiled prefill kernel
is **sym8-only** — `TiledBatchedMetalSwitchGLU.__init__` raises for any other scheme — and the
26B ships `aff4` experts for the QAT-grid reason above. An `aff4` tiled kernel is future work,
gated on the 8B `mf64-tp` Mac verdict (**which came back negative**: that kernel failed its gate
on divergence and delivered 1.17× rather than the predicted 6.5×) and then an `aff4` equality
proof. What this bundle uses instead is the **shipped** batched path, the one behind the 8B
`mf64` bundle that has a Mac measurement.

**The falsifiable prediction, for whoever runs it first.** The batched kernel's anchor is the
8B-A1B `mf64` bundle's measured **6.3 ms/prompt-token**, and this bundle should land in that
mechanism's class rather than the ~21.5 ms/token of a decode-only bundle walking the prompt one
token at a time. Do not read 6.3 ms as the expected number: the 26B activates ~4B parameters per
token against the 8B-A1B's ~1B, and its dense MLP runs in parallel with the expert branch, so
several times 6.3 ms would still confirm the mechanism. **What would falsify it** is prefill
landing at or above the per-token cost of decode — that would mean the sorted grouped-GEMM path
is not engaging and the chunk is being walked serially.

### Decode-only, and a prefill variant is not deliverable by a flag

`gpu-pipelined/` has no `prefill` function, for two independent kernel reasons:

1. `--metal-sdpa`'s flash-decode kernel is structurally q=1 with no causal mask. The full
   attention layers need that kernel for the same scratch-heap reason the dense 12B/31B do.
2. `MetalSwitchGLU.forward` is itself decode-only — it asserts a token batch of 1 and expands the
   single activation row across the routed slots.

`hybrid-mf64/` is the bundle that solves both at once.

## Usage

Swift Package Manager, via [CoreAIKit](https://github.com/john-rocky/coreai-kit) — a community
package, not affiliated with Apple, requiring macOS 27 beta:

```swift
.package(url: "https://github.com/john-rocky/coreai-kit", branch: "main")
// target dependency: .product(name: "CoreAIKit", package: "coreai-kit")
```

```swift
import CoreAIKit

let model = ModelID(
    "visible-cx/gemma4-26b-a4b-CoreAI",
    path: "hybrid-mf64/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8_pf64")

let chat = try await ChatSession(model: model)   // ~17.6 GB download on first use
```

These are plain 2-input graphs — **no PLE table sidecar, no static input buffers** — so either
engine will load them, and the sequential engine is available for grammar-constrained decoding
(`configuration.engineVariant = .sequential`). Budget disk for the compiled graph on top of the
17.6 GB bundle, and expect a long cold compile.

## Integrity

Core AI `.aimodel` bundles are **not byte-reproducible**: the exporter is not deterministic even
against itself. Verify by digesting the exact published bytes rather than by rebuilding. Every
bundle carries `main.hash`, the raw 32 bytes of `sha256(main.mlirb)`; on the Hub the same value
is recoverable from the LFS oid without fetching the 17.6 GB file.

## Status

| Artifact | Status |
|---|---|
| `gpu-pipelined/…_moeaff4_msdpa_g8` | **UNQUALIFIED** — never executed on target hardware. Established: the checkpoint is the QAT source, the graph exports, the producer fingerprint passes, and the MoE block wiring matches the reference implementation to 3.5e-7 relative error. |
| `hybrid-mf64/…_msdpa_g8_pf64` | **EXPERIMENTAL — not qualified.** Adds an S=64 `prefill` entrypoint over the same weights, with box-side gates on the block-GQA mapping and the trace accounting. No Mac-side oracle, no device benchmark. Do not route production traffic to it. |

A qualifying run would need, in priority order:

1. **A decode oracle** — generate and compare against an fp32 HF reference. This is the gate that
   catches an int4 or kernel-level error which the authoring gate cannot see.
2. **Isolate the two Metal kernels.** `gather_qmm` (the `aff4` variant) and the flash-decode SDPA
   have never been in the same graph, on any model. If decode produces garbage, re-export without
   `--metal-sdpa` first — it is an optimisation, the MoE kernel is not.
3. **Chunk parity** on `hybrid-mf64/`. Two independent reasons to check rather than assume: the
   entrypoints use different attention implementations, and the expert path does a sort/unsort
   round trip the q=1 path does not.
4. **Measure residency**, both dirty footprint and wired growth. No MoE Gemma bundle has been
   measured this way, and the ~17.6 GB figure above is arithmetic.
5. **Compare tok/s against the dense 12B and 31B** on the same machine. That is the whole case
   for this model.
6. **Router sanity at depth.** 128 experts / top-8 with a learned `per_expert_scale` is far
   sparser routing than the LFM2.5-8B-A1B's 32/top-4. Check expert-utilisation spread on a real
   prompt set before trusting long-form output.

## License

Google publishes the upstream QAT checkpoint under **Apache-2.0** with a `license_link` to the
[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license), and this repo mirrors that
declaration. Use is governed by those terms and by the
[Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy); the
obligations travel with any redistribution of these bundles. The contribution here is the port
and the recipe, not the weights.