README.md

---
license: gemma
base_model: google/gemma-4-12B-it-qat-q4_0-unquantized
tags:
- core-ai
- coreai
- apple
- gemma
- gemma-4
- on-device
- metal
pipeline_tag: text-generation
library_name: coreai
---

# Gemma 4 12B (dense) — Core AI

Apple **Core AI** (`.aimodel`) conversion of Google's **Gemma 4 12B** dense text decoder,
ported directly from the QAT release
[`google/gemma-4-12B-it-qat-q4_0-unquantized`](https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-unquantized).
Decode-only, runs on the **stock pipelined engine** on Apple Silicon (M-series Macs).

> **First Core AI runtime for a ≥16-head × head_dim-512 full-attention model.** Gemma 4 12B's
> full (global) attention layers have a 16-head × 512 Q tensor (16 KB fp16) that overflows
> MPSGraph's GPU decode scratch heap — the stock SDPA crashes at the first token
> ([apple/coreai-models#27](https://github.com/apple/coreai-models/issues/27)). These bundles
> ship a **custom Metal flash-decode kernel** on the full layers that removes the offending op,
> so the model runs. (The plain non-kernel bundles still crash — these `_msdpa` bundles are the
> runnable ones.)

<!-- gen-cards:use-it begin id=gemma-4-12b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
## Use it

▶️ **Run it (source)** — the [ChatDemo runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/ChatDemo)
(GUI + CLI, one app for every chat model in the catalog):

```bash
git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Gemma 4 12B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model gemma-4-12b --prompt "What can you do, offline?"
```

💻 **Build with it** — complete; the glue is kit API, copy-paste runs:

```swift
import CoreAIKit

let chat = try await ChatSession(catalog: "gemma-4-12b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device
```

The take-home is [`Examples/ChatDemo/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/ChatDemo/Sources/QuickStart.swift)
— this exact code as one typed function, no UI; the CLI is an argument shell over it, and
the GUI drives the same `ChatSession` across turns for its transcript.
Multi-turn? Hold the `ChatSession` and call `respond(to:)` per turn — it keeps the
conversation history; `streamResponse(to:)` yields tokens as they decode.

**Integration checklist**

- SPM: `https://github.com/john-rocky/coreai-kit` → product **CoreAIKit**
- Info.plist: none needed
- Entitlements: none needed (macOS)
- First run downloads the model — 13.0 GB (Mac) — then it loads from the
  local cache (Application Support; progress via the `downloadProgress` callback)
- Measure in Release — Debug is ~3× slower on per-token host work
<!-- gen-cards:use-it end -->

## Bundles (`gpu-pipelined/`)

| bundle | quant | size | decode (M4 Max) | quality |
|---|---|---|---|---|
| `gemma4_12b_qat_decode_int8lin_msdpa_g8` | int8 (per-block-32) | 14 GB | **~23 tok/s** (prefill 27.5) | verified-clean: engine greedy == fp32 oracle |
| `gemma4_12b_qat_decode_int4linsym_msdpa_g8` | int4 (q4_0-aligned absmax) | 8.2 GB | **~33 tok/s** (prefill 43.4) | answers correctly, slightly 4-bit-lossy phrasing |

The `_g8` suffix is the higher-occupancy flash-decode kernel (8 SIMD-groups per head split the
global layers' KV scan) — it holds throughput at long context (int8 decode 17.5 → **20.3 tok/s** at
1024 generated tokens vs the simple kernel) with identical numerics.

**int8 is the verified-clean default** (its teacher-forced greedy reproduces the fp32 oracle's
"The capital of France is Paris." exactly). **int4** is the faster / smaller option (16 GB-Mac
accessible) at a small quality cost — the same precision class as MLX 4-bit, *not* a conversion
bug (the int8 graph is exact).

## Architecture

Clean dense `gemma4_unified` text decoder — **no** PLE / AltUp / Laurel / MoE / KV-sharing
(unlike the on-device E2B/E4B siblings). 48 layers, hidden 3840, 16 heads, vocab 262144, final
logit softcap 30, tied embeddings. 5:1 sliding:full interleave; dual head_dim (sliding 256 / full
`global_head_dim` 512); full layers use a single global KV head with `attention_k_eq_v` (value =
raw k_proj). Both attention shapes ride **one** growing KV pair, so the bundle loads on the stock
`CoreAIPipelinedEngine` (2 states, no engine patch); the full layers' SDPA runs as a custom Metal
flash-decode kernel.

## Usage

Download a bundle and run with Apple's `llm-runner` / `llm-benchmark` (the pipelined engine; set
`COREAI_CHUNK_THRESHOLD=1`):

```bash
huggingface-cli download mlboydaisuke/Gemma-4-12B-CoreAI \
    --include "gpu-pipelined/gemma4_12b_qat_decode_int8lin_msdpa_g8/*" \
    --local-dir ./gemma4-12b-coreai

COREAI_CHUNK_THRESHOLD=1 llm-runner \
    --model ./gemma4-12b-coreai/gpu-pipelined/gemma4_12b_qat_decode_int8lin_msdpa_g8 \
    --prompt "What is the capital of France?" --max-tokens 64 --chunk-size 1
```

Each bundle is self-contained: the `.aimodel`, `metadata.json`, and the Gemma tokenizer.

## Conversion

Community zoo (recipe, overlays, model card):
[github.com/john-rocky/coreai-model-zoo → `zoo/gemma4-12b.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/gemma4-12b.md).

## License

Gemma — governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). By using these
weights you agree to those terms. The conversion (Core AI bundles, custom Metal kernel) adds no
additional restrictions.