gen-cards: regenerate Use-it block

feff92c verified 6 days ago

5.63 kB

license: gemma
base_model: google/gemma-4-12B-it-qat-q4_0-unquantized
tags:
  - core-ai
  - coreai
  - apple
  - gemma
  - gemma-4
  - on-device
  - metal
pipeline_tag: text-generation
library_name: coreai

Gemma 4 12B (dense) — Core AI

Apple Core AI (.aimodel) conversion of Google's Gemma 4 12B dense text decoder, ported directly from the QAT release google/gemma-4-12B-it-qat-q4_0-unquantized. Decode-only, runs on the stock pipelined engine on Apple Silicon (M-series Macs).

First Core AI runtime for a ≥16-head × head_dim-512 full-attention model. Gemma 4 12B's full (global) attention layers have a 16-head × 512 Q tensor (16 KB fp16) that overflows MPSGraph's GPU decode scratch heap — the stock SDPA crashes at the first token (apple/coreai-models#27). These bundles ship a custom Metal flash-decode kernel on the full layers that removes the offending op, so the model runs. (The plain non-kernel bundles still crash — these _msdpa bundles are the runnable ones.)

Use it

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "Gemma 4 12B" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model gemma-4-12b --prompt "What can you do, offline?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let chat = try await ChatSession(catalog: "gemma-4-12b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
Info.plist: none needed
Entitlements: none needed (macOS)
First run downloads the model — 13.0 GB (Mac) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
Measure in Release — Debug is ~3× slower on per-token host work

Bundles (`gpu-pipelined/`)

bundle	quant	size	decode (M4 Max)	quality
`gemma4_12b_qat_decode_int8lin_msdpa_g8`	int8 (per-block-32)	14 GB	~23 tok/s (prefill 27.5)	verified-clean: engine greedy == fp32 oracle
`gemma4_12b_qat_decode_int4linsym_msdpa_g8`	int4 (q4_0-aligned absmax)	8.2 GB	~33 tok/s (prefill 43.4)	answers correctly, slightly 4-bit-lossy phrasing

The _g8 suffix is the higher-occupancy flash-decode kernel (8 SIMD-groups per head split the global layers' KV scan) — it holds throughput at long context (int8 decode 17.5 → 20.3 tok/s at 1024 generated tokens vs the simple kernel) with identical numerics.

int8 is the verified-clean default (its teacher-forced greedy reproduces the fp32 oracle's "The capital of France is Paris." exactly). int4 is the faster / smaller option (16 GB-Mac accessible) at a small quality cost — the same precision class as MLX 4-bit, not a conversion bug (the int8 graph is exact).

Architecture

Clean dense gemma4_unified text decoder — no PLE / AltUp / Laurel / MoE / KV-sharing (unlike the on-device E2B/E4B siblings). 48 layers, hidden 3840, 16 heads, vocab 262144, final logit softcap 30, tied embeddings. 5:1 sliding:full interleave; dual head_dim (sliding 256 / full global_head_dim 512); full layers use a single global KV head with attention_k_eq_v (value = raw k_proj). Both attention shapes ride one growing KV pair, so the bundle loads on the stock CoreAIPipelinedEngine (2 states, no engine patch); the full layers' SDPA runs as a custom Metal flash-decode kernel.

Usage

Download a bundle and run with Apple's llm-runner / llm-benchmark (the pipelined engine; set COREAI_CHUNK_THRESHOLD=1):

huggingface-cli download mlboydaisuke/Gemma-4-12B-CoreAI \
    --include "gpu-pipelined/gemma4_12b_qat_decode_int8lin_msdpa_g8/*" \
    --local-dir ./gemma4-12b-coreai

COREAI_CHUNK_THRESHOLD=1 llm-runner \
    --model ./gemma4-12b-coreai/gpu-pipelined/gemma4_12b_qat_decode_int8lin_msdpa_g8 \
    --prompt "What is the capital of France?" --max-tokens 64 --chunk-size 1

Each bundle is self-contained: the .aimodel, metadata.json, and the Gemma tokenizer.

Conversion

Community zoo (recipe, overlays, model card): github.com/john-rocky/coreai-model-zoo → zoo/gemma4-12b.md.

License

Gemma — governed by the Gemma Terms of Use. By using these weights you agree to those terms. The conversion (Core AI bundles, custom Metal kernel) adds no additional restrictions.