visible-cx/Gemma-4-12B-CoreAI

Gemma 4 12B — Core AI (.aimodel)

google/gemma-4-12B-it-qat-q4_0-unquantized converted to Core AI .aimodel bundles for Apple silicon by visible-cx. These are derivative artifacts: Google's QAT-trained weights rounded onto the int4 (or int8) grid and re-expressed as a Core AI graph. They load through Core AI on macOS and are not usable by PyTorch, GGUF or MLX. Gemma 4 12B is a dense model — no Per-Layer Embeddings and no gather-table sidecar, unlike the E2B and E4B bundles in this org.

The repo holds two generations of build. The three dense2in* folders are the original 2-input dense exports. tiered/ is the current one: a single bundle carrying seven pinned context capacities from 4,096 to 262,144, gated bit-for-bit against each other, for 4.54 MB more than a single-capacity bundle.

tiered/ is STAGED, not qualified. No tier has ever been run on a Mac, the multifunction bundle has never been loaded by the engine, and its compiled blob is a forecast. Every per-rung requirement below inherits that forecast. See The seven-rung ladder.

Known issue on the dense2in* bundles — memory growth on long generations. The Core AI runtime retains one compiled specialization per sequence-length signature, and these decode graphs make sequence length load-bearing, so a long free-form generation retains roughly 80 MB of GPU allocations per generated token until the process exits. Measured on the sibling E2B/E4B exports (~81 MB/token at 3.3k of context, guided and unguided alike). The tiered/ line does not have this defect — shape-stability was measured directly on its wkv_stable predecessor: 6,962 forward passes, one shape signature, stable=true on every one, with total process-footprint growth of 5.0 MB across the whole run.

Contents

Path	Bytes	Weights	Context	Functions	Status
tiered/gemma4_12b_qat_decode_int4linsym_e4_msdpa_g8_wkv_stable_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k	main.mlirb 6,705,129,126	int4	7 rungs, 4,096 → 262,144	main ×7 (decode only)	STAGED
dense2in/gemma4_12b_qat_decode_int4linsym	8,746,180,229	int4	4096	main	EXPERIMENTAL
dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64	8,747,078,381	int4	4096	main + prefill	EXPERIMENTAL
dense2in-int8/gemma4_12b_qat_decode_int8lin_pf64	14,700,368,629	int8	16384	main + prefill	EXPERIMENTAL

Each folder holds <name>.aimodel/ (main.mlirb, main.hash, asset metadata.json), a bundle-level metadata.json, and tokenizer/ (tokenizer.json 32,169,626 B, tokenizer_config.json 3,090 B, chat_template.jinja 18,683 B, generation_config.json). With the shared tokenizer the staged tiered/ bundle is 6.27 GiB on disk.

dense2in-pf64's weights are bit-identical to dense2in's; the multifunction build adds an S=64 prefill entrypoint for under a megabyte, because the exporter deduplicates weights across entrypoints.

Stop token: every bundle declares eos_token = "<turn|>" (id 106), the turn terminator Gemma 4 emits. A host that stops on the raw upstream <eos> instead will overrun every reply.

Manifest context is metadata only — except on tiered/. On the dense2in* bundles --max-ctx sets language.max_context_length and nothing else, so the numbers above can be raised or lowered without a re-export (whether the result fits in memory is a separate question). On tiered/ the capacity is a real, pinned graph extent per entrypoint, and language.max_context_length reports the top rung — see the warning under Selecting a rung.

Provenance

Base checkpoint	google/gemma-4-12B-it-qat-q4_0-unquantized (ungated)
Zoo recipe	gemma-4-12b, status = "verified" — the dense recipe
Recipe (dense2in)	that recipe with int4lin --lin-sym and no --metal-sdpa, --max-ctx 4096
Recipe (dense2in-pf64)	as dense2in, multifunction, via export_gemma4_dense_pf.py
Recipe (dense2in-int8)	int8lin (symmetric-with-clipping, per-block-32), multifunction, same 2-input dense graph
Recipe (tiered/)	export_gemma4_12b_tiered.py — int4lin --lin-sym, --quant-embed int4 (e4), --metal-sdpa split-8 (msdpa_g8), windowed KV (wkv), shape-stable (stable), seven capacities
Toolchain base	apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 + the coreai-model-zoo python overlay
Toolchain	coreai-torch 0.4.1, coreai-core 1.0.0b2, coreai-opt 0.2.1, torch 2.9.0
Producer fingerprint	coreai-core 1.0.0b2 on every inner asset metadata.json
Weight format	int4 per-block-32 symmetric absmax (--lin-sym) — the ggml q4_0 grid the QAT checkpoint was trained on; int8 per-block-32 for dense2in-int8
Vocab	262,144

"QAT-unquantized" means QAT-trained, stored full width; the int4 rounding happens at export onto the grid training already targeted. That is why int4 is the primary recipe rather than a quality compromise — the int8 variant is an anchor, not the default.

tiered/ — the seven-rung ladder

One bundle, seven main entrypoints, each with its own pinned KV capacity:

4,096 / 8,192 / 16,384 / 32,768 / 65,536 / 131,072 / 262,144

The top rung is the checkpoint's own text_config.max_position_embeddings, so the ladder spans the architecture end to end.

It is nearly free, and that was measured before it was built. A pinned capacity costs no compiled bytes: at c4096 and c16384 the same graph's main.mlirb differs by 26 bytes on a 6.7 GB file, because capacity is a runtime state allocation and nothing else.

	main.mlirb	Δ vs 1 entrypoint	per extra entrypoint
1 entrypoint (c4096)	6,700,591,835 B	—	—
4 entrypoints (c4k…c128k)	6,702,885,614 B	+2,293,779 B	764,593 B
7 entrypoints (c4k…c256k)	6,705,129,126 B	+4,537,291 B	756,215 B
seven separate single-tier bundles	46,904,142,845 B	—	—
one tiered bundle instead	6,705,129,126 B	−37.44 GiB

Six extra capacities — including two that quadruple and octuple the previous top rung — cost 4.54 MB, 0.068% of the bundle, and dedup gets marginally better with more entrypoints: the four-rung build's cost law predicted 6,705,179,393 B for seven and the file came in 50,267 B under, 0.0007%.

The one idea that makes it work: the tiers are the same quantized module, copy.copy-ed so each carries its own capacity — a fresh __dict__ over the same _modules and _parameters, so every weight tensor is the identical object. Quantizing per tier would produce numerically identical but distinct tensors, and distinct tensors are what dedup cannot merge.

Build-time memory is the one place seven rungs are not free. Peak export RSS was 98.10 GB against 67.29 GB at four rungs and 67.28 GB at one. It is a build cost, not an artifact cost, but a machine with less than ~128 GB should not attempt a seven-rung export.

The cross-tier gates — bitwise, not "same SNR class"

Bottom rung against each higher rung, on identical weights, one S=1 decode step at a time, fp32, window 128, on the length-bounded Metal kernel's registered torch reference:

legs	layers	tokens	argmax mismatches	bitwise-identical	shape sigs
c4096 vs c16384, masked + bounded	48	220	0/220	220/220	1 / 1
c4096 vs c16384, masked + bounded	6	4,096	0/4096	4096/4096	1 / 1
c4096 vs c8192, bounded kernel	48	220	0/220	220/220	1 / 1
c4096 vs c65536, bounded kernel	48	220	0/220	220/220	1 / 1
c4096 vs c131072, bounded kernel	48	220	0/220	220/220	1 / 1
c4096 vs c262144, bounded kernel	48	220	0/220	220/220	1 / 1

max|Δlogit| is exactly 0.0 at every step of every pair, so the verdict is bitwise identity rather than a tolerance. Wraparound is exercised by construction in all of them (92 ring writes land on a used slot at 220 tokens against a 128 window), and each rung carries exactly one shape signature on each side — the property the whole design rests on, measured per rung rather than assumed.

The top pair is the one that mattered: c4096 against c262144 is a 64× capacity ratio and 4,227,858,432 B of KV the small side never allocates, and it is bit-for-bit identical anyway.

What these gates do not touch: they exercise the bounded kernel's torch reference, not its MSL (there is no Metal on the export box), and no tier has been run on device.

The per-tier cost, and what a 16 GB Mac needs to hold it

pinned KV is exact arithmetic from the state shapes: a global pair [8,1,1,C,512] at fp16 is 16,384 B per token of capacity, and the sliding ring [40,1,8,1024,256] is 0.3125 GiB and identical in all seven rungs — it does not scale with capacity at all.

rung	global KV	ring	pinned KV	steady wired	requirement (steady + 1.25 floor)	what a 16 GB Mac needs	32 GB
c4,096	0.0625	0.3125	0.3750	7.03	8.28	a good quiet window	untested
c8,192	0.1250	0.3125	0.4375	7.10	8.35	a good quiet window	untested
c16,384	0.2500	0.3125	0.5625	7.22 (measured)	8.47	a good quiet window	untested
c32,768	0.5000	0.3125	0.8125	7.47	8.72	a strong window	untested
c65,536	1.0000	0.3125	1.3125	7.97	9.22	a fresh-boot-class window	untested
c131,072	2.0000	0.3125	2.3125	8.97	10.22	24/32 GB machines	untested
c262,144	4.0000	0.3125	4.3125	10.97	12.22	24/32 GB machines	untested

All GiB. The forecast is blob + pinned KV(C) + R, where the tiered blob is 6.241 GiB and R = 0.419 GiB is the non-KV resident remainder, obtained by decomposing the measured 7.22 GiB steady plateau at c16384. The c16384 row therefore reproduces a measurement by construction, and every other row is that residual carried across.

The right-hand column is a window class, not a yes/no. Comparing each requirement against a machine's median afternoon would assert that the only window a machine ever offers is its median one, and this project's own record says otherwise. What the 16 GB reference machine has actually been measured to hold:

reading	GiB	what it is
ordinary reclaimable	4.5–7.3	the median afternoon
a real admission	7.62	an 8B-class bundle was admitted on a sustained 7.62 GiB over 33 s and produced tokens. Windows in the 7.6+ class occur on this machine
best-ever	9.97	five days old at the time of writing, never repeated — an outlier, and still not the top of the range

So c4k/c8k/c16k are the ordinary case for a 16 GB Mac left alone for a few minutes; c32k is plausible on a quiet machine and not something to promise; c64k is reachable and not routine; and c128k/c256k are honestly out of reach on 16 GB. The 32 GB column is "untested", not "yes" — this project has no 32 GB machine, reclaimable is not a fixed fraction of RAM, and writing "yes" there would be a guess wearing a measurement's clothes.

There is no separate prefill-workspace adder. The prompt-length term was measured directly — +0.018 GiB per 1,000 prompt tokens, within 18% of the KV-per-token arithmetic the table already charges, and 20× below the naive extrapolation that had been feared. So the requirements above stand as published, including c131,072 at 10.22 GiB. With int8 KV the top rung would fall to 9.07 GiB; that lever is real, arithmetic and not yet built.

64k is the rung the evidence asks for

A retrieval-side audit of eleven real report questions against the app's own graph, run at each rung with no model in the loop:

window	input budget	retrieval depth	GRAPH-BOUNDED	ceiling-capped
8,192	4,864	608	0	11
32,768	28,416	3,552	0	11
65,536	61,184	7,648	8	3
131,072	126,720	15,840	8	3
262,144	257,792	32,224	8	3

65,536 is where the ladder changes state. Eight of the eleven questions want 39,786–40,021 tokens; a 64k window offers 61,184, so all eight become graph-bounded — their entire scoped universe fits, and no larger window improves those answers. 128k and 256k add nothing for them at 1.00 and 3.00 GiB more KV. On evidence per gibibyte, 64k dominates 128k. The other three questions want ~1,052,400 tokens and are unsatisfiable at every rung; what 256k buys them is that their need is finally computed against the whole pool instead of a truncated one.

Selecting a rung

1. Do not read the bundle's language.max_context_length to size a prompt. On a tiered bundle it is the top rung (262,144). A session that acquired main_c4096 and budgeted against 262,144 would fail silently: the engine's stop is min(requested, max(0, maxContextLength − input.count)), so it returns 0, yields nil before producing anything, and the stream finishes empty. This is the one place a tiered bundle can go wrong quietly.
2. metadata.json carries language.capacity_tiers[] — per rung function, context_window_tokens, global_kv_bytes, ring_kv_bytes, total_kv_bytes — plus capacity_tier_notes.global_kv_bytes_per_token, so a host can price every rung without introspecting the graph.
3. If that field is ever absent, the rungs are still recoverable from function_map's main role because each entrypoint is named for its capacity — but the KV cost is not in the name, and a host must not invent one. An unpriced ladder should be climbed from the cheapest rung, not the largest.
4. A downgrade is a new session. Re-acquiring rebuilds the cache and therefore discards the KV, so "between-batch downgrade" is honest and "mid-session downgrade" is not.

tiered/ is decode-only: its function_map is {"main": [...]} with no prefill function, so prompts are prefilled one token at a time through the same main.

Requirements

• Apple silicon Mac, Core AI runtime. Mac-only; this does not fit an iPhone or iPad.

• Engine contract: 2 inputs — input_ids [1,1], position_ids [1,S] → logits, no per-step mask. That is the contract both engines accept, and the logits-capable sequential engine is what grammar-constrained decoding needs.

• dense2in* states: keyCache / valueCache Float16, 48 × 1 × 8 × ? × 512, dynamic sequence dim → GrowingKVCache (initial 256, doubling). KV cost 786,432 bytes per token — 3.22 GB at 4096, 12.9 GB at 16384. This dominates the dense2in* envelope.

• tiered/ states: a global pair [8,1,1,C,512] at the selected rung plus a capacity-invariant sliding ring [40,1,8,1024,256]. See the per-tier table.

• dense2in* memory envelope:

  	int4	int8
  Weights (main.mlirb)	8.71 GB	14.67 GB
  + KV at 4096	~11.9 GB	~17.9 GB
  + KV at 16384	~21.6 GB	~27.6 GB

  Against a 16 GB Mac's Metal recommendedMaxWorkingSetSize of ≈ 10.7 GB, the int4 bundle's affordable context is roughly 2,500 tokens, and the int8 bundle does not fit that machine at all. Minimum practical machine memory for dense2in*: 32 GB.

• The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1.

Measurements

No bundle in this repo has been measured on Apple silicon. What follows is measured on a sibling build — gemma4_12b_qat_decode_int4linsym_e4_msdpa_g8_wkv_stable_c16384, the single-capacity predecessor of tiered/, which is not published here — on a 16 GB M2 Pro under a watchdog. It is included because it is the only Gemma 4 12B evidence this project owns and because the tiered/ line inherits its graph.

	value
answered	1 / 11, and that one did not parse
killed by	the wired ceiling — 11.13 GiB against 11.08
wired peak / growth	11.13 / 8.41 GiB (baseline 2.72)
steady plateau (18 min, 2,316 samples)	9.94 wired, 7.22 growth = blob × 1.157
decode	5.37 tok/s
TTFT, 3,228-token prompt	629.93 s
swapped out	1.52 GiB
shape signatures across 6,962 passes	1, stable=true throughout
ANE co-tenant busy	1 / 652 samples — the 8.41 GiB is the model's own

Three things come out of it, and they point in different directions:

• Shape-stability held, under memory pressure severe enough to kill the run, and process footprint grew 5.0 MB across 6,962 passes. The wired memory is held driver-side, not in the process's own footprint. Respecialisation and leaks are both excluded.
• The 12B does not fit a 16 GB Mac at c16384. A steady 9.94 GiB wired against a machine whose ordinary reclaimable is 4.5–7.3 GiB is not a "wait for a quieter moment" case.
• Most of the 5.37 tok/s is thrash, not architecture. A dense 12B against an 8B-A1B activating ~1.5B parameters is perhaps a 6–8× compute ratio; the observed prefill ratio was 32×, on a machine that swapped 1.52 GiB and held reclaimable under 2.6 GiB for eighteen minutes. An unpressured estimate is prefill in the tens of tok/s — an inference from the ratio, not a measurement.

The plateau law is per-architecture, and this is what established that. Wired ÷ compiled blob: LFM2.5 8B-A1B MoE 1.092 peak (flat in prompt length); Gemma 4 E2B 1.101 against blob + PLE tables; Gemma 4 12B dense 1.157 steady / 1.348 peak, and not flat in prompt length. A single constant applied to every bundle under-charges the dense 12B by 0.33 GiB at its plateau and 1.51 GiB at its peak.

The c4096 re-export was run, and it bought 0.1875 GiB — the whole prize, because the sliding ring is five-sixths of the KV at c4096 and does not scale with capacity. Steady demand 7.22 → 7.03 GiB, requirement 8.28 GiB. It works, it is free, and it does not rescue the 12B on a 16 GB machine.

For context on the dense2in* bundles specifically, the only figure that exists is upstream's, on hardware not used here: approximately 33 tok/s decode on an M4 Max for the int4 12B. Read published Gemma 4 Q4 figures of 40–50 tok/s with care — those use Gemma 4's MTP speculative drafters, which Core AI does not implement. Compare plain-decode to plain-decode.

Usage

Swift Package Manager, via CoreAIKit — a community package, not affiliated with Apple, requiring macOS 27 beta:

.package(url: "https://github.com/john-rocky/coreai-kit", branch: "main")
// target dependency: .product(name: "CoreAIKit", package: "coreai-kit")

ModelID addresses a bundle as repo + path + revision; the subtree downloads from the Hub on first use and is cached afterwards:

import CoreAIKit

let model = ModelID(
    "visible-cx/Gemma-4-12B-CoreAI",
    path: "dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64")

var config = ChatSession.Configuration()
config.engineVariant = .sequential   // required for guided / grammar-constrained decoding
let chat = try await ChatSession(model: model, configuration: config)

Unlike the E2B/E4B bundles in this org, these are plain 2-input graphs: no PLE table sidecar, no static input buffers, and either engine will load them.

tiered/ needs a host that can name which of its seven main entrypoints to load and can size prompts against the selected rung rather than the bundle scalar. ChatSession takes one function name for the role today; driving a specific rung means loading the bundle directly and passing that entrypoint's name.

Digests

artifact	file	bytes	sha256
tiered/…_tiered_c4k…c256k	main.mlirb	6,705,129,126	1ffe4997cb88c0d84350a70d6be8fdaa50ffc60d07dea4aa71a6cb06fa5522f2

Taken independently on both ends of the transfer that moved it, and matching. Core AI .aimodel bundles are not byte-reproducible — the exporter is not deterministic even against itself — so integrity rests on digests of the exact published bytes rather than on a rebuild.

Status

Artifact	Status
tiered/…_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k	STAGED — built, gated bitwise across all seven rungs in torch, never run on device. Established: the ladder converts, every rung reads back its own pinned extents, cross-tier logits are bit-for-bit identical, and the marginal cost of a rung is 756,215 B. Not established: the compiled blob (6.241 GiB is a forecast carried from the c16384 graph constant, and a graph constant is not a monotone function of the IR); whether AIModel exposes seven function names to loadFunction(named:) on device; and whether the length-bounded Metal kernel holds its per-step cost as capacity grows — that kernel has never been timed on hardware, and it is the entire argument for the upper rungs.
dense2in/gemma4_12b_qat_decode_int4linsym	EXPERIMENTAL — built and fingerprinted, unmeasured. Gate: a Mac-side decode oracle against an fp32 reference, then a guided-decoding run on a real schema.
dense2in-pf64/…_pf64	EXPERIMENTAL — built and fingerprinted, unmeasured. Same weights as dense2in plus the S=64 prefill function, which was worth 7.7× on time-to-first-token when measured on the E2B sibling. Same gate.
dense2in-int8/…_int8lin_pf64	EXPERIMENTAL — built and fingerprinted, unmeasured. Same gate, plus a quality comparison against int4; it exists as an anchor for the case where int4 phrasing quality fails qualification, at 1.7× the resident weights.

No numerics gate has been run on device for any bundle in this repo. The producer fingerprint proves the toolchain, not the numerics.

License

Google publishes the upstream QAT checkpoint under Apache-2.0 with a license_link to the Gemma 4 license, and this repo mirrors that declaration. Use is governed by those terms and by the Gemma Prohibited Use Policy; the obligations travel with any redistribution of these bundles. The contribution here is the conversion and the qualification evidence, not the weights.