| --- |
| license: apache-2.0 |
| license_link: https://ai.google.dev/gemma/docs/gemma_4_license |
| base_model: google/gemma-4-12B-it-qat-q4_0-unquantized |
| base_model_relation: quantized |
| library_name: coreai |
| pipeline_tag: text-generation |
| tags: |
| - core-ai |
| - aimodel |
| - apple-silicon |
| - on-device |
| - coreai-kit |
| - quantized |
| - int4 |
| - int8 |
| - qat |
| - gemma4 |
| --- |
| |
| # Gemma 4 12B β Core AI (.aimodel) |
|
|
| `google/gemma-4-12B-it-qat-q4_0-unquantized` converted to Core AI `.aimodel` bundles for |
| Apple silicon by [visible-cx](https://huggingface.co/visible-cx). These are derivative |
| artifacts: Google's QAT-trained weights rounded onto the int4 (or int8) grid and re-expressed |
| as a Core AI graph. They load through Core AI on macOS and are not usable by PyTorch, GGUF or |
| MLX. Gemma 4 12B is a **dense** model β no Per-Layer Embeddings and no gather-table sidecar, |
| unlike the E2B and E4B bundles in this org. |
|
|
| The repo holds two generations of build. The three `dense2in*` folders are the original 2-input |
| dense exports. **`tiered/` is the current one**: a single bundle carrying **seven pinned |
| context capacities** from 4,096 to 262,144, gated bit-for-bit against each other, for 4.54 MB |
| more than a single-capacity bundle. |
|
|
| > **`tiered/` is STAGED, not qualified.** No tier has ever been run on a Mac, the multifunction |
| > bundle has never been loaded by the engine, and its compiled blob is a **forecast**. Every |
| > per-rung requirement below inherits that forecast. See |
| > [The seven-rung ladder](#tiered--the-seven-rung-ladder). |
|
|
| > **Known issue on the `dense2in*` bundles β memory growth on long generations.** The Core AI |
| > runtime retains one compiled specialization per sequence-length signature, and these decode |
| > graphs make sequence length load-bearing, so a long free-form generation retains roughly |
| > 80 MB of GPU allocations per generated token until the process exits. Measured on the sibling |
| > E2B/E4B exports (~81 MB/token at 3.3k of context, guided *and* unguided alike). The |
| > `tiered/` line does not have this defect β shape-stability was measured directly on its |
| > `wkv_stable` predecessor: **6,962 forward passes, one shape signature, `stable=true` on every |
| > one**, with total process-footprint growth of 5.0 MB across the whole run. |
| |
| ## Contents |
| |
| | Path | Bytes | Weights | Context | Functions | Status | |
| |---|---:|---|---|---|---| |
| | `tiered/gemma4_12b_qat_decode_int4linsym_e4_msdpa_g8_wkv_stable_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k` | `main.mlirb` 6,705,129,126 | int4 | **7 rungs, 4,096 β 262,144** | main Γ7 (decode only) | **STAGED** | |
| | `dense2in/gemma4_12b_qat_decode_int4linsym` | 8,746,180,229 | int4 | 4096 | main | EXPERIMENTAL | |
| | `dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64` | 8,747,078,381 | int4 | 4096 | main + prefill | EXPERIMENTAL | |
| | `dense2in-int8/gemma4_12b_qat_decode_int8lin_pf64` | 14,700,368,629 | int8 | 16384 | main + prefill | EXPERIMENTAL | |
|
|
| Each folder holds `<name>.aimodel/` (`main.mlirb`, `main.hash`, asset `metadata.json`), a |
| bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json` 32,169,626 B, |
| `tokenizer_config.json` 3,090 B, `chat_template.jinja` 18,683 B, `generation_config.json`). |
| With the shared tokenizer the staged `tiered/` bundle is **6.27 GiB** on disk. |
|
|
| `dense2in-pf64`'s weights are bit-identical to `dense2in`'s; the multifunction build adds an |
| S=64 `prefill` entrypoint for under a megabyte, because the exporter deduplicates weights |
| across entrypoints. |
|
|
| **Stop token:** every bundle declares `eos_token = "<turn|>"` (id 106), the turn terminator |
| Gemma 4 emits. A host that stops on the raw upstream `<eos>` instead will overrun every reply. |
|
|
| **Manifest context is metadata only β except on `tiered/`.** On the `dense2in*` bundles |
| `--max-ctx` sets `language.max_context_length` and nothing else, so the numbers above can be |
| raised or lowered without a re-export (whether the result fits in memory is a separate |
| question). On `tiered/` the capacity is a real, pinned graph extent per entrypoint, and |
| `language.max_context_length` reports the **top rung** β see the warning under |
| [Selecting a rung](#selecting-a-rung). |
|
|
| ## Provenance |
|
|
| | | | |
| |---|---| |
| | Base checkpoint | `google/gemma-4-12B-it-qat-q4_0-unquantized` (ungated) | |
| | Zoo recipe | `gemma-4-12b`, `status = "verified"` β the dense recipe | |
| | Recipe (`dense2in`) | that recipe with `int4lin --lin-sym` and **no** `--metal-sdpa`, `--max-ctx 4096` | |
| | Recipe (`dense2in-pf64`) | as `dense2in`, multifunction, via `export_gemma4_dense_pf.py` | |
| | Recipe (`dense2in-int8`) | `int8lin` (symmetric-with-clipping, per-block-32), multifunction, same 2-input dense graph | |
| | Recipe (`tiered/`) | `export_gemma4_12b_tiered.py` β `int4lin --lin-sym`, `--quant-embed int4` (`e4`), `--metal-sdpa` split-8 (`msdpa_g8`), windowed KV (`wkv`), shape-stable (`stable`), seven capacities | |
| | Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay | |
| | Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` | |
| | Producer fingerprint | `coreai-core 1.0.0b2` on every inner asset `metadata.json` | |
| | Weight format | int4 per-block-32 symmetric absmax (`--lin-sym`) β the ggml q4_0 grid the QAT checkpoint was trained on; int8 per-block-32 for `dense2in-int8` | |
| | Vocab | 262,144 | |
| |
| "QAT-unquantized" means QAT-*trained*, stored full width; the int4 rounding happens at export |
| onto the grid training already targeted. That is why int4 is the primary recipe rather than a |
| quality compromise β the int8 variant is an anchor, not the default. |
| |
| ## `tiered/` β the seven-rung ladder |
| |
| One bundle, seven `main` entrypoints, each with its own pinned KV capacity: |
| |
| 4,096 / 8,192 / 16,384 / 32,768 / 65,536 / 131,072 / 262,144 |
| |
| The top rung is the checkpoint's own `text_config.max_position_embeddings`, so the ladder spans |
| the architecture end to end. |
|
|
| **It is nearly free, and that was measured before it was built.** A pinned capacity costs no |
| compiled bytes: at c4096 and c16384 the same graph's `main.mlirb` differs by **26 bytes** on a |
| 6.7 GB file, because capacity is a runtime state allocation and nothing else. |
|
|
| | | `main.mlirb` | Ξ vs 1 entrypoint | per extra entrypoint | |
| |---|---:|---:|---:| |
| | 1 entrypoint (c4096) | 6,700,591,835 B | β | β | |
| | 4 entrypoints (c4kβ¦c128k) | 6,702,885,614 B | +2,293,779 B | 764,593 B | |
| | **7 entrypoints (c4kβ¦c256k)** | **6,705,129,126 B** | **+4,537,291 B** | **756,215 B** | |
| | seven separate single-tier bundles | 46,904,142,845 B | β | β | |
| | **one tiered bundle instead** | **6,705,129,126 B** | **β37.44 GiB** | | |
|
|
| Six extra capacities β including two that quadruple and octuple the previous top rung β cost |
| **4.54 MB, 0.068% of the bundle**, and dedup gets marginally *better* with more entrypoints: |
| the four-rung build's cost law predicted 6,705,179,393 B for seven and the file came in |
| **50,267 B under, 0.0007%**. |
|
|
| The one idea that makes it work: the tiers are the **same quantized module**, `copy.copy`-ed so |
| each carries its own capacity β a fresh `__dict__` over the same `_modules` and `_parameters`, |
| so every weight tensor is the identical object. Quantizing per tier would produce numerically |
| identical but *distinct* tensors, and distinct tensors are what dedup cannot merge. |
|
|
| **Build-time memory is the one place seven rungs are not free.** Peak export RSS was |
| **98.10 GB** against 67.29 GB at four rungs and 67.28 GB at one. It is a build cost, not an |
| artifact cost, but **a machine with less than ~128 GB should not attempt a seven-rung export**. |
|
|
| ### The cross-tier gates β bitwise, not "same SNR class" |
|
|
| Bottom rung against each higher rung, on identical weights, one S=1 decode step at a time, |
| fp32, window 128, on the length-bounded Metal kernel's registered torch reference: |
|
|
| | legs | layers | tokens | argmax mismatches | bitwise-identical | shape sigs | |
| |---|---:|---:|---:|---:|---:| |
| | c4096 vs c16384, masked + bounded | 48 | 220 | 0/220 | 220/220 | 1 / 1 | |
| | c4096 vs c16384, masked + bounded | 6 | 4,096 | 0/4096 | 4096/4096 | 1 / 1 | |
| | **c4096 vs c8192**, bounded kernel | 48 | 220 | **0/220** | **220/220** | **1 / 1** | |
| | **c4096 vs c65536**, bounded kernel | 48 | 220 | **0/220** | **220/220** | **1 / 1** | |
| | c4096 vs c131072, bounded kernel | 48 | 220 | 0/220 | 220/220 | 1 / 1 | |
| | **c4096 vs c262144**, bounded kernel | 48 | 220 | **0/220** | **220/220** | **1 / 1** | |
|
|
| `max|Ξlogit|` is **exactly 0.0** at every step of every pair, so the verdict is bitwise |
| identity rather than a tolerance. Wraparound is exercised by construction in all of them |
| (92 ring writes land on a used slot at 220 tokens against a 128 window), and **each rung |
| carries exactly one shape signature on each side** β the property the whole design rests on, |
| measured per rung rather than assumed. |
|
|
| The top pair is the one that mattered: c4096 against c262144 is a 64Γ capacity ratio and |
| 4,227,858,432 B of KV the small side never allocates, and it is bit-for-bit identical anyway. |
|
|
| **What these gates do not touch:** they exercise the bounded kernel's *torch reference*, not |
| its MSL (there is no Metal on the export box), and **no tier has been run on device.** |
|
|
| ### The per-tier cost, and what a 16 GB Mac needs to hold it |
|
|
| `pinned KV` is exact arithmetic from the state shapes: a global pair `[8,1,1,C,512]` at fp16 is |
| **16,384 B per token of capacity**, and the sliding ring `[40,1,8,1024,256]` is **0.3125 GiB |
| and identical in all seven rungs** β it does not scale with capacity at all. |
|
|
| | rung | global KV | ring | **pinned KV** | steady wired | **requirement** (steady + 1.25 floor) | what a 16 GB Mac needs | 32 GB | |
| |---|---:|---:|---:|---:|---:|---|---| |
| | c4,096 | 0.0625 | 0.3125 | **0.3750** | 7.03 | **8.28** | **a good quiet window** | untested | |
| | c8,192 | 0.1250 | 0.3125 | **0.4375** | 7.10 | **8.35** | **a good quiet window** | untested | |
| | c16,384 | 0.2500 | 0.3125 | **0.5625** | **7.22** *(measured)* | **8.47** | **a good quiet window** | untested | |
| | c32,768 | 0.5000 | 0.3125 | **0.8125** | 7.47 | **8.72** | **a strong window** | untested | |
| | c65,536 | 1.0000 | 0.3125 | **1.3125** | 7.97 | **9.22** | **a fresh-boot-class window** | untested | |
| | c131,072 | 2.0000 | 0.3125 | **2.3125** | 8.97 | **10.22** | 24/32 GB machines | untested | |
| | c262,144 | 4.0000 | 0.3125 | **4.3125** | 10.97 | **12.22** | 24/32 GB machines | untested | |
|
|
| All GiB. The forecast is `blob + pinned KV(C) + R`, where the tiered blob is **6.241 GiB** and |
| `R = 0.419 GiB` is the non-KV resident remainder, obtained by *decomposing* the measured |
| 7.22 GiB steady plateau at c16384. The c16384 row therefore reproduces a measurement by |
| construction, and every other row is that residual carried across. |
|
|
| **The right-hand column is a window class, not a yes/no.** Comparing each requirement against a |
| machine's *median afternoon* would assert that the only window a machine ever offers is its |
| median one, and this project's own record says otherwise. What the 16 GB reference machine has |
| actually been measured to hold: |
|
|
| | reading | GiB | what it is | |
| |---|---:|---| |
| | ordinary reclaimable | **4.5β7.3** | the median afternoon | |
| | **a real admission** | **7.62** | an 8B-class bundle was admitted on a sustained 7.62 GiB over 33 s and produced tokens. Windows in the 7.6+ class **occur** on this machine | |
| | best-ever | **9.97** | five days old at the time of writing, never repeated β an outlier, and still not the top of the range | |
|
|
| So c4k/c8k/c16k are the ordinary case for a 16 GB Mac left alone for a few minutes; c32k is |
| plausible on a quiet machine and not something to promise; c64k is *reachable* and not |
| *routine*; and c128k/c256k are honestly out of reach on 16 GB. **The 32 GB column is |
| "untested", not "yes"** β this project has no 32 GB machine, reclaimable is not a fixed |
| fraction of RAM, and writing "yes" there would be a guess wearing a measurement's clothes. |
|
|
| **There is no separate prefill-workspace adder.** The prompt-length term was measured directly |
| β **+0.018 GiB per 1,000 prompt tokens**, within 18% of the KV-per-token arithmetic the table |
| already charges, and 20Γ below the naive extrapolation that had been feared. So the |
| requirements above stand as published, including **c131,072 at 10.22 GiB**. With int8 KV the |
| top rung would fall to 9.07 GiB; that lever is real, arithmetic and not yet built. |
|
|
| ### 64k is the rung the evidence asks for |
|
|
| A retrieval-side audit of eleven real report questions against the app's own graph, run at each |
| rung with no model in the loop: |
|
|
| | window | input budget | retrieval depth | GRAPH-BOUNDED | ceiling-capped | |
| |---|---:|---:|---:|---:| |
| | 8,192 | 4,864 | 608 | **0** | 11 | |
| | 32,768 | 28,416 | 3,552 | **0** | 11 | |
| | **65,536** | **61,184** | **7,648** | **8** | **3** | |
| | 131,072 | 126,720 | 15,840 | **8** | 3 | |
| | 262,144 | 257,792 | 32,224 | **8** | 3 | |
|
|
| **65,536 is where the ladder changes state.** Eight of the eleven questions want |
| 39,786β40,021 tokens; a 64k window offers 61,184, so all eight become graph-bounded β their |
| entire scoped universe fits, and no larger window improves those answers. 128k and 256k add |
| nothing for them at 1.00 and 3.00 GiB more KV. On evidence per gibibyte, **64k dominates 128k**. |
| The other three questions want ~1,052,400 tokens and are unsatisfiable at every rung; what |
| 256k buys them is that their *need* is finally computed against the whole pool instead of a |
| truncated one. |
|
|
| ### Selecting a rung |
|
|
| 1. **Do not read the bundle's `language.max_context_length` to size a prompt.** On a tiered |
| bundle it is the **top rung** (262,144). A session that acquired `main_c4096` and budgeted |
| against 262,144 would fail *silently*: the engine's stop is |
| `min(requested, max(0, maxContextLength β input.count))`, so it returns 0, yields nil before |
| producing anything, and the stream finishes empty. This is the one place a tiered bundle can |
| go wrong quietly. |
| 2. `metadata.json` carries `language.capacity_tiers[]` β per rung `function`, |
| `context_window_tokens`, `global_kv_bytes`, `ring_kv_bytes`, `total_kv_bytes` β plus |
| `capacity_tier_notes.global_kv_bytes_per_token`, so a host can price every rung without |
| introspecting the graph. |
| 3. If that field is ever absent, the rungs are still recoverable from `function_map`'s `main` |
| role because each entrypoint is named for its capacity β but the **KV cost is not in the |
| name**, and a host must not invent one. An unpriced ladder should be climbed from the |
| *cheapest* rung, not the largest. |
| 4. **A downgrade is a new session.** Re-acquiring rebuilds the cache and therefore discards the |
| KV, so "between-batch downgrade" is honest and "mid-session downgrade" is not. |
|
|
| `tiered/` is **decode-only**: its `function_map` is `{"main": [...]}` with no `prefill` |
| function, so prompts are prefilled one token at a time through the same `main`. |
|
|
| ## Requirements |
|
|
| - **Apple silicon Mac**, Core AI runtime. Mac-only; this does not fit an iPhone or iPad. |
| - **Engine contract: 2 inputs** β `input_ids [1,1]`, `position_ids [1,S]` β logits, no per-step |
| mask. That is the contract both engines accept, and the logits-capable sequential engine is |
| what grammar-constrained decoding needs. |
| - **`dense2in*` states:** `keyCache` / `valueCache` `Float16, 48 Γ 1 Γ 8 Γ ? Γ 512`, dynamic |
| sequence dim β `GrowingKVCache` (initial 256, doubling). **KV cost 786,432 bytes per token** |
| β 3.22 GB at 4096, 12.9 GB at 16384. This dominates the `dense2in*` envelope. |
| - **`tiered/` states:** a global pair `[8,1,1,C,512]` at the selected rung plus a |
| capacity-invariant sliding ring `[40,1,8,1024,256]`. See the per-tier table. |
| - **`dense2in*` memory envelope:** |
| |
| | | int4 | int8 | |
| |---|---|---| |
| | Weights (`main.mlirb`) | 8.71 GB | 14.67 GB | |
| | + KV at 4096 | ~11.9 GB | ~17.9 GB | |
| | + KV at 16384 | ~21.6 GB | ~27.6 GB | |
| |
| Against a 16 GB Mac's Metal `recommendedMaxWorkingSetSize` of β **10.7 GB**, the int4 |
| bundle's affordable context is roughly **2,500 tokens**, and the int8 bundle does not fit |
| that machine at all. **Minimum practical machine memory for `dense2in*`: 32 GB.** |
| - The bundle manifest declares `runtime_env COREAI_CHUNK_THRESHOLD=1`. |
|
|
| ## Measurements |
|
|
| **No bundle in this repo has been measured on Apple silicon.** What follows is measured on a |
| *sibling* build β `gemma4_12b_qat_decode_int4linsym_e4_msdpa_g8_wkv_stable_c16384`, the |
| single-capacity predecessor of `tiered/`, which is not published here β on a 16 GB M2 Pro under |
| a watchdog. It is included because it is the only Gemma 4 12B evidence this project owns and |
| because the `tiered/` line inherits its graph. |
|
|
| | | value | |
| |---|---:| |
| | answered | **1 / 11**, and that one did not parse | |
| | killed by | the **wired ceiling** β 11.13 GiB against 11.08 | |
| | wired peak / growth | 11.13 / **8.41** GiB (baseline 2.72) | |
| | steady plateau (18 min, 2,316 samples) | 9.94 wired, **7.22** growth = blob Γ **1.157** | |
| | decode | **5.37 tok/s** | |
| | TTFT, 3,228-token prompt | **629.93 s** | |
| | swapped out | 1.52 GiB | |
| | shape signatures across 6,962 passes | **1**, `stable=true` throughout | |
| | ANE co-tenant busy | 1 / 652 samples β the 8.41 GiB is the model's own | |
|
|
| Three things come out of it, and they point in different directions: |
|
|
| - **Shape-stability held**, under memory pressure severe enough to kill the run, and process |
| footprint grew 5.0 MB across 6,962 passes. The wired memory is held driver-side, not in the |
| process's own footprint. Respecialisation and leaks are both excluded. |
| - **The 12B does not fit a 16 GB Mac at c16384.** A steady 9.94 GiB wired against a machine |
| whose ordinary reclaimable is 4.5β7.3 GiB is not a "wait for a quieter moment" case. |
| - **Most of the 5.37 tok/s is thrash, not architecture.** A dense 12B against an 8B-A1B |
| activating ~1.5B parameters is perhaps a 6β8Γ compute ratio; the observed prefill ratio was |
| 32Γ, on a machine that swapped 1.52 GiB and held reclaimable under 2.6 GiB for eighteen |
| minutes. An unpressured estimate is prefill in the tens of tok/s β an inference from the |
| ratio, **not a measurement**. |
|
|
| **The plateau law is per-architecture, and this is what established that.** Wired Γ· compiled |
| blob: LFM2.5 8B-A1B MoE **1.092** peak (flat in prompt length); Gemma 4 E2B **1.101** against |
| `blob + PLE tables`; Gemma 4 12B dense **1.157 steady / 1.348 peak**, and *not* flat in prompt |
| length. A single constant applied to every bundle under-charges the dense 12B by 0.33 GiB at |
| its plateau and 1.51 GiB at its peak. |
|
|
| **The c4096 re-export was run, and it bought 0.1875 GiB** β the whole prize, because the |
| sliding ring is five-sixths of the KV at c4096 and does not scale with capacity. Steady demand |
| 7.22 β 7.03 GiB, requirement 8.28 GiB. It works, it is free, and it does not rescue the 12B on |
| a 16 GB machine. |
|
|
| For context on the `dense2in*` bundles specifically, the only figure that exists is upstream's, |
| on hardware not used here: approximately **33 tok/s decode on an M4 Max** for the int4 12B. |
| Read published Gemma 4 Q4 figures of 40β50 tok/s with care β those use Gemma 4's MTP |
| speculative drafters, which Core AI does not implement. Compare plain-decode to plain-decode. |
|
|
| ## Usage |
|
|
| Swift Package Manager, via [CoreAIKit](https://github.com/john-rocky/coreai-kit) β a community |
| package, not affiliated with Apple, requiring macOS 27 beta: |
|
|
| ```swift |
| .package(url: "https://github.com/john-rocky/coreai-kit", branch: "main") |
| // target dependency: .product(name: "CoreAIKit", package: "coreai-kit") |
| ``` |
|
|
| `ModelID` addresses a bundle as **repo + path + revision**; the subtree downloads from the Hub |
| on first use and is cached afterwards: |
|
|
| ```swift |
| import CoreAIKit |
| |
| let model = ModelID( |
| "visible-cx/Gemma-4-12B-CoreAI", |
| path: "dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64") |
| |
| var config = ChatSession.Configuration() |
| config.engineVariant = .sequential // required for guided / grammar-constrained decoding |
| let chat = try await ChatSession(model: model, configuration: config) |
| ``` |
|
|
| Unlike the E2B/E4B bundles in this org, these are plain 2-input graphs: **no PLE table sidecar, |
| no static input buffers**, and either engine will load them. |
|
|
| `tiered/` needs a host that can name *which* of its seven `main` entrypoints to load and can |
| size prompts against the selected rung rather than the bundle scalar. `ChatSession` takes one |
| function name for the role today; driving a specific rung means loading the bundle directly |
| and passing that entrypoint's name. |
|
|
| ## Digests |
|
|
| | artifact | file | bytes | sha256 | |
| |---|---|---:|---| |
| | `tiered/β¦_tiered_c4kβ¦c256k` | `main.mlirb` | 6,705,129,126 | `1ffe4997cb88c0d84350a70d6be8fdaa50ffc60d07dea4aa71a6cb06fa5522f2` | |
|
|
| Taken independently on both ends of the transfer that moved it, and matching. Core AI |
| `.aimodel` bundles are not byte-reproducible β the exporter is not deterministic even against |
| itself β so integrity rests on digests of the exact published bytes rather than on a rebuild. |
|
|
| ## Status |
|
|
| | Artifact | Status | |
| |---|---| |
| | `tiered/β¦_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k` | **STAGED β built, gated bitwise across all seven rungs in torch, never run on device.** Established: the ladder converts, every rung reads back its own pinned extents, cross-tier logits are bit-for-bit identical, and the marginal cost of a rung is 756,215 B. Not established: the compiled blob (6.241 GiB is a **forecast** carried from the c16384 graph constant, and a graph constant is not a monotone function of the IR); whether `AIModel` exposes seven function names to `loadFunction(named:)` on device; and whether the length-bounded Metal kernel holds its per-step cost as capacity grows β that kernel has never been timed on hardware, and it is the entire argument for the upper rungs. | |
| | `dense2in/gemma4_12b_qat_decode_int4linsym` | **EXPERIMENTAL** β built and fingerprinted, unmeasured. Gate: a Mac-side decode oracle against an fp32 reference, then a guided-decoding run on a real schema. | |
| | `dense2in-pf64/β¦_pf64` | **EXPERIMENTAL** β built and fingerprinted, unmeasured. Same weights as `dense2in` plus the S=64 prefill function, which was worth 7.7Γ on time-to-first-token when measured on the E2B sibling. Same gate. | |
| | `dense2in-int8/β¦_int8lin_pf64` | **EXPERIMENTAL** β built and fingerprinted, unmeasured. Same gate, plus a quality comparison against int4; it exists as an anchor for the case where int4 phrasing quality fails qualification, at 1.7Γ the resident weights. | |
|
|
| **No numerics gate has been run on device for any bundle in this repo.** The producer |
| fingerprint proves the toolchain, not the numerics. |
|
|
| ## License |
|
|
| Google publishes the upstream QAT checkpoint under **Apache-2.0** with a |
| `license_link` to the [Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license), and |
| this repo mirrors that declaration. Use is governed by those terms and by the |
| [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy); the |
| obligations travel with any redistribution of these bundles. The contribution here is the |
| conversion and the qualification evidence, not the weights. |
|
|
| ## tiered-v3/ β an fp16 vocab path, seven rungs, prefill and verify |
|
|
| The Quality-tier build, and the vocab path carries no int4 at all. |
|
|
| **The embedding is fp16 and bit-exact to the checkpoint.** Measured on the real tensor: the |
| fp16 cast of the bf16 table is **lossless β worst |delta| 0.000e+00** β because bf16's 8 |
| mantissa bits fit inside fp16's 11 and the table maxes at 1.3906, nowhere near fp16's range |
| limits. Not an approximation of the checkpoint's values; the values themselves. |
|
|
| That replaces a 4-bit embedding whose cost had never been priced against the checkpoint. When |
| it was, both candidates came out around 15β17 dB end-to-end (the previously published `_e4` |
| build measures **14.794 dB / 254-of-256 top-1**), which is why this revision spends the bytes. |
|
|
| **The head is fp16 too**, because Gemma ties embedding and head: leaving the head int4 would |
| keep a `blockwise_shift_scale` reconstructing a full `[262144, 3840]` fp16 table for the |
| matmul. With both fp16 **no full-table reconstruction remains anywhere on the vocab path**. |
|
|
| **Entrypoints.** Seven capacity rungs (4k, 8k, 16k, 32k, 64k, 128k, 256k), each with a |
| last-row-only `prefill_c<N>` at S=64, plus **`verify_c262144` at S=8 returning FULL |
| `[1, 8, 262144]` logits** β the shape a prompt-lookup or speculative loop needs to check eight |
| drafted tokens at once. Every entrypoint has zero symbolic extents. |
| |
| **Not yet run on a Mac.** No token generated, no decode speed claimed. |
| |
| ### `tiered-v3/gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k` β 8,190,509,056 bytes |
| |
| gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/main.hash 32 B 2d9b6fde48da0ec177fadaf5f90c53bf86d5dcb5cbaa35397713fa89a005e50b |
| gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/main.mlirb 8,158,295,378 B 57bf5d507ca70e001c58bf72ea20308b232cb56835bfe3e5fcde13ac017cb34d |
| gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/metadata.json 105 B 93893e9d87ff83172d1ca84eec61e939233e4956b3a677b49f873463c0e2499b |
| metadata.json 4,174 B c782edb705134877ac58bac96c9ffe7e8010b24dbac3c929ed91cb4073a55fda |
| tier_report.json 17,968 B b33ebdeb508d561805930a85f34acc2148dff2d91f20cab6e1ff2e2619229c19 |
| tokenizer/chat_template.jinja 18,683 B ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4 |
| tokenizer/tokenizer.json 32,169,626 B cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f |
| tokenizer/tokenizer_config.json 3,090 B c3fd34e321798e184e53894aaa1d8c4b37d08daf1fb1826dbceb3dcda0c62d06 |
| |
