README: hybrid-mf64 bundles

README.md

@@ -282,3 +282,102 @@ model and **the Gemma terms apply downstream**, which is why this repo declares
 [Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
 travel with any redistribution of this bundle. The contribution here is the port and the
 recipe, not the weights.
+
+
+## `hybrid-mf64/` bundles
+
+> **EXPERIMENTAL — not qualified.** These bundles have not passed a Mac-side
+> oracle/parity gate or a device benchmark. Do not route production traffic to
+> them until they have.
+
+**What this is.** The 26B-A4B with a `prefill` entrypoint, from one copy of the weights
+the published `gpu-pipelined/` decode bundle already ships. Two things had to be
+per-entrypoint at once, and both are:
+
+* **Experts.** `BatchedMetalSwitchGLU` sorts the `S*k` (token, expert) pairs of a 64-token
+  chunk by expert id so each expert slab is read once, and falls through to
+  `MetalSwitchGLU.forward` verbatim whenever `b*s == 1`. One metalization, both
+  entrypoints, **decode arithmetic unchanged** — `main` traces the identical q=1 gather
+  matvec the published decode bundle carries.
+* **Attention.** The 26B inherits the dense 12B/31B full-attention layers, so it needs the
+  flash-decode Metal kernel for the scratch-heap reason, and that kernel is structurally
+  q=1. `SplitSDPA` branches on `query.shape[2]`, which is a concrete int in each trace, so
+  `main` gets the kernel and `prefill` gets the MPSGraph composite (query axis +
+  `is_causal=True` intra-chunk mask). Same technique as the 31B `hybrid-pf64/` bundle.
+
+Both kernels are registered on one converter before either `add_pytorch_module`.
+
+**This is `mf64`, not `mf64-tp`, and it cannot be `-tp` today.** The token-tiled prefill
+kernel (`moe_metal_tiled`, the 8B-A1B `mf64-tp` experiment) is **sym8-only** —
+`TiledBatchedMetalSwitchGLU.__init__` raises for any other scheme. The 26B ships **aff4**
+experts, and that is not incidental: the checkpoint is QAT-trained on the ggml q4_0
+per-block-32 grid, which an affine int4 block-32 grid represents exactly. An aff4 tiled
+kernel is future work, gated on two things — the 8B `mf64-tp` Mac verdict proving the
+tiling principle at all, and then an aff4 equality proof of the kind `moek_tilegate.py`
+gives the sym8 kernel. What this bundle uses instead is the **shipped** batched path,
+the one behind the 8B `mf64` bundle that has a Mac measurement.
+
+**Measured on the conversion box (Linux x86_64, no GPU).** Export wall 305 s; peak RSS
+**87.5 GB** (well under the 12B's 68.7 → 31B's 148.9 band this was budgeted against, and
+never near the 75% kill line); bundle 17,580,916,612 B; producer `coreai-core 1.0.0b2`;
+ctx 16384; `function_map {"main": ["main", "prefill"]}`. Trace accounting: all 5 full
+layers took the metal branch at S=1 **and** the composite branch at S=64, 5/5 both ways —
+the export fails if either count is 0. 30 MoE layers metalized. Graph bytecode declares
+`input_ids`, `position_ids`, `keyCache`, `valueCache`, `logits`, `prefill`, 90
+`gather_qmm_int4aff` call sites (30 layers x gate/up/down) and
+`gemma4_dense_full_sdpa_occ8` — **2 inputs, no `mask`**, which is the arity
+`CoreAISequentialEngine` requires.
+
+**Numerics gate (box-side, `gate_hybrid_sdpa.json`).** The 26B is a *third* distinct
+block-GQA mapping and was gated as its own case: 16 query heads, **2** global KV heads
+replicated across 8 cache slots, so the kernel's `kv = h / (H / slots)` = `h//2` must land
+on real head `(h//2)//4 = h//8`. Kernel reference vs the composite at those exact shapes:
+max abs 2.50e-2 (S=137) and 1.17e-2 (S=1024) — the composite's own fp16 error. Against an
+fp32 ground truth the kernel is the more accurate of the two: **9.74e-4 / 74.9 dB** at
+S=137 and 9.76e-4 / 75.1 dB at S=1024, versus the composite's 53.0 and 57.9 dB. The 12B
+(1 global head, rep 8) and 31B (4 global heads, rep 4 over 16 slots) rows are in the same
+file as controls; a mapping bug that cancelled in those would show here. `MetalDenseFullSDPA`
+also evaluates in eager at 26B dims and matches its own reference exactly (max abs 0.0).
+This gates the mapping and the scale, not the Metal source (no GPU on the box).
+
+**Tier — read the arithmetic, not the bundle size.** KV is `2 x 30 layers x 8 KV slots x
+512 head_dim x fp16` = **491,520 B/token**:
+
+| Context | KV | + 17.6 GB weights | Verdict |
+|---|---|---|---|
+| 4096 | 2.01 GB | 19.6 GB | 32 GB machine |
+| 8192 | 4.03 GB | 21.6 GB | 32 GB machine, marginal against a ~21-24 GB working set |
+| 16384 (this manifest) | 8.05 GB | 25.6 GB | **64 GB machine** |
+
+So: a 32 GB Mac can run it at shallow context, the manifest's full 16384 wants 64 GB, and
+no 16 GB machine runs it at any context.
+
+**EXPERIMENTAL — unqualified until a big-machine measurement exists.** Nothing here has
+produced a token. In priority order:
+
+1. *Does it load and decode?* Decode should be indistinguishable from the published
+   `gpu-pipelined/` bundle: same weights, same q=1 gather kernel, same flash-decode SDPA.
+   A decode regression against that bundle would mean the batched wrapper is not falling
+   through at `b*s == 1` as its source says it does.
+2. *Prefill throughput.* The **falsifiable prediction**: the batched kernel's anchor is the
+   8B-A1B `mf64` bundle's measured **6.3 ms/prompt-token**, and this bundle should land in
+   that mechanism's class rather than the 21.5 ms/token of a decode-only bundle walking
+   the prompt one token at a time. Do not read 6.3 ms as the expected number for this
+   model: the 26B activates ~4B parameters per token against the 8B-A1B's ~1B, and its
+   dense MLP runs in parallel with the expert branch, so a figure several times 6.3 ms
+   would still confirm the mechanism. What would falsify it is prefill landing at or above
+   the per-token cost of decode — that would mean the sorted grouped-GEMM path is not
+   engaging and the chunk is being walked serially.
+3. *Chunk parity.* A prompt walked in 64-token chunks must give the same logits as the same
+   prompt walked token by token. Two independent reasons to check it here rather than
+   assume: the entrypoints use different attention implementations, and the expert path
+   does a sort/unsort round trip that the q=1 path does not.
+4. *Compare against the dense 31B on the same machine.* That is the whole case for this
+   model — 17.6 GB and ~4B active per token against 20 GB and 31B dense.
+
+Nothing already published was touched. `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8`
+is unchanged and remains the only 26B artifact anyone has run.
+
+| Bundle | Source checkpoint | Context | Functions | Bytes | Producer | Created |
+|---|---|---|---|---|---|---|
+| `hybrid-mf64/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8_pf64` | google/gemma-4-26B-A4B-it-qat-q4_0-unquantized | 16384 | main+prefill | 17,580,916,612 | coreai-core 1.0.0b2 | 20260818T141434Z |