GLM-4.7-Flash: gather_qmm kernel card (2.6x, clean)

README.md

@@ -1,80 +1,52 @@
 ---
 license: mit
 base_model: zai-org/GLM-4.7-Flash
-tags:
-  - coreai
-  - aimodel
-  - apple-silicon
-  - on-device
-  - glm
-  - mixture-of-experts
-  - mla
 pipeline_tag: text-generation
+library_name: core-ai
+tags:
+- core-ai
+- coreml
+- apple
+- moe
+- mla
+- on-device
+- metal
 ---
 
-# GLM-4.7-Flash — Apple Core AI (`.aimodel`)
-
-**zai-org's GLM-4.7-Flash converted to Apple's Core AI** (the Core ML successor announced at
-WWDC26), ready to run on macOS 27. This is the **first model on the zoo with Multi-head Latent
-Attention (MLA)** — the DeepSeek-V3 attention family — riding Apple's **`coreai-pipelined` GPU
-engine** with zero custom kernels.
-
-> Requires the macOS 27 beta (Core AI ships with the OS). **Mac-only** (~30 GB int8 + a
-> materialized full-MHA KV cache). Conversion code, parity ladder, and the Swift runner:
-> **[coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo)** (see
-> [`zoo/glm-4.7-flash.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/glm-4.7-flash.md)).
+# GLM-4.7-Flash — Core AI (`gather_qmm` kernel, 2.6× faster)
 
-## Architecture
+Apple **Core AI** (`.aimodel`) conversion of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
+(text decoder): MLA attention + a **64-expert top-4 sparse MoE** (+ non-gated shared expert).
+~30B total / **~3B active per token** — a strong local coder.
 
-**~30B parameters, ~3B active per token** — **MLA attention on all 47 layers + a sparse MoE FFN**:
+Part of the community Core AI model zoo: **https://github.com/john-rocky/coreai-model-zoo**
+(full card: [`zoo/glm-4.7-flash.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/glm-4.7-flash.md)).
 
-- **MLA** (heads 20, `q_lora_rank` 768, `kv_lora_rank` 512, `qk_nope` 192 + decoupled `qk_rope`
-  64 = head_dim 256, `v_head_dim` 256, RoPE θ=1e6 interleaved), authored in the **naive /
-  materialized** form: latent caches are projected up to per-head q/k/v, decoupled RoPE is
-  applied to the 64-dim rope slice, and the full per-head `[20, 256]` Q/K/V run through Apple's
-  standard SDPA composite. The KV cache stores the materialized per-head K/V.
-- **MoE FFN** (layers 1–46; layer 0 dense, `first_k_dense_replace=1`): 64 routed experts, top-4,
-  plus one **non-gated shared expert**. `noaux_tc` routing reduces (n_group=1) to sigmoid scoring
-  + selection-only bias correction; experts ride Apple's `SwitchGLU`/`GatherMM` composite.
+## The `gather_qmm` kernel — 20.3 → 52.4 tok/s (2.6×)
 
-## Measured (macOS 27 beta, M4 Max 128 GB, release `llm-benchmark`, `COREAI_CHUNK_THRESHOLD=1`)
+Apple's `GatherMM` reads **all 64 experts' weights every token**; a custom
+`coreai_torch.TorchMetalKernel` reads **only the 4 routed experts** (4/64) → decode runs at
+active-param bandwidth: **52.4 tok/s, 2.6×** (the biggest relative gain of the zoo's three MoE
+gather ports — a 16× over-read removed).
 
-| config | bundle | prefill tok/s | decode tok/s |
-|---|---:|---:|---:|
-| **int8 linear per-block-32 + untied absmax int8 head (`int8hu --head-sym`) = SHIP** | **30 GB** | **20.5** | **20.3** |
+**Quality is clean and unchanged.** The kernel reads the **`sym8`** scheme = the same
+symmetric-linear int8 (per-K-block-32) recipe the standard int8 bundle uses, via a **bit-exact**
+gather: **0 introduced flips / 18 vs fp16**. Pure speed win at the same quality.
 
-Decode is context-stable (set by the per-token weight read, not KV growth). **~20 tok/s for a
-30B-class local coder on a Mac** — comfortably above reading speed.
+| bundle | size | decode tok/s | quality |
+|---|---:|---:|---|
+| `gpu-pipelined/glm_4_7_flash_decode_sym8_gather/` | 30 GB | **52.4** | clean (0 flips/18 vs fp16) ✅ |
 
-## Numerics (gated)
-
-- **Parity** (fp16 port vs fp32 HF oracle; 30B is too large for an fp32-resident oracle on
-  128 GB): RoPE / isolated MLA block / isolated MoE block all cosine **1.000000**; all 47 layers
-  ≥0.9995; logits **top-1 16/16**; stateful decode + teacher-forced sweep all match. The
-  interleaved decoupled-RoPE convention is verified **bit-exact vs HF**.
-- **int8 end-to-end**: the real `coreai-sequential` engine reproduces the eager-int8 CPU greedy
-  continuation **token-for-token** over 32 tokens — a coherent, on-topic continuation. Engine ≡
-  python at int8.
+Mac-only (30 GB int8). Remaining speed lever = absorbed-MLA (GLM runs full MLA on all 47 layers).
 
 ## Run
 
-This is a **decode-only loop-free** bundle for the pipelined engine. Drive it token-by-token with
-the `coreai-sequential` engine variant (the default / `coreai-pipelined` variants feed a prefill
-chunk the static-`[1,1]` graph can't take):
-
-```bash
-# tok/s
-COREAI_CHUNK_THRESHOLD=1 llm-benchmark \
-    --model gpu-pipelined/glm_4_7_flash_decode_int8hu_block32_sym -p 64 -g 128 -n 3
-# greedy generation
-COREAI_CHUNK_THRESHOLD=1 llm-runner \
-    --model gpu-pipelined/glm_4_7_flash_decode_int8hu_block32_sym \
-    --inference-engine-variant coreai-sequential --sampling-strategy greedy \
-    --prompt "Write a Python function to merge two sorted lists." --max-tokens 128
 ```
+COREAI_CHUNK_THRESHOLD=1 llm-benchmark --model gpu-pipelined/glm_4_7_flash_decode_sym8_gather -p 128 -g 256 -n 3
+```
+
+Convert your own with [`conversion/export_glm47_moe_metal_decode_pipelined.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_glm47_moe_metal_decode_pipelined.py).
 
 ## License
 
-GLM-4.7-Flash is released by zai-org under the **MIT License** (see `LICENSE`). The conversion
-code (in [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo)) derives from Apple's
-BSD-3-clause `coreai_models`.
+MIT (upstream GLM license). Conversion + `gather_qmm` kernel: community.