README.md

---
license: other
license_name: glm-5.2
tags:
- gguf
- moe
- glm
- flash-moe
- llama.cpp
- metal
library_name: llama.cpp
---

# GLM-5.2 Flash-MoE Sidecar (UD-IQ1_M)

SSD-streamed Mixture-of-Experts **expert sidecar** for **GLM-5.2** (Unsloth Dynamic `UD-IQ1_M`),
built for the **Flash-MoE slot-bank** runtime in the
[`anemll/flash-llama.cpp`](https://github.com/Anemll/anemll-flash-llama.cpp/tree/GLM-5.2-Moe) fork.

The routed experts are stored as per-layer `layer_*.bin` files and streamed from SSD on demand into
a small resident **slot bank** during decode, so the full MoE runs on a unified-memory Mac without
keeping every expert in RAM. The dense / shared weights live in a separate small GGUF.

## What's in this repo

| Path | Size | Description |
|------|------|-------------|
| `dense/model-dense.gguf` | ~15.5 GB | Dense + shared weights, router, attention (the model you pass to `-m`) |
| `dense/flashmoe-package.json` | — | Flash-MoE package descriptor |
| `layer_003.bin` … `layer_NNN.bin` | ~213 GB total | Per-layer routed-expert tensors (IQ1_M), streamed on demand |
| `manifest.json` | — | Sidecar manifest (tensor map, quant types, byte offsets) |

**Model facts:** arch `glm-dsa`, 256 routed experts, top-8 per token, 3 leading dense layers,
`n_embd = 6144`, routed `n_ff = 2048`, experts quantized `IQ1_M`. Layout: `layer_major_whole_tensor`.

> Total download is ~**213 GB**. You need a fast SSD; decode is I/O-bound on expert streaming.

## Download

```bash
hf download anemll/GLM-5.2-sidecar --repo-type model --local-dir ~/Models/GLM-5.2-sidecar
```

## Build the runtime (Apple Metal)

This sidecar requires the Flash-MoE fork on the **`GLM-5.2-Moe`** branch:

```bash
git clone -b GLM-5.2-Moe https://github.com/Anemll/anemll-flash-llama.cpp
cd anemll-flash-llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j --target llama-cli
```

## Run / test

```bash
./build/bin/llama-cli --perf \
  -m ~/Models/GLM-5.2-sidecar/dense/model-dense.gguf \
  --moe-mode slot-bank \
  --moe-sidecar ~/Models/GLM-5.2-sidecar/ \
  --moe-verify-sidecar \
  --moe-slot-bank 64 \
  --moe-topk 8 \
  --moe-cache-io-split 2 \
  --moe-prefetch-temporal \
  -fit on \
  -ub 1 -b 64 \
  -ngl 999 \
  -c 512 \
  --seed 123 --temp 0 \
  -p "What is Apple Neural Engine? Answer in one sentence." \
  -n 2000 -st \
  --slot8
```

### `--slot8` (fused single-kernel routed FFN)

This branch adds `--slot8`, which collapses the whole routed FFN — gate, up, SwiGLU, down, and the
routed weighted-sum over all selected experts — into a **single fused op** (two Metal kernels,
IQ1_M) for single-token decode. It reads the resident slot ids once at encode time, so the
per-expert `mul_mat_id` decode replay / ICB cache is no longer used on that path. Output is
validated byte-identical to the unfused reference path.

Toggles:
- `--slot8` / `--no-slot8` — enable/disable the fused path (only engages on eligible top-k decode layers).
- `LLAMA_FLASH_MOE_SLOT8_REFERENCE=1` — force the `mul_mat` reference path (A/B comparison / fallback).
- `LLAMA_FLASH_MOE_SLOT8_DEBUG=1` — log which layers take the fused path.

> Tested on Apple M5 Max (128 GB). `--slot8` is a decode-only fast path; prefill and non-eligible
> layers use the normal slot-bank route.

## License

Derived from **GLM-5.2** (Z.ai / Zhipu AI). Use is subject to the original GLM-5.2 model license;
this sidecar only repackages those weights for SSD-streamed inference.