---
> An **MLX** build of the official [`zai-org/GLM-5.3`](https://huggingface.co/zai-org/GLM-5.3)
> — Z.AI's **753B-total / ~39B-active** MoE flagship for frontier coding and long-horizon
> agentic engineering, built on the **`glm_moe_dsa`** architecture (MLA + DeepSeek-style
> **sparse attention with a shared indexer**, 1M context) — quantized to **MLX** for
> **Apple Silicon** and the MLX CUDA backend. Four precisions (**2 / 3 / 4 / 6-bit**) ship as
> subfolders with the **4-bit build mirrored at the repo root**, produced with **OrcaSAQ
> (Sensitivity-Aware Quantization)**: calibration-free and architecture-aware, giving sensitive
> tensors more bits (shared experts `+2`, `down_proj` `+1`) while **attention stays at 8-bit and
> the DSA indexer stays in BF16** in every build. Browse all models in the
> [OrcaRouter Model Catalog](https://www.orcarouter.ai/models); deployed as API
> [here](https://www.orcarouter.ai/models/z-ai/glm-5.3). Put this model to work reviewing your
> pull requests with **[OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review)**.
---
## Available quantizations
| Folder | Expert base bits | Group size | Size | Min RAM | Quality vs FP8 |
|---|---|---|---:|---:|---|
| `6-bit/` | 6 | 64 | **671 GB** (625 GiB) | ~700 GB | **Near-lossless** — best quality |
| `4-bit/` | 4 | 64 | **459 GB** (427 GiB) | ~480 GB | Very good — **recommended default** |
| `3-bit/` | 3 | 64 | **368 GB** (343 GiB) | ~390 GB | Good — aggressive but usable |
| `2-bit/` | 2 | 32 | **322 GB** (300 GiB) | ~340 GB | Aggressive — best-effort |
The **repository root is a mirror of `4-bit/`**, so `orcarouter/GLM-5.3-MLX` resolves to the recommended default with no subfolder needed.
> Quantized from the official **FP8** release (756 GB, block-wise `e4m3` 128×128, dynamic activation scheme). Bit-widths above are the **expert base** precision; the OrcaSAQ policy below raises sensitive tensors above the base. Group size is 64 everywhere except the `2-bit` routed experts, which use 32 — at two bits the finer grid is what keeps the build usable (see the fidelity table).
**Hardware reality check.** This is a 753B model; even the smallest build is 322 GB. On a **512 GB M3 Ultra Mac Studio**, `2-bit` and `3-bit` leave comfortable KV headroom and `4-bit` fits but runs tight — raise the wired-memory limit and keep the context bounded. `6-bit` does not fit any single Apple Silicon machine today: run it across two 512 GB boxes with `mlx.distributed`, or on a CUDA host (8×H200 = 1128 GB) via the MLX CUDA backend. If the hardware isn't there, the [hosted API](https://www.orcarouter.ai/models/z-ai/glm-5.3) needs no local memory at all.
No 8-bit build was produced: `6-bit` already lands at cosine ≥ 0.9997 against FP8, and 8-bit expert weights would exceed the FP8 source in size.
---
## OrcaSAQ — Sensitivity-Aware Quantization
OrcaSAQ is a **calibration-free, architecture-aware mixed-precision quantization method** designed to preserve model quality while aggressively reducing memory footprint.
Instead of running expensive calibration datasets or per-layer sensitivity searches, OrcaSAQ uses **architectural and tensor-role priors** to determine which weights are most sensitive to quantization. Critical components receive higher precision, while more quantization-tolerant tensors remain at the target base precision.
For MoE architectures, for example, OrcaSAQ can preserve shared experts at **base +2 bits**, `down_proj` at **base +1 bit**, keep `gate_proj` and `up_proj` at the base precision, and retain sensitive non-quantized components at their source dtype.
The result is a **fast, deterministic, zero-calibration quantization pipeline** that can be applied to new architectures without costly dataset-driven optimization.
**Architecture-aware. Calibration-free. Quality-preserving.**
Two properties of this checkpoint drive the whole recipe:
- **Attention is FP8 upstream** (unlike GLM-5.3-Flash, where it shipped BF16), so it *is* quantizable — and it is **pinned to 8-bit in every build**, never dropped to the base tier.
- **The DSA indexer is kept in BF16 everywhere.** It selects which tokens attention attends to, so it sits directly on the long-context path and is left at full precision.
Everything that was *not* FP8 in the base release — the indexer, router `gate` weights, the FP32 `e_score_correction_bias`, every norm, `embed_tokens` and `lm_head` — is carried through at its **original dtype**, never a lossy cast.
### Bit allocation for GLM-5.3
| Component | Tensors | Params | 2-bit | 3-bit | 4-bit | 6-bit | Policy |
|---|---:|---:|---|---|---|---|---|
| Expert `gate_proj` / `up_proj` | 38,400 | 483.2B | 2 `gs32` | 3 | 4 | 6 | base |
| Expert `down_proj` | 19,200 | 241.6B | 3 `gs32` | 4 | 5 | 8 | base **+1** |
| Shared expert `gate` / `up` / `down` | 225 | 2.83B | 4 | 5 | 6 | 8 | base **+2** |
| MLA `q_a` / `q_b` / `kv_a_with_mqa` / `kv_b` / `o_proj` | 390 | 12.87B | **8** | **8** | **8** | **8** | pinned 8-bit |
| Dense-MLP `gate` / `up` / `down` (layers 0–2) | 9 | 0.68B | **6** | **6** | **6** | **6** | pinned 6-bit |
| DSA indexer `wk` / `wq_b` / `weights_proj` | 63 | 0.19B | BF16 | BF16 | BF16 | BF16 | never quantized |
| Router `gate.weight` | 75 | 0.12B | BF16 | BF16 | BF16 | BF16 | never quantized |
| Router `e_score_correction_bias` | 75 | — | FP32 | FP32 | FP32 | FP32 | never quantized |
| Norms (incl. indexer `k_norm`), `embed_tokens`, `lm_head` | 357 | 1.90B | BF16 | BF16 | BF16 | BF16 | never quantized |
**58,224 tensors are quantized** in total (97.5 % of all parameters live in the routed experts). Bits are rounded to the nearest MLX-supported width (`{2,3,4,5,6,8}`), and group size must divide the tensor's last dim — `gs64` on attention is mandatory because `qk_nope_head_dim` is 192 and `gs128` breaks mlx-lm's `kv_b → embed_q` re-quantization.
Effective width on the experts, including the FP16 scales and biases MLX stores per group:
| Build | Expert effective bpw | Experts as share of build |
|---|---:|---:|
| `2-bit` | 3.33 | 93.7 % |
| `3-bit` | 3.83 | 94.4 % |
| `4-bit` | 4.83 | 95.4 % |
| `6-bit` | 7.17 | 96.8 % |
The assignment is recorded in `config.json` → `quantization` (and mirrored to `quantization_config`) as **per-module `{group_size, bits}` overrides keyed by MLX module path** — e.g. `model.layers.3.mlp.switch_mlp.down_proj`. MLX fuses a layer's 256 routed experts into one `switch_mlp`, so **699 entries** cover all 58,224 tensors. This is what the loader reads at load time and it is **required**: the top-level `bits`/`group_size` alone would mis-shape every raised-precision tensor. A full per-tensor record is also shipped as `quantization_map.json`.
The **MTP (multi-token prediction) layer — layer 78 — is dropped**, and `num_nextn_predict_layers` is set to `0`. 791 tensors (~9.95B params) are removed; MLX runtimes drop it at load anyway.
---
## Quality vs FP8
All three tables compare each build against the full **FP8** reference (dequantized to BF16 and run
through the identical `glm_moe_dsa` forward, so the only variable is the quantization). Sizes are
decimal GB, matching the file sizes in this repo.
**Perplexity**
Measured against the **FP8 reference** on wikitext-2 test, 4 chunks × 1024 tokens
(4,092 predicted tokens). Both sides run the identical `glm_moe_dsa` forward on the
same FP8 kernels — each build's actual dequantized bytes are injected back into the
transformers model — so quantization is the only variable.
| Build | Size | PPL | ΔPPL |
|---|---:|---:|---:|
| **FP8 (ref)** | 756 GB | **1.6666** | — |
| **6-bit** | 671 GB | **1.6642** | **−0.14 %** |
| **4-bit** | 459 GB | **1.7015** | **+2.10 %** |
| **3-bit** | 368 GB | **1.8249** | **+9.50 %** |
| **2-bit** | 322 GB | **2.6385** | **+58.3 %** |
`6-bit` lands within noise of the FP8 reference — read it as indistinguishable.
**KL divergence & Top-1 token agreement**
KLD is `KL(ref ‖ quant)` per token against the FP8 reference distribution, Top-1 is how
often the build's argmax matches the reference's. Lower KLD and higher Top-1 = closer to
the original. Same corpus and token budget as the PPL table.
| Build | Size | Mean KLD | KLD p95 | Top-1 agreement |
|---|---:|---:|---:|---:|
| **FP8 (ref)** | 756 GB | — | — | 100 % |
| **6-bit** | 671 GB | **0.0295** | 0.1629 | **96.39 %** |
| **4-bit** | 459 GB | **0.0574** | 0.3049 | **95.46 %** |
| **3-bit** | 368 GB | **0.1668** | 0.9764 | **90.43 %** |
| **2-bit** | 322 GB | **0.5272** | 2.7405 | **80.18 %** |
All three measurements — weight fidelity, perplexity, and KLD/Top-1 — agree on the
ordering: `6-bit` is effectively lossless, `4-bit` costs about **2 % perplexity for 43 %
less memory** and is the recommended default, and `3-bit` holds up well at 40 % under
FP8. `2-bit` is the memory-first option — reach for it when the hardware cannot hold
anything larger.
**Weight-space fidelity** — measured on every quantized tensor at pack time (58,224 per build):
| Build | Size | Cosine sim | SNR (dB) | Rel. error |
|---|---:|---:|---:|---:|
| `6-bit` | 671 GB | **0.9998** | **37.2** | **1.7 %** |
| `4-bit` | 459 GB | **0.9968** | **22.9** | **7.6 %** |
| `3-bit` | 368 GB | **0.9865** | **16.7** | **15.7 %** |
| `2-bit` | 322 GB | **0.9517** | **11.2** | **29.7 %** |
Worst-case per build (`cos_min` / `snr_min`): `6-bit` 0.99972 / 32.5 dB · `4-bit` 0.99509 / 20.1 dB · `3-bit` 0.97869 / 13.7 dB · `2-bit` 0.92718 / 8.3 dB.
> Relative error is the mean of the **per-tensor** relative error `10^(-SNR/20)` over all 58,224 quantized tensors — not a value back-derived from the mean SNR, which would understate it (1.4 / 7.1 / 14.7 / 27.5 %).
Per role — this is where the OrcaSAQ policy shows up:
| Role | bits/gs | 2-bit cos | 3-bit cos | 4-bit cos | 6-bit cos |
|---|---|---:|---:|---:|---:|
| Expert `gate_proj`/`up_proj` | base | 0.93417 | 0.98169 | 0.99570 | 0.99975 |
| Expert `down_proj` | base +1 | 0.98510 | 0.99568 | 0.99898 | 0.99998 |
| Shared expert | base +2 | 0.99561 | 0.99896 | 0.99974 | 0.99998 |
| MLA attention | 8/64 | 0.99998 | 0.99998 | 0.99998 | 0.99998 |
| Dense MLP | 6/64 | 0.99975 | 0.99975 | 0.99975 | 0.99975 |
Cosine distribution across all quantized tensors (tight percentiles = no outlier tensors hiding behind a good mean):
| Build | min | p1 | p5 | median | p95 | max |
|---|---:|---:|---:|---:|---:|---:|
| `2-bit` | 0.92718 | 0.93133 | 0.93390 | 0.93437 | 0.98515 | 0.99999 |
| `3-bit` | 0.97869 | 0.98047 | 0.98157 | 0.98178 | 0.99571 | 0.99999 |
| `4-bit` | 0.99509 | 0.99546 | 0.99566 | 0.99571 | 0.99899 | 0.99999 |
| `6-bit` | 0.99972 | 0.99974 | 0.99975 | 0.99975 | 0.99999 | 0.99999 |
The worst tensors in every build are the same ones — early-layer routed experts (`layers.3–5`, e.g. `layers.4.mlp.experts.168.gate_proj`) — and `6-bit` still holds them above 0.9997.
**Build verification** — every build passed a structural release gate before upload:
- **accounting** — 175,242 tensors on disk, 58,224 quantized modules, `weight`/`scales`/`biases` triplets all paired
- **dtype preservation** — kept tensors byte-identical in their source dtype (FP32 router bias stays FP32; FP8 `keep` goes through a proper block dequant, never a raw cast)
- **index integrity** — `model.safetensors.index.json` resolves every tensor; `total_size` matches the shards on disk
- **config ↔ map reconstruction** — the 699 per-module overrides reproduce the full per-tensor map exactly
- **spot dequant** — randomly sampled tensors per role dequantized and compared against the golden FP8 values
- **runtime skeleton** — mlx-vlm's `glm_moe_dsa` module tree vs the post-sanitize key set: **1497/1497**, zero missing, zero extra
---
---
## Usage
### Hosted API — no download
The full-precision model is live on OrcaRouter as **`z-ai/glm-5.3`** — 1M context, 128K max output, $1.26 / $3.96 per 1M input / output tokens:
```python
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.orcarouter.ai/v1",
api_key=os.environ["ORCAROUTER_API_KEY"],
)
response = client.chat.completions.create(
model="z-ai/glm-5.3",
messages=[{"role": "user", "content": "Refactor this module and explain the tradeoffs."}],
)
print(response.choices[0].message.content)
```
**https://www.orcarouter.ai/models/z-ai/glm-5.3**
### Run it locally (mlx-vlm)
> **Use `mlx-vlm`, not `mlx-lm`.** GLM-5.3 is a text-only model, but as of **mlx-lm 0.31.3** the `glm_moe_dsa` implementation builds an indexer for *every* layer, while this checkpoint shares one indexer across four layers (`indexer_types`) — 285 tensors come up missing. **mlx-vlm 0.6.17** implements the shared-indexer layout and loads these builds as-is. Check for mlx-lm support before switching back.
```bash
pip install -U "mlx-vlm>=0.6.17"
```
The repo root is the 4-bit build, so the recommended default needs no subfolder — the excludes are what stop `hf download` from also pulling the four variant folders:
```bash
hf download orcarouter/GLM-5.3-MLX --local-dir ./GLM-5.3-MLX \
--exclude "2-bit/*" "3-bit/*" "4-bit/*" "6-bit/*"
python -m mlx_vlm.generate --model ./GLM-5.3-MLX \
--prompt "Write a Rust function that merges overlapping intervals." --max-tokens 512
```
To pick a different precision, download just that subfolder and load it from the local path:
```bash
hf download orcarouter/GLM-5.3-MLX --include "6-bit/*" --local-dir ./GLM-5.3-MLX
python -m mlx_vlm.generate --model ./GLM-5.3-MLX/6-bit \
--prompt "Write a Rust function that merges overlapping intervals." --max-tokens 512
```
Python API:
```python
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("./GLM-5.3-MLX") # repo root == 4-bit
prompt = apply_chat_template(processor, model.config, "Explain MLA vs GQA in three sentences.")
print(generate(model, processor, prompt, max_tokens=512, verbose=True))
```
**Reasoning effort.** GLM-5.3's chat template takes a `reasoning_effort` variable — `low`, `high`, or `max` (the default when unset) — emitted as a system directive, plus a `clear_thinking` flag that strips prior `
