---
> An **MLX** build of the official [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash)
> — a **320B-total / 18B-active** MoE, the first natively multimodal model in the GLM-5 series, built on
> the new **`glm5_next`** hybrid architecture (sparse + linear attention with Manifold-Constrained
> Hyper-Connections) — quantized to **MLX** for **Apple Silicon** and the MLX CUDA backend. Five
> builds (**2bit-lite / 2 / 3 / 4 / 6-bit**) ship as subfolders, with the **4-bit build mirrored at the repo root**,
> produced with **OrcaSAQ (Sensitivity-Aware Quantization)** — calibration-free and
> architecture-aware: sensitive tensors get more bits
> (shared experts `+2`, `down_proj` `+1`), and every tensor that was *not* FP8 in the base release —
> **the 34 linear-attention layers, the sparse indexer, hyper-connections, norms, `embed_tokens`,
> `lm_head`, and the entire vision tower — is carried through in BF16**. The four projections of the
> sparse-attention blocks *were* FP8 upstream and are quantized at the base width. Browse all models in the
> [OrcaRouter Model Catalog](https://www.orcarouter.ai/models); deployed as API
> [here](https://www.orcarouter.ai/models/z-ai/glm-5.3-flash). Put this model to work reviewing your
> pull requests with **[OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review)**.
---
## Available quantizations
| Folder | Base bits | Group size | Size | Min RAM | Quality vs FP8 |
|---|---|---|---:|---|---|
| `6-bit/` | 6 | 64 | ~296 GB | 320 GB | **Near-lossless** — best quality |
| `4-bit/` | 4 | 64 | ~204 GB | 224 GB | Very good — **recommended default** |
| `3-bit/` | 3 | 32 | ~184 GB | 200 GB | Good — aggressive but usable |
| `2-bit/` | 2 | 32 | ~145 GB | 160 GB | Aggressive — best-effort |
| `2bit-lite/` | 2 | 128 | ~102 GB | 112 GB | Smallest — fits 128 GB Macs & a single H200 |
The **repository root is a mirror of `4-bit/`**, so `orcarouter/GLM-5.3-Flash-MLX` resolves to the
recommended default with no subfolder needed.
> Quantized from the official **FP8** release. Bit-widths above are the **base** precision; the OrcaSAQ
> policy below raises sensitive tensors above the base. Group size is 64 for 4/6-bit, 32 for 2/3-bit
> (shared experts always use 64), and 128 throughout `2bit-lite`.
No 8-bit build was produced. `2bit-lite` follows a separate, memory-first recipe — see below.
---
## OrcaSAQ — Sensitivity-Aware Quantization
OrcaSAQ is a **calibration-free, architecture-aware mixed-precision quantization method** designed to
preserve model quality while aggressively reducing memory footprint.
Instead of running expensive calibration datasets or per-layer sensitivity searches, OrcaSAQ uses
**architectural and tensor-role priors** to determine which weights are most sensitive to quantization.
Critical components receive higher precision, while more quantization-tolerant tensors remain at the
target base precision.
For MoE architectures, for example, OrcaSAQ can preserve shared experts at **base +2 bits**, `down_proj`
at **base +1 bit**, keep `gate_proj` and `up_proj` at the base precision, and retain sensitive
non-quantized components in BF16.
The result is a **fast, deterministic, zero-calibration quantization pipeline** that can be applied to
new architectures without costly dataset-driven optimization.
**Architecture-aware. Calibration-free. Quality-preserving.**
### Bit allocation for GLM-5.3-Flash
| Component | Count | 2-bit | 3-bit | 4-bit | 6-bit | Policy |
|---|---:|---|---|---|---|---|
| Expert `gate_proj` / `up_proj` | 24 768 | 2 | 3 | 4 | 6 | base |
| Expert `down_proj` | 12 384 | 3 | 4 | 5 | 8 | base **+1** |
| Dense-MLP `gate_proj` / `up_proj` (layers 0–2) | 6 | 2 | 3 | 4 | 6 | base |
| Dense-MLP `down_proj` (layers 0–2) | 3 | 3 | 4 | 5 | 8 | base **+1** |
| Shared expert `gate_proj` / `up_proj` / `down_proj` | 129 | 4 | 5 | 6 | 8 | base **+2** |
| Sparse-attn `q_a_proj` / `q_b_proj` / `kv_a_proj_with_mqa` / `o_proj` | 48 | 2 | 3 | 4 | 6 | base |
| Linear-attn layers, sparse indexer, mHC, norms, `embed_tokens`, `lm_head`, vision tower | — | BF16 | BF16 | BF16 | BF16 | never quantized |
Bits are rounded up to the nearest MLX-supported width (`{2,3,4,5,6,8}`). The architectural prior that
selects the quantizable set is mechanical and needs no data: a tensor is re-quantized **iff the FP8
release shipped it with a `_scale_inv` companion**. For GLM-5.3-Flash that set is the MoE and dense-MLP
linears *plus* the four projections of every `deepseek_sparse_attention` block — the 11 sparse layers at
depth 3, 7, 11 … 43, and the MTP block, hence 12 × 4 = 48 tensors. The 34 `linear_attention` layers, the
sparse indexer, and the vision tower were never FP8 and stay BF16.
37 338 tensors are quantized in total. The assignment is recorded in `config.json` →
`quantization`, as **per-module `{group_size, bits}` overrides keyed by MLX module path** —
e.g. `model.layers.3.mlp.switch_mlp.down_proj`. MLX fuses a layer's routed experts into one
`switch_mlp`, so 173 entries cover all 37 338 tensors. This is what the loader reads at load time and
it is **required**: the top-level `bits`/`group_size` alone would mis-shape every raised-precision
tensor.
The **MTP (multi-token prediction) layer — layer 45 — is included inside the quantized weights** rather
than exported as a separate `mtp/` module.
---
## Quality vs FP8
All three tables compare each build against the full **FP8** reference (dequantized to BF16 and run
through the identical `glm5_next` forward, so the only variable is the quantization). Sizes are decimal
GB, matching the file sizes in this repo.
**Perplexity**
| Build | Size | PPL | ΔPPL |
|---|---:|---:|---:|
| **FP8 (ref)** | 328 GB | 2.7797 | — |
| **6-bit** | 296 GB | 2.7864 | +0.24 % |
| **4-bit** | 204 GB | 2.8620 | +2.96 % |
| **3-bit** | 184 GB | 3.0566 | +9.96 % |
| **2-bit** | 145 GB | 4.3622 | +56.9 % |
| **2bit-lite** | 102 GB | 6.7018 | +141 % |
**KL divergence & Top-1 token agreement** — lower KLD and higher Top-1 = closer to the original.
| Build | Size | Mean KLD | KLD p95 | Top-1 agreement |
|---|---:|---:|---:|---:|
| **FP8 (ref)** | 328 GB | — | — | 100 % |
| **6-bit** | 296 GB | 0.0063 | 0.0142 | 97.76 % |
| **4-bit** | 204 GB | 0.0131 | 0.0477 | 96.13 % |
| **3-bit** | 184 GB | 0.0421 | 0.1332 | 92.06 % |
| **2-bit** | 145 GB | 0.1647 | 0.6528 | 86.56 % |
| **2bit-lite** | 102 GB | 0.3456 | 1.2617 | 77.19 % |
**Weight-space fidelity** (mean over sampled MoE/MLP tensors, quantized vs FP8):
| Build | Size | Cosine sim | SNR (dB) | Rel. error |
|---|---:|---:|---:|---:|
| **6-bit** | 296 GB | 0.9998 | 37.3 | 1.6 % |
| **4-bit** | 204 GB | 0.9969 | 22.9 | 7.6 % |
| **3-bit** | 184 GB | 0.9892 | 17.6 | 14.0 % |
| **2-bit** | 145 GB | 0.9518 | 11.1 | 29.7 % |
| **2bit-lite** | 102 GB | 0.9135 | 7.5 | 42.4 % |
The three measurements agree on the ordering, and they agree on where the cliff is: everything down to
`3-bit` degrades gently, `2-bit` costs a lot, and `2bit-lite` costs a lot more. Pick it for fit, not
for quality.
---
## `2bit-lite/` — smallest build, for 128 GB Macs and a single H200
A memory-first build that trades quality for the smallest footprint that still loads and runs on
`mlx-vlm`. It exists to fit machines where the regular 2-bit build (145 GB) is just too big:
- **128 GB MacBook Pro (M4 / M5 Max)** — regular 2-bit does not fit; this does, with a raised
wired-memory limit.
- **A single H200 (141 GB)** — regular 2-bit overflows one card; `2bit-lite` fits with ~39 GB left for
the KV cache.
**Recipe** (group_size 128 throughout):
| Component | Bits |
|---|---|
| Routed experts (`gate_proj` / `up_proj` / `down_proj`) | 2 |
| Attention MLA + dense MLP | 2 |
| Shared expert (fires on every token) | 3 |
| `embed_tokens` / `lm_head` / linear-attention path | BF16 |
**Size: ~102 GB** (≈2.25 bpw on the experts). That is the floor for this architecture: MLX's affine
quantizer cannot go below 2 bits, and the fused MoE experts — ~97 % of the weights — cannot use the
1-bit path.
### Field notes — what `2bit-lite` is actually good for
Verified on a single **H200**: it deploys cleanly, multi-turn conversation is stable at **~10 tok/s**,
and everyday Q&A and short-form text come out fine.
**Long code generation is not reliable at this precision.** Three failure modes reproduce consistently:
- **Repetition loops** — the model gets stuck emitting the same block over and over.
- **Missing glue code** — the overall structure is right, but load-bearing lines (imports, wiring,
error handling) are silently dropped.
- **Rewrite churn** — it keeps restarting the answer and never commits to a final version.
So: reach for `2bit-lite` when the hardware cannot hold anything larger, and keep it to chat,
Q&A and short generations. For coding and long-horizon agentic work use [`2-bit`](./2-bit) or
higher — the quality tables above show why the gap shows up in practice, and the API option needs no
local memory at all.
---
## Usage
### Hosted API — no download
The full-precision model is live on OrcaRouter as **`z-ai/glm-5.3-flash`**, so you can call it
without pulling 100–300 GB of weights — endpoint, keys and pricing are on the model page:
**
