Add model card

README.md

@@ -0,0 +1,377 @@
+---
+license: other
+license_name: glm-5.3-license
+license_link: https://huggingface.co/orcarouter/GLM-5.3-MLX/blob/main/LICENSE
+base_model: zai-org/GLM-5.3
+base_model_relation: quantized
+pipeline_tag: text-generation
+library_name: mlx
+language:
+  - en
+  - zh
+tags:
+  - glm
+  - glm-5
+  - glm_moe_dsa
+  - mlx
+  - apple-silicon
+  - quantized
+  - 2-8bit
+  - moe
+  - orcasaq
+  - dynamic-quant
+  - reasoning
+  - coding
+  - agentic
+---
+
+# GLM-5.3-MLX
+
+*An MLX build of the official GLM-5.3 — 2 / 3 / 4 / 6-bit OrcaSAQ quant for Apple Silicon & the MLX CUDA backend*
+
+**▶ [Run GLM-5.3 instantly on the OrcaRouter API](https://www.orcarouter.ai/models/z-ai/glm-5.3)** — no download, full precision.
+
+[Website](https://www.orcarouter.ai) · [Model Catalog](https://www.orcarouter.ai/models) · [Model API](https://www.orcarouter.ai/models/z-ai/glm-5.3) · [GitHub](https://github.com/Continuum-AI-Corp) · [OrcaCode Review](https://github.com/Continuum-AI-Corp/Orca-Code-Review) · [Discord](https://discord.gg/yAh6Tex6kx) · [X](https://x.com/OrcaRouter)
+
+---
+
+## Overview
+
+An **MLX** build of the official [`zai-org/GLM-5.3`](https://huggingface.co/zai-org/GLM-5.3) — Z.AI's **753B-total / ~39B-active** MoE flagship for frontier coding and long-horizon agentic engineering, built on the **`glm_moe_dsa`** architecture (MLA + DeepSeek-style **sparse attention with a shared indexer**) — quantized to **MLX** for **Apple Silicon** and the MLX CUDA backend.
+
+Four builds (**2 / 3 / 4 / 6-bit**) ship as subfolders, with the **4-bit build mirrored at the repo root**, produced with **OrcaSAQ (Sensitivity-Aware Quantization)** — calibration-free and architecture-aware: sensitive tensors get more bits (shared experts `+2`, `down_proj` `+1`), and the components that decide *what attention sees* are never quantized at all.
+
+Two things are specific to this checkpoint and drive the whole recipe:
+
+- **Attention is FP8 upstream** (unlike GLM-5.3-Flash, where it shipped BF16), so it *is* quantizable — and it is **pinned to 8-bit in every build**, never dropped to the base tier.
+- **The DSA indexer is kept in BF16 everywhere.** It selects which tokens attention attends to; degrading it degrades the long-context path itself, which is exactly the failure mode teacher-forced evals hide.
+
+Everything that was *not* FP8 in the base release — the indexer, router `gate` weights, the FP32 `e_score_correction_bias`, every norm, `embed_tokens` and `lm_head` — is carried through at its **original dtype**, never a lossy cast.
+
+---
+
+## Available Quantizations
+
+| Folder | Expert base bits | Group size | Size | Min RAM | Quality vs FP8 |
+|---|---|---|---:|---:|---|
+| `6-bit/` | 6 | 64 | **671 GB** (625 GiB) | ~700 GB | **Near-lossless** — best quality |
+| `4-bit/` | 4 | 64 | **459 GB** (427 GiB) | ~480 GB | Very good — **recommended default** |
+| `3-bit/` | 3 | 64 | **368 GB** (343 GiB) | ~390 GB | Good — aggressive but usable |
+| `2-bit/` | 2 | 32 | **322 GB** (300 GiB) | ~340 GB | Aggressive — best-effort |
+
+The **repository root is a mirror of `4-bit/`**, so `orcarouter/GLM-5.3-MLX` resolves to the recommended default with no subfolder needed.
+
+> Quantized from the official **FP8** release (756 GB, block-wise `e4m3` 128×128, dynamic activation scheme). Bit-widths above are the **expert base** precision; the OrcaSAQ policy below raises sensitive tensors above the base. Group size is 64 everywhere except the `2-bit` routed experts, which use 32 — at two bits the finer grid is what keeps the build usable (see the fidelity table).
+
+**Hardware reality check.** This is a 753B model; even the smallest build is 322 GB. On a **512 GB M3 Ultra Mac Studio**, `2-bit` and `3-bit` leave comfortable KV headroom and `4-bit` fits but runs tight — raise the wired-memory limit and keep the context bounded. `6-bit` does not fit any single Apple Silicon machine today: run it across two 512 GB boxes with `mlx.distributed`, or on a CUDA host (8×H200 = 1128 GB) via the MLX CUDA backend. If the hardware isn't there, the [hosted API](https://www.orcarouter.ai/models/z-ai/glm-5.3) needs no local memory at all.
+
+No 8-bit build was produced: `6-bit` already lands at cosine ≥ 0.9997 against FP8, and 8-bit expert weights would exceed the FP8 source in size.
+
+---
+
+## OrcaSAQ — Sensitivity-Aware Quantization
+
+OrcaSAQ is a **calibration-free, architecture-aware mixed-precision quantization method** designed to preserve model quality while aggressively reducing memory footprint.
+
+Instead of running expensive calibration datasets or per-layer sensitivity searches, OrcaSAQ uses **architectural and tensor-role priors** to determine which weights are most sensitive to quantization. Critical components receive higher precision, while more quantization-tolerant tensors remain at the target base precision.
+
+For MoE architectures, for example, OrcaSAQ can preserve shared experts at **base +2 bits**, `down_proj` at **base +1 bit**, keep `gate_proj` and `up_proj` at the base precision, and retain sensitive non-quantized components at their source dtype.
+
+The result is a **fast, deterministic, zero-calibration quantization pipeline** that can be applied to new architectures without costly dataset-driven optimization.
+
+**Architecture-aware. Calibration-free. Quality-preserving.**
+
+### Bit Allocation for GLM-5.3
+
+| Component | Tensors | Params | 2-bit | 3-bit | 4-bit | 6-bit | Policy |
+|---|---:|---:|---|---|---|---|---|
+| Expert `gate_proj` / `up_proj` | 38,400 | 483.2B | 2 `gs32` | 3 | 4 | 6 | base |
+| Expert `down_proj` | 19,200 | 241.6B | 3 `gs32` | 4 | 5 | 8 | base **+1** |
+| Shared expert `gate` / `up` / `down` | 225 | 2.83B | 4 | 5 | 6 | 8 | base **+2** |
+| MLA `q_a` / `q_b` / `kv_a_with_mqa` / `kv_b` / `o_proj` | 390 | 12.87B | **8** | **8** | **8** | **8** | pinned 8-bit |
+| Dense-MLP `gate` / `up` / `down` (layers 0–2) | 9 | 0.68B | **6** | **6** | **6** | **6** | pinned 6-bit |
+| DSA indexer `wk` / `wq_b` / `weights_proj` | 63 | 0.19B | BF16 | BF16 | BF16 | BF16 | never quantized |
+| Router `gate.weight` | 75 | 0.12B | BF16 | BF16 | BF16 | BF16 | never quantized |
+| Router `e_score_correction_bias` | 75 | — | FP32 | FP32 | FP32 | FP32 | never quantized |
+| Norms (incl. indexer `k_norm`), `embed_tokens`, `lm_head` | 357 | 1.90B | BF16 | BF16 | BF16 | BF16 | never quantized |
+
+**58,224 tensors are quantized** in total (97.5 % of all parameters live in the routed experts). Bits are rounded to the nearest MLX-supported width (`{2,3,4,5,6,8}`), and group size must divide the tensor's last dim — `gs64` on attention is mandatory because `qk_nope_head_dim` is 192 and `gs128` breaks mlx-lm's `kv_b → embed_q` re-quantization.
+
+Effective width on the experts, including the FP16 scales and biases MLX stores per group:
+
+| Build | Expert effective bpw | Experts as share of build |
+|---|---:|---:|
+| `2-bit` | 3.33 | 93.7 % |
+| `3-bit` | 3.83 | 94.4 % |
+| `4-bit` | 4.83 | 95.4 % |
+| `6-bit` | 7.17 | 96.8 % |
+
+The assignment is recorded in `config.json` → `quantization` (and mirrored to `quantization_config`) as **per-module `{group_size, bits}` overrides keyed by MLX module path** — e.g. `model.layers.3.mlp.switch_mlp.down_proj`. MLX fuses a layer's 256 routed experts into one `switch_mlp`, so **699 entries** cover all 58,224 tensors. This is what the loader reads at load time and it is **required**: the top-level `bits`/`group_size` alone would mis-shape every raised-precision tensor. A full per-tensor record is also shipped as `quantization_map.json`.
+
+The **MTP (multi-token prediction) layer — layer 78 — is dropped**, and `num_nextn_predict_layers` is set to `0`. 791 tensors (~9.95B params) are removed; MLX runtimes drop it at load anyway.
+
+---
+
+## Quality vs FP8
+
+### Weight-Space Fidelity
+
+Measured on **every quantized tensor** at pack time (58,224 per build), against the FP8 source block-dequantized to BF16 — so the only variable is the quantization.
+
+| Build | Size | Cosine sim | SNR (dB) | Rel. error |
+|---|---:|---:|---:|---:|
+| `6-bit` | 671 GB | **0.9998** | **37.2** | **1.7 %** |
+| `4-bit` | 459 GB | **0.9968** | **22.9** | **7.6 %** |
+| `3-bit` | 368 GB | **0.9865** | **16.7** | **15.7 %** |
+| `2-bit` | 322 GB | **0.9517** | **11.2** | **29.7 %** |
+
+Worst-case per build (`cos_min` / `snr_min`): `6-bit` 0.99972 / 32.5 dB · `4-bit` 0.99509 / 20.1 dB · `3-bit` 0.97869 / 13.7 dB · `2-bit` 0.92718 / 8.3 dB.
+
+> Relative error is the mean of the **per-tensor** relative error `10^(-SNR/20)` over all 58,224 quantized tensors — not a value back-derived from the mean SNR, which would understate it (1.4 / 7.1 / 14.7 / 27.5 %).
+
+Per role — this is where the OrcaSAQ policy shows up:
+
+| Role | bits/gs | 2-bit cos | 3-bit cos | 4-bit cos | 6-bit cos |
+|---|---|---:|---:|---:|---:|
+| Expert `gate_proj`/`up_proj` | base | 0.93417 | 0.98169 | 0.99570 | 0.99975 |
+| Expert `down_proj` | base +1 | 0.98510 | 0.99568 | 0.99898 | 0.99998 |
+| Shared expert | base +2 | 0.99561 | 0.99896 | 0.99974 | 0.99998 |
+| MLA attention | 8/64 | 0.99998 | 0.99998 | 0.99998 | 0.99998 |
+| Dense MLP | 6/64 | 0.99975 | 0.99975 | 0.99975 | 0.99975 |
+
+Cosine distribution across all quantized tensors (tight percentiles = no outlier tensors hiding behind a good mean):
+
+| Build | min | p1 | p5 | median | p95 | max |
+|---|---:|---:|---:|---:|---:|---:|
+| `2-bit` | 0.92718 | 0.93133 | 0.93390 | 0.93437 | 0.98515 | 0.99999 |
+| `3-bit` | 0.97869 | 0.98047 | 0.98157 | 0.98178 | 0.99571 | 0.99999 |
+| `4-bit` | 0.99509 | 0.99546 | 0.99566 | 0.99571 | 0.99899 | 0.99999 |
+| `6-bit` | 0.99972 | 0.99974 | 0.99975 | 0.99975 | 0.99999 | 0.99999 |
+
+The worst tensors in every build are the same ones — early-layer routed experts (`layers.3–5`, e.g. `layers.4.mlp.experts.168.gate_proj`) — and `6-bit` still holds them above 0.9997.
+
+### Perplexity
+
+Measured against the **FP8 reference** on wikitext-2 test, 4 chunks × 1024 tokens
+(4,092 predicted tokens). Both sides run the identical `glm_moe_dsa` forward on the
+same FP8 kernels — each build's actual dequantized bytes are injected back into the
+transformers model — so quantization is the only variable.
+
+| Build | Size | PPL | ΔPPL |
+|---|---:|---:|---:|
+| **FP8 (ref)** | 756 GB | **1.6666** | — |
+| **6-bit** | 671 GB | **1.6642** | **−0.14 %** |
+| **4-bit** | 459 GB | **1.7015** | **+2.10 %** |
+| **3-bit** | 368 GB | **1.8249** | **+9.50 %** |
+| **2-bit** | 322 GB | **2.6385** | **+58.3 %** |
+
+`6-bit` scoring a hair *below* the reference is sampling noise at this corpus size, not
+an improvement — read it as "indistinguishable from FP8."
+
+### KL Divergence & Top-1 Token Agreement
+
+KLD is `KL(ref ‖ quant)` per token against the FP8 reference distribution, Top-1 is how
+often the build's argmax matches the reference's. Lower KLD and higher Top-1 = closer to
+the original. Same corpus and token budget as the PPL table.
+
+| Build | Size | Mean KLD | KLD p95 | Top-1 agreement |
+|---|---:|---:|---:|---:|
+| **FP8 (ref)** | 756 GB | — | — | 100 % |
+| **6-bit** | 671 GB | **0.0295** | 0.1629 | **96.39 %** |
+| **4-bit** | 459 GB | **0.0574** | 0.3049 | **95.46 %** |
+| **3-bit** | 368 GB | **0.1668** | 0.9764 | **90.43 %** |
+| **2-bit** | 322 GB | **0.5272** | 2.7405 | **80.18 %** |
+
+All three measurements — weight fidelity, perplexity, and KLD/Top-1 — agree on the
+ordering and on where the cliff is: `6-bit` is effectively lossless, `4-bit` costs about
+2 % perplexity for 43 % less memory, `3-bit` degrades gently, and `2-bit` is a large
+step down (+58 % PPL, 1 token in 5 disagreeing with the reference). Pick `2-bit` for
+fit, not for quality.
+
+> **Scope of these numbers.** This is a 4,092-token quick evaluation, enough to rank the
+> builds and size the gap, but a small sample. The full-corpus suite (2048 × 12) and the
+> free-running generation gate below have not been run against these builds yet.
+
+### Long-Context Generation Gate
+
+Teacher-forced metrics can look fine on a build that loops in a real chat, so the
+pipeline also has a **free-running greedy generation gate**: long prompts, cycle
+detection and distinct-4, compared against the FP8 reference. A build fails if it
+degenerates where the reference does not.
+
+**Not yet run for these builds** — the reference outputs exist, the per-build runs do
+not. Treat the `2-bit` build in particular as unvalidated for long-form generation until
+this gate has passed; that is exactly the failure mode the teacher-forced tables above
+cannot see.
+
+### Build Verification
+
+Independent of quality, every build passed a structural release gate before upload:
+
+- **accounting** — 175,242 tensors on disk, 58,224 quantized modules, `weight`/`scales`/`biases` triplets all paired
+- **dtype preservation** — kept tensors byte-identical in their source dtype (FP32 router bias stays FP32; FP8 `keep` goes through a proper block dequant, never a raw cast)
+- **index integrity** — `model.safetensors.index.json` resolves every tensor; `total_size` matches the shards on disk
+- **config ↔ map reconstruction** — the 699 per-module overrides reproduce the full per-tensor map exactly
+- **spot dequant** — randomly sampled tensors per role dequantized and compared against the golden FP8 values
+- **runtime skeleton** — mlx-vlm's `glm_moe_dsa` module tree vs the post-sanitize key set: **1497/1497**, zero missing, zero extra
+
+---
+
+## Usage
+
+### Hosted API — No Download
+
+The full-precision model is live on OrcaRouter as **`z-ai/glm-5.3`** — 1M context, 128K max output, $1.26 / $3.96 per 1M input / output tokens:
+
+```python
+import os
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="https://api.orcarouter.ai/v1",
+    api_key=os.environ["ORCAROUTER_API_KEY"],
+)
+
+response = client.chat.completions.create(
+    model="z-ai/glm-5.3",
+    messages=[{"role": "user", "content": "Refactor this module and explain the tradeoffs."}],
+)
+print(response.choices[0].message.content)
+```
+
+**https://www.orcarouter.ai/models/z-ai/glm-5.3**
+
+### Run it Locally (mlx-vlm)
+
+> **Use `mlx-vlm`, not `mlx-lm`.** GLM-5.3 is a text-only model, but as of **mlx-lm 0.31.3** the `glm_moe_dsa` implementation builds an indexer for *every* layer, while this checkpoint shares one indexer across four layers (`indexer_types`) — 285 tensors come up missing. **mlx-vlm 0.6.17** implements the shared-indexer layout and loads these builds as-is. Check for mlx-lm support before switching back.
+
+```bash
+pip install -U "mlx-vlm>=0.6.17"
+```
+
+The repo root is the 4-bit build, so the recommended default needs no subfolder — the excludes are what stop `hf download` from also pulling the four variant folders:
+
+```bash
+hf download orcarouter/GLM-5.3-MLX --local-dir ./GLM-5.3-MLX \
+    --exclude "2-bit/*" "3-bit/*" "4-bit/*" "6-bit/*"
+
+python -m mlx_vlm.generate --model ./GLM-5.3-MLX \
+    --prompt "Write a Rust function that merges overlapping intervals." --max-tokens 512
+```
+
+To pick a different precision, download just that subfolder and load it from the local path:
+
+```bash
+hf download orcarouter/GLM-5.3-MLX --include "6-bit/*" --local-dir ./GLM-5.3-MLX
+
+python -m mlx_vlm.generate --model ./GLM-5.3-MLX/6-bit \
+    --prompt "Write a Rust function that merges overlapping intervals." --max-tokens 512
+```
+
+Python API:
+
+```python
+from mlx_vlm import load, generate
+from mlx_vlm.prompt_utils import apply_chat_template
+
+model, processor = load("./GLM-5.3-MLX")          # repo root == 4-bit
+prompt = apply_chat_template(processor, model.config, "Explain MLA vs GQA in three sentences.")
+
+print(generate(model, processor, prompt, max_tokens=512, verbose=True))
+```
+
+**Reasoning effort.** GLM-5.3's chat template takes a `reasoning_effort` variable — `low`, `high`, or `max` (the default when unset) — emitted as a system directive, plus a `clear_thinking` flag that strips prior `<think>` blocks from the history. Both pass straight through `apply_chat_template`:
+
+```python
+prompt = apply_chat_template(processor, model.config, "Port this service to async.",
+                             reasoning_effort="low")
+```
+
+Tool calling is in the template too (`<tool_call>` / `<arg_key>` blocks, `tools=[...]`).
+
+> **Long context.** `max_position_embeddings` is **1,048,576** (`rope_theta` 8e6). The KV cache, not the weights, is what will bite you at those lengths — budget for it, or cap it with `--max-kv-size` / quantize it with `--kv-bits`.
+>
+> On Apple Silicon the Metal backend is used automatically. On the Linux **CUDA** backend install `mlx[cuda]` and set `MLX_CUDA_USE_CUDNN_SDPA=0`.
+
+---
+
+## 🐳 Run it with OrcaCode Review
+
+[Website](https://www.orcarouter.ai) · [GitHub](https://github.com/Continuum-AI-Corp/Orca-Code-Review)
+
+**Models are only half the system.**
+
+[**OrcaCode Review**](https://github.com/Continuum-AI-Corp/Orca-Code-Review) turns every model listed on [**OrcaRouter**](https://www.orcarouter.ai) into a production code-review agent:
+
+- reviews every PR
+- finds security + correctness issues
+- posts inline findings
+- P0/P1 can block merges
+- swap models anytime
+
+**Open model. Open harness. Open bill.**
+
+- Website — https://www.orcarouter.ai
+- GitHub — https://github.com/Continuum-AI-Corp/Orca-Code-Review
+
+---
+
+## GLM-5.3
+
+> Available full-precision on the OrcaRouter API as **`z-ai/glm-5.3`** —
+> **https://www.orcarouter.ai/models/z-ai/glm-5.3**
+
+GLM-5.3 is Z.AI (Zhipu AI)'s flagship model for **complex software engineering and long-horizon agentic tasks** — roughly **50 % better at coding than its predecessor**, tuned for repo-scale work and autonomous engineering loops, with tool calling and JSON output over an OpenAI-compatible API.
+
+### Model Overview
+
+- **Type:** Mixture-of-Experts Causal LM (`glm_moe_dsa`, `GlmMoeDsaForCausalLM`)
+- **Parameters:** **753B total** · **~39B active** per token (22.6B routed + 16.7B always-on; 743B after the MTP layer is dropped)
+- **Experts:** 256 routed + 1 shared, **top-8** routing, sigmoid scoring with `e_score_correction_bias`, single group, `routed_scaling_factor` 2.5
+- **Layers:** 78 (3 dense + 75 MoE), plus 1 MTP layer (dropped in these builds)
+- **Attention:** **MLA** — `q_lora_rank` 2048, `kv_lora_rank` 512, 64 heads, `qk_nope` 192 + `qk_rope` 64, `v_head_dim` 256 — with a **DeepSeek-style sparse-attention indexer** (`index_topk` 2048, 32 index heads) built on every 4th layer and shared by the three that follow
+- **Precision (base release):** FP8 (block-wise `e4m3`, 128×128), dynamic activation scheme
+- **Context:** 1,048,576 tokens · vocab 154,880
+- **Modality:** text
+
+### Best Practices
+
+- **Pick a precision:** `6-bit` for near-lossless, `4-bit` (repo root) as the everyday default, `3-bit` when memory is the binding constraint, `2-bit` only when nothing else fits — the cosine table shows the cliff between 3 and 2 bits, and coding/agentic work is where it shows up first.
+- **Sampling:** follow the official GLM-5.3 guidance. The shipped `generation_config.json` is `temperature` 1.0, `top_p` 0.95; long-horizon agentic and coding tasks want generous `max-tokens` headroom.
+- **Reasoning effort:** leave it at the default `max` for hard engineering work; drop to `low` for latency-sensitive turns.
+- **Long context:** budget KV, not just weights — at 1M tokens the cache dwarfs the difference between two of these builds.
+- **Or skip the download:** the unquantized model is served at full precision on the [OrcaRouter API](https://www.orcarouter.ai/models/z-ai/glm-5.3) as `z-ai/glm-5.3`.
+
+---
+
+## Build Provenance
+
+| | |
+|---|---|
+| Source | `zai-org/GLM-5.3` (FP8, 141 shards, 756 GB) |
+| Quantized | 2026-08-28 |
+| Toolchain | MLX 0.32.2, safetensors 0.8.0, NumPy 2.5.2 |
+| Method | OrcaSAQ — calibration-free, role-based mixed precision |
+| Per-build files | 140 shards + `config.json`, `quantization_map.json`, `build_manifest.json`, `fidelity_summary.json`, tokenizer, chat template |
+
+Each folder carries its own `build_manifest.json` (exact recipe, source path, dropped-tensor count, toolchain versions) and `fidelity_summary.json` (per-role cosine/SNR with the five worst tensors named), so any claim in this card can be checked against the build itself.
+
+---
+
+## Citation
+
+```bibtex
+@misc{glm5team2026glm5vibecodingagentic,
+      title={GLM-5: from Vibe Coding to Agentic Engineering},
+      author={GLM-5-Team and Aohan Zeng and Xin Lv and Zhenyu Hou and Zhengxiao Du and others},
+      year={2026},
+      eprint={2602.15763},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2602.15763},
+}
+```
+
+## License
+
+Inherited from [`zai-org/GLM-5.3`](https://huggingface.co/zai-org/GLM-5.3) — an MIT-style grant (use, modify, distribute, fine-tune, sell) with one added condition: operators of a **Model-as-a-Service** business whose aggregate revenue exceeds **US$10B over any 12 consecutive months** must pass a Z.AI security review before commercial use. The full text ships as [`LICENSE`](./LICENSE) in this repo. Quantization does not change the underlying license obligations.