README.md

---
language:
- en
- zh
license: mit
base_model:
- zai-org/GLM-5.2
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
tags:
- glm
- glm-5.2
- glm_moe_dsa
- mixture-of-experts
- autoround
- int4
- w4a16
- w4g64
- quantized
- vllm
- mtp
- reasoning
- tool-calling
- conversational
---

# GLM-5.2 AutoRound W4G64

This is an **unofficial community quantization** of
[`zai-org/GLM-5.2`](https://huggingface.co/zai-org/GLM-5.2). It is not an
official Z.ai release and is not affiliated with Z.ai.

The checkpoint targets high-throughput reasoning and coding-agent serving on a
single 8-GPU Hopper node. Routed MoE expert weights use symmetric 4-bit
AutoRound quantization with group size 64. Attention, the DSA/IndexShare
indexer, shared experts, sensitive projections, and the output head remain in
BF16. The MTP layer is included and can be used for speculative decoding.

## Checkpoint summary

| Property | Value |
|---|---|
| Base model | `zai-org/GLM-5.2` |
| Architecture | `GlmMoeDsaForCausalLM` |
| Quantization | AutoRound W4A16, symmetric W4G64 |
| Packing | `auto_round:auto_gptq` |
| Checkpoint layout | 81 backbone shards + 5 MTP shards |
| Approximate storage | 405 GiB |
| MTP | One quantized next-token-prediction layer |
| License | MIT, inherited from the base model |

## Quantization procedure

This is post-training weight-only quantization; the model was not fine-tuned.
The run used AutoRound 0.14.0 on four NVIDIA H20 GPUs with:

| Setting | Value |
|---|---:|
| Weight bits | 4 |
| Group size | 64 |
| Symmetric | Yes |
| Optimization iterations | 200 |
| Calibration samples | 512 |
| Calibration sequence length | 2,048 |
| Batch size | 2 |
| Gradient accumulation | 4 |
| Scale dtype | FP16 |

The calibration set consisted of packed coding-agent-style text samples. The
calibration corpus is not distributed with this checkpoint.

The mixed-precision policy was intentionally conservative:

- Backbone layers 3–77: routed-expert `gate_proj`, `up_proj`, and `down_proj`
  matrices are W4G64.
- Backbone layers 0–2, all attention and indexer modules, shared experts,
  `eh_proj`, `weights_proj`, embeddings, and `lm_head` remain BF16.
- MTP layer 78 was processed separately: 768 routed-expert weights are W4G64;
  23 non-expert tensors remain BF16. The MTP shared head reuses the BF16 target
  head at runtime.

## Validated runtime

The tested software stack was:

| Component | Version |
|---|---|
| vLLM | `0.23.1rc1.dev471+ge312c5cb2` |
| vLLM commit | [`e312c5cb25427e76fc3830ab14e7b6bc0963a55c`](https://github.com/vllm-project/vllm/commit/e312c5cb25427e76fc3830ab14e7b6bc0963a55c) |
| Python | 3.12.12 |
| PyTorch | 2.11.0+cu130 |
| CUDA | 13.0 |
| Hardware | 8 × NVIDIA H20 96 GB, tensor parallel size 8 |

`apply_vllm_glm52_patches.py` contains the source-level compatibility changes
used for this checkpoint. It is strictly hash-pinned to the vLLM commit and
Triton source above, stages all edits before installation, compiles and verifies
every output, installs files atomically, and is idempotent. It refuses unknown
or partially modified sources instead of attempting a fuzzy patch.

The patch covers:

1. GLM-5.2 sparse-indexer and missing-parameter guards, plus AutoRound routed
   expert and MTP checkpoint namespace compatibility.
2. Header-first safetensors filtering so MTP eager loading does not read every
   unrelated backbone shard payload.
3. CUDA Graph pre-capture warmup and real cubin generation through `ptxas`.
4. Explicit PyNCCL loader/device handling and correct CUDA-device binding for
   the CPU KV-cache pinning thread.

These changes affect loading and serving only; they do not modify model weights.
Use a separate environment for this pinned runtime. A newer vLLM version may
already contain equivalent fixes and should be validated independently.

## vLLM serving example

The following is a public, minimal reproduction of the tested serving profile.
It assumes that vLLM has already been built at the commit above and that
`ptxas` is available on `PATH`. Replace the example paths and public model name.

```bash
export VENV=/path/to/vllm-venv
export MODEL_DIR=/path/to/GLM-5.2-AutoRound-W4G64

"$VENV/bin/python" "$MODEL_DIR/apply_vllm_glm52_patches.py" \
  --venv-root "$VENV" \
  --model-dir "$MODEL_DIR" \
  --apply

# Keep compilation and CUDA Graphs enabled. These settings match the validated
# single-node runtime; adapt NCCL transport selection to your topology.
export VLLM_DISABLE_COMPILE_CACHE=1
export VLLM_DISABLE_PYNCCL=0
export NCCL_CUMEM_ENABLE=0
export NCCL_CUMEM_HOST_ENABLE=0
export VLLM_ALLREDUCE_USE_SYMM_MEM=0
export VLLM_USE_NCCL_SYMM_MEM=0
export TRITON_STORE_BINARY_ONLY=0
export CUDA_MODULE_LOADING=EAGER

"$VENV/bin/vllm" serve "$MODEL_DIR" \
  --served-model-name glm-5.2-autoround-w4g64 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --dtype auto \
  --max-model-len 130000 \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --chat-template-content-format string \
  --trust-remote-code \
  --safetensors-load-strategy eager \
  --distributed-timeout-seconds 1800 \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --compilation-config.pass_config.fuse_allreduce_rms false \
  --speculative-config '{"method":"mtp"}' \
  --spec-tokens 2 \
  --kv-offloading-size 256 \
  --kv-offloading-backend native \
  --disable-hybrid-kv-cache-manager
```

The 256 GiB CPU KV offload setting is optional and requires sufficient pinned
host memory. Reduce or remove it for systems without that capacity.

Example OpenAI-compatible request:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-autoround-w4g64",
    "messages": [{"role": "user", "content": "Explain and solve x^2 = 1 (mod 105)."}],
    "max_tokens": 8192,
    "temperature": 0.6,
    "top_p": 0.95,
    "stream": true
  }'
```

Reasoning models may use several thousand tokens before final content. For
long-horizon coding-agent workloads, a 32K output allowance was materially more
reliable than an 8K allowance.

## Measured performance

Measurements below used the validated stack on one 8 × H20 node with TP8,
130K maximum context, MTP speculative depth 2, `max_num_seqs=8`, an 8,192-token
batch ceiling, prefix caching, CUDA Graphs, and 256 GiB native CPU KV offload.

The fixed-shape serving test used the same 117-token prompt, exactly 512 output
tokens per request, streaming responses, and a 60-second load window. Throughput
is the server generation-token counter divided by wall time.

| Concurrency | Successful requests | Aggregate generation | Mean / P95 TTFT | Mean prefill | Mean decode |
|---:|---:|---:|---:|---:|---:|
| 3 | 30 / 30 | 229.09 tok/s | 0.411 / 0.859 s | 0.351 s | 6.116 s |
| 5 | 40 / 40 | 310.30 tok/s | 0.438 / 0.679 s | 0.361 s | 7.586 s |
| 8 | 56 / 56 | 441.11 tok/s | 0.437 / 0.692 s | 0.346 s | 8.417 s |

No request preemption or metrics scrape error occurred in these windows. MTP
accepted 57.7%–58.7% of drafted tokens, for an effective accepted length of
about 2.16 tokens per verification iteration.

A separate reasoning smoke test produced 4,083 tokens at 110.16 tok/s after a
0.266-second TTFT, stopped normally, returned both reasoning and final content,
and correctly enumerated the eight solutions of `x² ≡ 1 (mod 105)`.

On isolated coding-agent tasks, the model passed all public and held-out tests
for the four completed single-concurrency tasks, ranging from configuration
merging to a parser/formatter/evaluator. In concurrent agent runs, aggregate
generation was 218.20 tok/s at concurrency 3 and 278.04–292.79 tok/s at
concurrency 5. These are workload observations, not standardized leaderboard
results.

Performance varies with prompt length, output distribution, MTP acceptance,
cache state, host-memory bandwidth, software build, and GPU topology. The
numbers above should not be compared with other model cards unless the harness
and serving configuration are matched.

## Quality and limitations

- This checkpoint has not been evaluated against the BF16 base model with a
  comprehensive, identical public benchmark harness. No claim of lossless
  quantization is made.
- Only the vLLM commit and 8 × H20 configuration documented above were fully
  validated. Other vLLM revisions, runtimes, GPU architectures, and tensor
  parallel sizes may require additional work.
- Quantization is concentrated in routed experts, but quality can still differ
  from the BF16 base model, especially on rare domains or long reasoning paths.
- The checkpoint inherits the capabilities, risks, usage restrictions, and
  limitations described in the
  [official GLM-5.2 model card](https://huggingface.co/zai-org/GLM-5.2).
- Users are responsible for application-specific safety, privacy, reliability,
  and output-quality evaluation before deployment.

## License and references

This quantized checkpoint follows the base model's
[MIT license](https://huggingface.co/zai-org/GLM-5.2/blob/main/LICENSE). Review
the base repository and license before redistribution or production use.

- [Official GLM-5.2 model card](https://huggingface.co/zai-org/GLM-5.2)
- [GLM-5 technical report](https://arxiv.org/abs/2602.15763)
- [AutoRound project](https://github.com/intel/auto-round)
- [AutoRound paper](https://arxiv.org/abs/2309.05516)
- [vLLM](https://github.com/vllm-project/vllm)