GLM-5.2-W4A8-CT

GLM-5.2-W4A8-CT is a 4-bit quantization of zai-org/GLM-5.2-FP8 for vLLM. It stores the routed MoE experts as group-wise INT4 weights for dynamic FP8 activations using the compressed-tensors format. The dense linear layers use block FP8 weights.

The quantized weights are the same as those in PhalaCloud/GLM-5.2-W4AFP8. That repository uses SGLang's W4AFP8 serialization; this repository packages the weights in vLLM's compressed-tensors serialization. This packaging step does not recalibrate or requantize the expert values. The runtime-aware AWQ-calibrated MTP layer is included.

Requirements

• NVIDIA Hopper GPU (H100, H200, or GH200)
• A recent vLLM build with compressed-tensors W4A8 MoE support
• About 373 GiB of storage for the checkpoint

The checkpoint was validated with unmodified vLLM commit 34b560b72 (0.23.1rc1.dev786+g34b560b72.precompiled) on one GH200 using UVA host-memory offload.

Serve with vLLM

The following configuration is intended for an eight-GPU Hopper server where the weights fit in aggregate VRAM:

vllm serve PhalaCloud/GLM-5.2-W4A8-CT \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

The server exposes the OpenAI-compatible chat-completions API. The validated single-GH200 deployment uses vLLM's UVA offloader because the model is larger than HBM. That setup is functional but substantially slower than keeping all weights in GPU memory.

Checkpoint format

Component	Stored format
Routed MoE expert weights	Group-wise INT4, group size 128
Routed MoE activations	Dynamic per-token FP8 E4M3
Dense linear weights	Block FP8
MTP layer	Included, with runtime-aware AWQ expert calibration
Serialization	Hugging Face safetensors, 41 shards

At startup, vLLM converts the portable compressed-tensors INT4 representation to the tiled layout required by its CUTLASS W4A8 kernels. This post-load step can add noticeable cold-start time for heavily offloaded deployments.

Validation

The converted checkpoint has passed these checks:

• all 41 safetensors shards load successfully in unmodified vLLM;
• native OpenAI-format tool calls work with the glm47 parser;
• the bundled MTP layer produces accepted speculative tokens;
• HumanEval: 159/164 base tests passed (96.95%);
• HumanEval+: 151/164 combined tests passed (92.07%).

BFCL single-turn tool-calling evaluation is in progress. Results will be added when the run completes.

Conversion details

The routed-expert INT4 weights originate from quantizing GLM-5.2-FP8. For this release, those already-quantized weights were converted from SGLang's W4AFP8 serialization to vLLM's compressed-tensors mixed-precision schema. The format conversion repacked 58,368 expert INT4 tensors, renamed 654 FP8 scale tensors, preserved sparse-attention indexer scales, and produced 176,997 tensors across 41 shards. The included vllm_conversion_manifest.json records the source hashes and conversion counts.

Limitations

• Only Hopper has been tested.
• A single-GPU deployment requires aggressive host-memory offload and is much slower than an all-VRAM deployment.
• Cold starts include a CUTLASS layout-conversion step.
• The full BFCL result is not yet available.

License

MIT, inherited from GLM-5.2. See the original model card for architecture, training, and citation details.