README.md

---
base_model: zai-org/GLM-5.3-Flash
base_model_relation: quantized
---

# GLM-5.3-Flash-AWQ-W4A16

Quantized version of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash).

If you come here with older cards like A100, A6000 or 3090, consider using [our vLLM fork](https://github.com/wtdcode/vllm-backport) which has served billions of tokens for frontier models with older GPUs.

## What is quantized

Weight-only INT4 (symmetric, group size 128) via AWQ, stored in the compressed-tensors `pack-quantized` format. Calibrated from the official FP8 release (dequantized to BF16 first).

**Quantized (INT4 W4A16):**
- Routed MoE experts in the 42 MoE decoder layers (layers 3-44): `mlp.experts.{0..287}.{gate_proj, up_proj, down_proj}` (~312B of the 321B parameters)

**Kept in BF16 (not quantized):**
- Token embedding (`embed_tokens`) and `lm_head`
- Linear attention / KDA (`self_attn.{q,k,v,o}_proj`, forget gate) and DSA/MLA attention (`q_a/q_b/kv_a/kv_b_proj`, indexer)
- Hyper-connections (`attn_hc.*`, `ffn_hc.*`)
- MoE router (`mlp.gate`) and shared experts (`mlp.shared_experts.*`)
- Dense MLPs of the first 3 layers
- Vision encoder (`model.visual.*`)
- NextN/MTP layer (`layers.45.*`, dequantized from the FP8 source to BF16)