---
language:
- en
- zh
- ko
- ja
license: mit
base_model: zai-org/GLM-5.1
tags:
- glm
- glm-5.1
- moe
- quantized
- fp8
- float8

pipeline_tag: text-generation
library_name: transformers
model_name: GLM-5.1-FP8-Dynamic
quantized_by: mconcat
---

# GLM-5.1-FP8-Dynamic

FP8 dynamic quantized version of [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1).

This checkpoint preserves the GLM-5.1 MoE + MLA + DSA architecture from the BF16 source, with all Linear weights quantized to FP8 E4M3 for ~2x compression.

## Quantization Strategy

Per-channel FP8 E4M3 weight quantization with dynamic per-token activation scaling:

| Precision | Layers |
|-----------|--------|
| **FP8 E4M3** | All Linear weights: MLA projections, MLP gate/up/down, expert projections, DSA indexer |
| **BF16** | `lm_head`, `embed_tokens`, MoE router gates, norms |

Architecture match with the BF16 source:

- `model_type=glm_moe_dsa`
- `78` layers (3 dense + 75 MoE, `first_k_dense_replace=3`)
- `n_routed_experts=256`, `num_experts_per_tok=8`, `n_shared_experts=1`

- `max_position_embeddings=202752`
- `hidden_size=6144`, `moe_intermediate_size=2048`
- `vocab_size=154880`

## Calibration

- 512 self-calibration samples generated from GLM-5.1 via OpenRouter (top-tier provider routing)
- 8 diverse categories: math, code, logic, analysis, creative writing, general knowledge, agentic/tool-calling, Korean
- Activation statistics collected layer-by-layer for per-channel FP8 scale computation

## Usage

### SGLang

```bash
python3 -m sglang.launch_server --model mconcat/GLM-5.1-FP8-Dynamic \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --trust-remote-code \
  --mem-fraction-static 0.80
```

### vLLM

```bash
vllm serve mconcat/GLM-5.1-FP8-Dynamic \
  --tensor-parallel-size 8 \
  --dtype bfloat16 \
  --trust-remote-code
```


## Compatibility

| Framework | Supported | Notes |
|-----------|-----------|-------|
| vLLM >= 0.19.0 | Yes | Requires `glm_moe_dsa` + compressed-tensors support |
| SGLang >= 0.5.10 | Yes | Requires GLM-5.1 architecture support |
| transformers >= 5.4.0 | Yes | Direct loading with `device_map="auto"` |

## Notes

- This is a 754B MoE model (~40B active per token). Requires multi-GPU setup for inference (8x 80GB+ GPUs recommended).
- FP8 E4M3 provides ~2x compression over BF16 with minimal quality degradation.
- Compatible with Hopper (SM90) and Blackwell GPUs.
- Dynamic activation scaling — scales computed at inference time, not baked into the checkpoint.
- GLM-5.1 does not ship MTP weights despite `num_nextn_predict_layers=1` in config.

## Blackwell SM120 Patch (RTX PRO 6000 / Workstation GPUs)

If running on Blackwell workstation GPUs (SM 12.0), vLLM 0.19.0 requires patches for FlashMLA sparse attention support:

```bash
# Patch 1: FlashMLA ops - add SM120 to sparse support check
FLASHMLA_OPS=$(python -c "import vllm, os; print(os.path.join(os.path.dirname(vllm.__file__), 'v1/attention/ops/flashmla.py'))") && \
sed -i 's/is_device_capability_family(90)\s*or current_platform.is_device_capability_family(100)/is_device_capability_family(90) or current_platform.is_device_capability_family(100) or current_platform.is_device_capability_family(120)/' "$FLASHMLA_OPS"

# Patch 2: FlashMLA sparse backend - add SM12 to capability check
FLASHMLA_SPARSE=$(python -c "import vllm, os; print(os.path.join(os.path.dirname(vllm.__file__), 'v1/attention/backends/mla/flashmla_sparse.py'))") && \
sed -i 's/return capability.major in \[9, 10\]/return capability.major in [9, 10, 12]/' "$FLASHMLA_SPARSE"

# Patch 3: FlashMLA dense backend (if exists)
FLASHMLA_DENSE=$(python -c "import vllm, os; print(os.path.join(os.path.dirname(vllm.__file__), 'v1/attention/backends/mla/flashmla.py'))") && \
sed -i 's/return capability.major in \[9, 10\]/return capability.major in [9, 10, 12]/' "$FLASHMLA_DENSE" 2>/dev/null || true
```

These patches add SM120 (Blackwell workstation) to the supported compute capability list for GLM-5.1's DSA sparse attention.

## Quantization Process

- **Tool**: Custom layer-by-layer pipeline with native `torch.float8_e4m3fn` dtype
- **Hardware**: Single NVIDIA RTX PRO 6000 Blackwell (96 GB), processed one layer at a time
- **Time**: ~319 minutes for 78 layers
- **Calibration**: 256 samples, per-module activation statistics with MoE expert input hooks