File size: 4,035 Bytes
73b8173 e0e68e2 73b8173 6252298 73b8173 98e9596 adb8a56 73b8173 cc67f34 6780005 20edbd7 6780005 73b8173 902e457 73b8173 411e3c2 902e457 73b8173 f6c4264 73b8173 f6c4264 73b8173 cc67f34 73b8173 411e3c2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | ---
license: mit
base_model:
- zai-org/GLM-5.2
library_name: transformers
tags:
- compressed-tensors
- vLLM
---
# RedHatAI/GLM-5.2-FP8-NVFP4
This is a quantized version of `zai-org/GLM-5.2` with MoE layers quantized to NVFP4 and attention layers quantized to FP8 block
## Usage
This model is intended for deployment with vLLM and requires the following fix: https://github.com/vllm-project/vllm/pull/47780.
You can serve the model using
```bash
vllm serve RedHatAI/GLM-5.2-NVFP4-FP8 \
--tensor-parallel-size 4 \
--reasoning-parser glm45 \
--tool-call-parser glm47 \
--enable-auto-tool-choice \
--kv-cache-dtype fp8
```
## Creation Process
This model was created using [LLM Compressor](https://github.com/vllm-project/llm-compressor). The example script can be found in `examples/quantizing_moe/glm5_example.py` [[Example] GLM5.2 Example](https://github.com/vllm-project/llm-compressor/pull/2869). Quantizing the model with data parallelism and 6xA100 takes about 3 hours.
<details><summary>LLM Compressor Creation Script</summary>
```python
import torch
from compressed_tensors.offload import init_dist
from compressed_tensors.quantization.quant_scheme import (
FP8_BLOCK,
NVFP4,
QuantizationScheme,
)
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.datasets.utils import get_rank_partition
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context
# Load the model
init_dist()
model_id = "zai-org/GLM-5.2"
with load_context():
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto_offload",
max_memory={},
offload_folder="/mnt/nvme-data/engine/kylesayrs/offload_folder",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
# Select calibration dataset.
DATASET_ID = "HuggingFaceH4/ultrachat_200k"
DATASET_SPLIT = "train_sft"
# Select number of samples. 512 samples is a good place to start.
# Increasing the number of samples can improve accuracy.
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048
# Load dataset and preprocess.
ds = load_dataset(
DATASET_ID, split=get_rank_partition(DATASET_SPLIT, NUM_CALIBRATION_SAMPLES)
)
ds = ds.shuffle(seed=42)
def preprocess(example):
return {
"text": tokenizer.apply_chat_template(
example["messages"],
tokenize=False,
)
}
ds = ds.map(preprocess)
# Tokenize inputs.
def tokenize(sample):
return tokenizer(
sample["text"],
padding=False,
max_length=MAX_SEQUENCE_LENGTH,
truncation=True,
add_special_tokens=False,
)
ds = ds.map(tokenize, remove_columns=ds.column_names)
# Configure the quantization algorithm to run.
recipe = QuantizationModifier(
config_groups={
"attention_shared_experts": QuantizationScheme(
targets=[r"re:.*self_attn\..*"],
**FP8_BLOCK,
),
"mlp": QuantizationScheme(
targets=[r"re:.*mlp\..*"],
**NVFP4,
),
},
ignore=[
r"re:^model\.layers\.[0-2]\..*"
r"re:.*mlp\.gate.*", # not technically necessary
r"re:.*indexer\.weights_proj$", # sensitive to quantization
r"lm_head",
],
)
# Apply algorithms.
oneshot(
model=model,
dataset=ds,
batch_size=4,
recipe=recipe,
shuffle_calibration_samples=False,
)
# Save to disk compressed.
# Note: base checkpoint generation_config needs fixing for newer transformers versions
model.generation_config.top_p = None
SAVE_DIR = (
"/mnt/nvme-data/engine/kylesayrs/"
+ model_id.rstrip("/").split("/")[-1]
+ "-NVFP4-FP8"
)
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
torch.distributed.destroy_process_group()
```
</details>
## Evaluation ##
| Benchmark | `zai-org/GLM-5.2` | `RedHatAI/GLM-5.2-NVFP4-FP8` |
| - | - | - |
| GPQA-Diamond | 91.2 | 89.1 |
|