Hugging Face's logo

GadflyII
/
GLM-4.7-Flash-NVFP4

Text Generation
Transformers
Safetensors
English
Chinese
glm4_moe_lite
Mixture of Experts
nvfp4
quantized
vllm
glm
30b
conversational
compressed-tensors
Model card Files
xet
 Community
8
GadflyII commited on
Commit
8cd3aac
·
verified ·
1 Parent(s): 0b9368a

Upload folder using huggingface_hub

Browse files
Files changed (12) hide show
  1. .gitattributes +1 -0
  2. README.md +148 -0
  3. calibration_amax.json +0 -0
  4. config.json +78 -0
  5. generation_config.json +11 -0
  6. model-00000-of-00004.safetensors +3 -0
  7. model-00001-of-00004.safetensors +3 -0
  8. model-00002-of-00004.safetensors +3 -0
  9. model-00003-of-00004.safetensors +3 -0
  10. model.safetensors.index.json +0 -0
  11. tokenizer.json +3 -0
  12. tokenizer_config.json +321 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,148 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ - zh
6
+ base_model: zai-org/GLM-4.7-Flash
7
+ tags:
8
+ - moe
9
+ - nvfp4
10
+ - quantized
11
+ - vllm
12
+ - glm
13
+ library_name: transformers
14
+ pipeline_tag: text-generation
15
+ ---
16
+
17
+ # GLM-4.7-Flash NVFP4 (Mixed Precision)
18
+
19
+ This is a **mixed precision NVFP4 quantization** of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash), a 31B parameter Mixture-of-Experts model.
20
+
21
+ ## Quantization Strategy
22
+
23
+ Based on NVIDIA's approach for DeepSeek-V3, this model uses **mixed precision** to preserve accuracy:
24
+
25
+ | Component | Precision | Rationale |
26
+ |-----------|-----------|-----------|
27
+ | MLP Experts | FP4 (E2M1) | 64 routed experts, 4 active per token |
28
+ | Dense MLP | FP4 (E2M1) | First layer dense MLP |
29
+ | **Attention (MLA)** | **BF16** | Low-rank compressed Q/KV projections are sensitive |
30
+ | Norms, Gates, Embeddings | BF16 | Standard practice |
31
+
32
+ ## Performance
33
+
34
+ | Metric | BF16 | Uniform FP4 | **This Model** |
35
+ |--------|------|-------------|----------------|
36
+ | MMLU-Pro | 24.83% | 16.84% | **23.55%** |
37
+ | Size | 62.4 GB | 18.9 GB | **20.4 GB** |
38
+ | Compression | 1x | 3.3x | **3.1x** |
39
+ | Accuracy Loss | - | -8.0% | **-1.3%** |
40
+
41
+ **Key Finding**: Uniform FP4 quantization causes 8% accuracy loss on this model due to the sensitive MLA attention layers. Mixed precision reduces this to only 1.3% while maintaining 3.1x compression.
42
+
43
+ ## Usage
44
+
45
+ ### Requirements
46
+
47
+ - **vLLM**: 0.14.0+ (for compressed-tensors NVFP4 support)
48
+ - **transformers**: 5.0.0+ (for `glm4_moe_lite` architecture)
49
+ - **GPU**: NVIDIA GPU with FP4 tensor core support (Blackwell, Hopper, Ada Lovelace)
50
+
51
+ ### Installation
52
+
53
+ ```bash
54
+ pip install vllm>=0.14.0
55
+ pip install git+https://github.com/huggingface/transformers.git
56
+ ```
57
+
58
+ ### Inference with vLLM
59
+
60
+ ```python
61
+ from vllm import LLM, SamplingParams
62
+
63
+ model = LLM(
64
+ "GadflyII/GLM-4.7-Flash-NVFP4",
65
+ tensor_parallel_size=1,
66
+ max_model_len=4096,
67
+ trust_remote_code=True,
68
+ gpu_memory_utilization=0.85,
69
+ )
70
+
71
+ params = SamplingParams(temperature=0.7, max_tokens=512)
72
+ outputs = model.generate(["Explain quantum computing in simple terms."], params)
73
+ print(outputs[0].outputs[0].text)
74
+ ```
75
+
76
+ ### Serving with vLLM
77
+
78
+ ```bash
79
+ vllm serve GadflyII/GLM-4.7-Flash-NVFP4 \
80
+ --tensor-parallel-size 1 \
81
+ --max-model-len 4096 \
82
+ --trust-remote-code
83
+ ```
84
+
85
+ ## Model Details
86
+
87
+ - **Base Model**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
88
+ - **Architecture**: `Glm4MoeLiteForCausalLM`
89
+ - **Parameters**: 31B total (5.5B active per token)
90
+ - **MoE Configuration**: 64 routed experts, 4 active, 1 shared expert
91
+ - **Layers**: 47
92
+ - **Context Length**: 202,752 tokens (max)
93
+ - **Languages**: English, Chinese
94
+
95
+ ## Quantization Details
96
+
97
+ - **Format**: compressed-tensors (NVFP4)
98
+ - **Block Size**: 16
99
+ - **Scale Format**: FP8 (E4M3)
100
+ - **Calibration**: 128 samples from neuralmagic/calibration dataset
101
+ - **Full Expert Calibration**: All 64 experts calibrated per sample
102
+
103
+ ## Why Mixed Precision?
104
+
105
+ GLM-4.7-Flash uses **MLA (Multi-head Latent Attention)** from DeepSeek-V2, which compresses queries and key-values through low-rank projections:
106
+
107
+ - `q_a_proj`, `q_b_proj` - Query compression
108
+ - `kv_a_proj`, `kv_b_proj` - Key-Value compression
109
+
110
+ These compressed representations are highly sensitive to quantization noise. NVIDIA's DeepSeek-V3 NVFP4 release explicitly keeps all attention layers in higher precision for this reason.
111
+
112
+ ## Evaluation
113
+
114
+ MMLU-Pro results by category:
115
+
116
+ | Category | Accuracy |
117
+ |----------|----------|
118
+ | Biology | 47.42% |
119
+ | Psychology | 42.48% |
120
+ | Health | 34.84% |
121
+ | Economics | 34.48% |
122
+ | History | 30.71% |
123
+ | Philosophy | 30.06% |
124
+ | Other | 25.87% |
125
+ | Computer Science | 21.46% |
126
+ | Business | 16.98% |
127
+ | Law | 16.35% |
128
+ | Physics | 14.70% |
129
+ | Math | 14.29% |
130
+ | Engineering | 14.04% |
131
+ | Chemistry | 13.34% |
132
+
133
+ ## Citation
134
+
135
+ If you use this model, please cite the original GLM-4.7-Flash:
136
+
137
+ ```bibtex
138
+ @misc{glm4flash2025,
139
+ title={GLM-4.7-Flash},
140
+ author={Zhipu AI},
141
+ year={2025},
142
+ howpublished={\url{https://huggingface.co/zai-org/GLM-4.7-Flash}}
143
+ }
144
+ ```
145
+
146
+ ## License
147
+
148
+ This model inherits the Apache 2.0 license from the base model.
calibration_amax.json ADDED
The diff for this file is too large to render. See raw diff
 
config.json ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Glm4MoeLiteForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "pad_token_id": 154820,
8
+ "eos_token_id": [
9
+ 154820,
10
+ 154827,
11
+ 154829
12
+ ],
13
+ "hidden_act": "silu",
14
+ "hidden_size": 2048,
15
+ "intermediate_size": 10240,
16
+ "max_position_embeddings": 202752,
17
+ "model_type": "glm4_moe_lite",
18
+ "moe_intermediate_size": 1536,
19
+ "topk_method": "noaux_tc",
20
+ "norm_topk_prob": true,
21
+ "num_attention_heads": 20,
22
+ "n_group": 1,
23
+ "topk_group": 1,
24
+ "n_routed_experts": 64,
25
+ "n_shared_experts": 1,
26
+ "routed_scaling_factor": 1.8,
27
+ "num_experts_per_tok": 4,
28
+ "first_k_dense_replace": 1,
29
+ "num_hidden_layers": 47,
30
+ "num_key_value_heads": 20,
31
+ "num_nextn_predict_layers": 1,
32
+ "partial_rotary_factor": 1.0,
33
+ "rms_norm_eps": 1e-05,
34
+ "rope_scaling": null,
35
+ "rope_theta": 1000000,
36
+ "tie_word_embeddings": false,
37
+ "dtype": "bfloat16",
38
+ "transformers_version": "5.0.0rc0",
39
+ "q_lora_rank": 768,
40
+ "kv_lora_rank": 512,
41
+ "qk_nope_head_dim": 192,
42
+ "qk_rope_head_dim": 64,
43
+ "v_head_dim": 256,
44
+ "vocab_size": 154880,
45
+ "quantization_config": {
46
+ "quant_method": "compressed-tensors",
47
+ "config_groups": {
48
+ "group_0": {
49
+ "input_activations": {
50
+ "num_bits": 4,
51
+ "type": "float",
52
+ "strategy": "tensor_group",
53
+ "group_size": 16,
54
+ "symmetric": true,
55
+ "dynamic": true
56
+ },
57
+ "weights": {
58
+ "num_bits": 4,
59
+ "type": "float",
60
+ "strategy": "tensor_group",
61
+ "group_size": 16,
62
+ "symmetric": true
63
+ },
64
+ "targets": [
65
+ "Linear"
66
+ ]
67
+ }
68
+ },
69
+ "format": "nvfp4-pack-quantized",
70
+ "ignore": [
71
+ "lm_head",
72
+ "re:.*embed.*",
73
+ "re:.*gate$",
74
+ "re:.*self_attn.*"
75
+ ],
76
+ "dynamic": true
77
+ }
78
+ }
generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "eos_token_id": [
4
+ 154820,
5
+ 154827,
6
+ 154829
7
+ ],
8
+ "pad_token_id": 154820,
9
+ "temperature": 1.0,
10
+ "transformers_version": "5.0.0.dev0"
11
+ }
model-00000-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88b660b55607cde8a869c2f841b1bb042d48b611c67ca6e812f188da5bf448a0
3
+ size 5369604384
model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b9cdccf1c4e022f4107f0e156cfcb3de6e0e7a8e0b8eacc3262014f75e448168
3
+ size 5369189784
model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eec9fd341824239a59e585a0a58dc9dc5e6e69ab221ae2e56b367df3f1e9ae66
3
+ size 5356474456
model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c3bc2d62c5a2b96493af5338df2b852a699ddab37f9bee00ff06cb8556f4eddc
3
+ size 4349953720
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d
3
+ size 20217442
tokenizer_config.json ADDED
@@ -0,0 +1,321 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "154820": {
4
+ "content": "<|endoftext|>",
5
+ "single_word": false,
6
+ "lstrip": false,
7
+ "rstrip": false,
8
+ "normalized": false,
9
+ "special": true
10
+ },
11
+ "154821": {
12
+ "content": "[MASK]",
13
+ "single_word": false,
14
+ "lstrip": false,
15
+ "rstrip": false,
16
+ "normalized": false,
17
+ "special": true
18
+ },
19
+ "154822": {
20
+ "content": "[gMASK]",
21
+ "single_word": false,
22
+ "lstrip": false,
23
+ "rstrip": false,
24
+ "normalized": false,
25
+ "special": true
26
+ },
27
+ "154823": {
28
+ "content": "[sMASK]",
29
+ "single_word": false,
30
+ "lstrip": false,
31
+ "rstrip": false,
32
+ "normalized": false,
33
+ "special": true
34
+ },
35
+ "154824": {
36
+ "content": "<sop>",
37
+ "single_word": false,
38
+ "lstrip": false,
39
+ "rstrip": false,
40
+ "normalized": false,
41
+ "special": true
42
+ },
43
+ "154825": {
44
+ "content": "<eop>",
45
+ "single_word": false,
46
+ "lstrip": false,
47
+ "rstrip": false,
48
+ "normalized": false,
49
+ "special": true
50
+ },
51
+ "154826": {
52
+ "content": "<|system|>",
53
+ "single_word": false,
54
+ "lstrip": false,
55
+ "rstrip": false,
56
+ "normalized": false,
57
+ "special": true
58
+ },
59
+ "154827": {
60
+ "content": "<|user|>",
61
+ "single_word": false,
62
+ "lstrip": false,
63
+ "rstrip": false,
64
+ "normalized": false,
65
+ "special": true
66
+ },
67
+ "154828": {
68
+ "content": "<|assistant|>",
69
+ "single_word": false,
70
+ "lstrip": false,
71
+ "rstrip": false,
72
+ "normalized": false,
73
+ "special": true
74
+ },
75
+ "154829": {
76
+ "content": "<|observation|>",
77
+ "single_word": false,
78
+ "lstrip": false,
79
+ "rstrip": false,
80
+ "normalized": false,
81
+ "special": true
82
+ },
83
+ "154830": {
84
+ "content": "<|begin_of_image|>",
85
+ "single_word": false,
86
+ "lstrip": false,
87
+ "rstrip": false,
88
+ "normalized": false,
89
+ "special": true
90
+ },
91
+ "154831": {
92
+ "content": "<|end_of_image|>",
93
+ "single_word": false,
94
+ "lstrip": false,
95
+ "rstrip": false,
96
+ "normalized": false,
97
+ "special": true
98
+ },
99
+ "154832": {
100
+ "content": "<|begin_of_video|>",
101
+ "single_word": false,
102
+ "lstrip": false,
103
+ "rstrip": false,
104
+ "normalized": false,
105
+ "special": true
106
+ },
107
+ "154833": {
108
+ "content": "<|end_of_video|>",
109
+ "single_word": false,
110
+ "lstrip": false,
111
+ "rstrip": false,
112
+ "normalized": false,
113
+ "special": true
114
+ },
115
+ "154834": {
116
+ "content": "<|begin_of_audio|>",
117
+ "single_word": false,
118
+ "lstrip": false,
119
+ "rstrip": false,
120
+ "normalized": false,
121
+ "special": true
122
+ },
123
+ "154835": {
124
+ "content": "<|end_of_audio|>",
125
+ "single_word": false,
126
+ "lstrip": false,
127
+ "rstrip": false,
128
+ "normalized": false,
129
+ "special": true
130
+ },
131
+ "154836": {
132
+ "content": "<|begin_of_transcription|>",
133
+ "single_word": false,
134
+ "lstrip": false,
135
+ "rstrip": false,
136
+ "normalized": false,
137
+ "special": true
138
+ },
139
+ "154837": {
140
+ "content": "<|end_of_transcription|>",
141
+ "single_word": false,
142
+ "lstrip": false,
143
+ "rstrip": false,
144
+ "normalized": false,
145
+ "special": true
146
+ },
147
+ "154838": {
148
+ "content": "<|code_prefix|>",
149
+ "single_word": false,
150
+ "lstrip": false,
151
+ "rstrip": false,
152
+ "normalized": false,
153
+ "special": false
154
+ },
155
+ "154839": {
156
+ "content": "<|code_middle|>",
157
+ "single_word": false,
158
+ "lstrip": false,
159
+ "rstrip": false,
160
+ "normalized": false,
161
+ "special": false
162
+ },
163
+ "154840": {
164
+ "content": "<|code_suffix|>",
165
+ "single_word": false,
166
+ "lstrip": false,
167
+ "rstrip": false,
168
+ "normalized": false,
169
+ "special": false
170
+ },
171
+ "154841": {
172
+ "content": "<think>",
173
+ "single_word": false,
174
+ "lstrip": false,
175
+ "rstrip": false,
176
+ "normalized": false,
177
+ "special": false
178
+ },
179
+ "154842": {
180
+ "content": "</think>",
181
+ "single_word": false,
182
+ "lstrip": false,
183
+ "rstrip": false,
184
+ "normalized": false,
185
+ "special": false
186
+ },
187
+ "154843": {
188
+ "content": "<tool_call>",
189
+ "single_word": false,
190
+ "lstrip": false,
191
+ "rstrip": false,
192
+ "normalized": false,
193
+ "special": false
194
+ },
195
+ "154844": {
196
+ "content": "</tool_call>",
197
+ "single_word": false,
198
+ "lstrip": false,
199
+ "rstrip": false,
200
+ "normalized": false,
201
+ "special": false
202
+ },
203
+ "154845": {
204
+ "content": "<tool_response>",
205
+ "single_word": false,
206
+ "lstrip": false,
207
+ "rstrip": false,
208
+ "normalized": false,
209
+ "special": false
210
+ },
211
+ "154846": {
212
+ "content": "</tool_response>",
213
+ "single_word": false,
214
+ "lstrip": false,
215
+ "rstrip": false,
216
+ "normalized": false,
217
+ "special": false
218
+ },
219
+ "154847": {
220
+ "content": "<arg_key>",
221
+ "single_word": false,
222
+ "lstrip": false,
223
+ "rstrip": false,
224
+ "normalized": false,
225
+ "special": false
226
+ },
227
+ "154848": {
228
+ "content": "</arg_key>",
229
+ "single_word": false,
230
+ "lstrip": false,
231
+ "rstrip": false,
232
+ "normalized": false,
233
+ "special": false
234
+ },
235
+ "154849": {
236
+ "content": "<arg_value>",
237
+ "single_word": false,
238
+ "lstrip": false,
239
+ "rstrip": false,
240
+ "normalized": false,
241
+ "special": false
242
+ },
243
+ "154850": {
244
+ "content": "</arg_value>",
245
+ "single_word": false,
246
+ "lstrip": false,
247
+ "rstrip": false,
248
+ "normalized": false,
249
+ "special": false
250
+ },
251
+ "154851": {
252
+ "content": "/nothink",
253
+ "single_word": false,
254
+ "lstrip": false,
255
+ "rstrip": false,
256
+ "normalized": false,
257
+ "special": false
258
+ },
259
+ "154852": {
260
+ "content": "<|begin_of_box|>",
261
+ "single_word": false,
262
+ "lstrip": false,
263
+ "rstrip": false,
264
+ "normalized": false,
265
+ "special": false
266
+ },
267
+ "154853": {
268
+ "content": "<|end_of_box|>",
269
+ "single_word": false,
270
+ "lstrip": false,
271
+ "rstrip": false,
272
+ "normalized": false,
273
+ "special": false
274
+ },
275
+ "154854": {
276
+ "content": "<|image|>",
277
+ "single_word": false,
278
+ "lstrip": false,
279
+ "rstrip": false,
280
+ "normalized": false,
281
+ "special": false
282
+ },
283
+ "154855": {
284
+ "content": "<|video|>",
285
+ "single_word": false,
286
+ "lstrip": false,
287
+ "rstrip": false,
288
+ "normalized": false,
289
+ "special": false
290
+ }
291
+ },
292
+ "additional_special_tokens": [
293
+ "<|endoftext|>",
294
+ "[MASK]",
295
+ "[gMASK]",
296
+ "[sMASK]",
297
+ "<sop>",
298
+ "<eop>",
299
+ "<|system|>",
300
+ "<|user|>",
301
+ "<|assistant|>",
302
+ "<|observation|>",
303
+ "<|begin_of_image|>",
304
+ "<|end_of_image|>",
305
+ "<|begin_of_video|>",
306
+ "<|end_of_video|>",
307
+ "<|begin_of_audio|>",
308
+ "<|end_of_audio|>",
309
+ "<|begin_of_transcription|>",
310
+ "<|end_of_transcription|>"
311
+ ],
312
+ "clean_up_tokenization_spaces": false,
313
+ "do_lower_case": false,
314
+ "eos_token": "<|endoftext|>",
315
+ "extra_special_tokens": {},
316
+ "model_max_length": 128000,
317
+ "pad_token": "<|endoftext|>",
318
+ "padding_side": "left",
319
+ "remove_space": false,
320
+ "tokenizer_class": "PreTrainedTokenizer"
321
+ }