Initial upload: GLM-4.7-Flash MMFP4 quantization (GPTQ + actorder)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,182 @@
+---
+license: mit
+language:
+  - en
+  - zh
+base_model: zai-org/GLM-4.7-Flash
+pipeline_tag: text-generation
+tags:
+  - quantized
+  - Mixture of Experts
+  - 4-bit
+  - GPTQ
+  - MMFP4
+  - glm
+  - metal-marlin
+  - moe
+library_name: transformers
+arxiv: "2508.06471"
+---
+
+# GLM-4.7-Flash-Marlin-MMFP4
+
+![](https://raw.githubusercontent.com/zai-org/GLM-4.5/refs/heads/main/resources/logo.svg)
+
+**MMFP4-quantized GLM-4.7-Flash** — a 30B-A3B MoE model compressed to **4 bits per weight** using GPTQ with actorder and Metal Marlin's E2M1 FP4 format.
+
+| Metric | Value |
+|--------|-------|
+| **Effective bits** | 4.0 bpw |
+| **Compression** | 4× vs FP16 |
+| **Model size** | ~16 GB (vs ~60 GB FP16) |
+| **Parameters** | 29.3B |
+| **Format** | HuggingFace sharded safetensors |
+
+## Model Description
+
+This is a quantized version of [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash), the strongest model in the 30B class that balances performance and efficiency.
+
+GLM-4.7-Flash features:
+
+- **30B-A3B MoE architecture** (64 experts + shared expert, 2-4 active per token)
+- **Multi-head Latent Attention (MLA)** for 8× KV cache compression
+- **State-of-the-art reasoning** (91.6% on AIME 2025, 59.2% on SWE-bench Verified)
+- **Bilingual** (English + Chinese)
+
+## Quantization Details
+
+Quantized using **MR-GPTQ** (Metal Marlin GPTQ) with CUDA acceleration:
+
+### Method
+
+- **Format**: MMFP4 (E2M1 FP4) — Metal Marlin's native FP4 format
+- **Quantization**: GPTQ with actorder (activation-order column permutation)
+- **Hessian calibration**: Pre-computed Hessians for attention layers
+- **Expert quantization**: Identity Hessian with actorder (no calibration data for MoE experts)
+- **Group size**: 128
+- **Hardware**: NVIDIA RTX 3090 Ti (CUDA-accelerated Cholesky factorization)
+
+### Quantization Statistics
+
+| Component | Bit Width | Notes |
+|-----------|-----------|-------|
+| Embeddings | FP16 | Full precision |
+| LM Head | FP16 | Full precision |
+| Attention (q/k/v/o) | 4-bit | GPTQ with Hessians |
+| MoE Experts (64×) | 4-bit | GPTQ with actorder |
+| Layer Norms | FP16 | Full precision |
+| Router Weights | FP16 | Full precision |
+
+- **Total tensors**: 19,066
+- **Shards**: 48 safetensors files
+- **Quantization time**: ~20 minutes (RTX 3090 Ti)
+
+## Files
+
+```
+GLM-4.7-Flash-Marlin-MMFP4/
+├── model-00001-of-00048.safetensors   # Layer 0 (embeddings)
+├── model-00002-of-00048.safetensors   # Layer 1
+├── ...
+├── model-00048-of-00048.safetensors   # Layer 47 + lm_head
+├── model.safetensors.index.json       # Weight map
+├── config.json                        # Model config
+├── generation_config.json
+├── tokenizer.json                     # Tokenizer
+└── tokenizer_config.json
+```
+
+## Usage
+
+### With Metal Marlin (Apple Silicon)
+
+```python
+from metal_marlin import MarlinForCausalLM
+from transformers import AutoTokenizer
+
+model = MarlinForCausalLM.from_pretrained(
+    "RESMP-DEV/GLM-4.7-Flash-Marlin-MMFP4",
+    device="mps"
+)
+tokenizer = AutoTokenizer.from_pretrained("zai-org/GLM-4.7-Flash")
+
+prompt = "<|user|>\nExplain quantum computing in simple terms.\n<|assistant|>\n"
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("mps")
+output = model.generate(input_ids, max_new_tokens=256, temperature=0.7)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+### Tensor Format
+
+Each quantized weight tensor has corresponding scale factors:
+
+- `{name}.weight`: Packed FP4 weights (uint8)
+- `{name}.scales`: FP16 per-group scales (group_size=128)
+
+## Hardware Requirements
+
+| Device | Memory | Notes |
+|--------|--------|-------|
+| Apple M4 Max | 36 GB+ | Via Metal Marlin |
+| Apple M2 Ultra | 64 GB+ | Via Metal Marlin |
+| NVIDIA RTX 3090 | 24 GB | With offloading |
+| NVIDIA RTX 4090 | 24 GB | Native |
+
+## Benchmarks
+
+### Original Model Performance (from Z.AI)
+
+| Benchmark | GLM-4.7-Flash | Qwen3-30B-A3B | GPT-OSS-20B |
+|-----------|---------------|---------------|-------------|
+| AIME 2025 | **91.6** | 85.0 | 91.7 |
+| GPQA | **75.2** | 73.4 | 71.5 |
+| SWE-bench Verified | **59.2** | 22.0 | 34.0 |
+| τ²-Bench | **79.5** | 49.0 | 47.7 |
+| BrowseComp | **42.8** | 2.29 | 28.3 |
+
+### Quantized Model Notes
+
+- GPTQ with actorder minimizes quality loss vs RTN
+- Expected degradation: ~1-2% on benchmarks vs FP16
+- E2M1 FP4 format optimized for Metal Performance Shaders
+
+## Comparison with Trellis Quant
+
+| Model | Format | Size | Bits | Method |
+|-------|--------|------|------|--------|
+| [GLM-4.7-Flash-Trellis-MM](https://huggingface.co/RESMP-DEV/GLM-4.7-Flash-Trellis-MM) | Trellis | 14 GB | 3.78 bpw | EXL3-style mixed precision |
+| **This model** | MMFP4 | 16 GB | 4.0 bpw | GPTQ + actorder |
+
+Choose **Trellis** for smaller size, **MMFP4** for simpler tensor format and potentially better compatibility.
+
+## Limitations
+
+- **Metal Marlin required** for optimal inference on Apple Silicon
+- **No speculative decoding** yet
+- **Quality loss**: ~1-2% on benchmarks vs FP16 (typical for 4-bit quantization)
+
+## Credits
+
+- **Original model**: [Z.AI / GLM Team](https://huggingface.co/zai-org/GLM-4.7-Flash)
+- **Quantization method**: GPTQ with actorder
+- **Quantization toolkit**: [Metal Marlin](https://github.com/RESMP-DEV/metal-marlin)
+
+## Citation
+
+If you use this model, please cite the original GLM-4.5 paper:
+
+```bibtex
+@misc{glm2025glm45,
+      title={GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models}, 
+      author={GLM Team and Aohan Zeng and Xin Lv and others},
+      year={2025},
+      eprint={2508.06471},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2508.06471}, 
+}
+```
+
+## License
+
+This quantized model inherits the **MIT License** from the original GLM-4.7-Flash model.

config.json

@@ -0,0 +1,45 @@
+{
+  "architectures": [
+    "Glm4MoeLiteForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "pad_token_id": 154820,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "intermediate_size": 10240,
+  "max_position_embeddings": 202752,
+  "model_type": "glm4_moe_lite",
+  "moe_intermediate_size": 1536,
+  "topk_method": "noaux_tc",
+  "norm_topk_prob": true,
+  "num_attention_heads": 20,
+  "n_group": 1,
+  "topk_group": 1,
+  "n_routed_experts": 64,
+  "n_shared_experts": 1,
+  "routed_scaling_factor": 1.8,
+  "num_experts_per_tok": 4,
+  "first_k_dense_replace": 1,
+  "num_hidden_layers": 47,
+  "num_key_value_heads": 20,
+  "num_nextn_predict_layers": 1,
+  "partial_rotary_factor": 1.0,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "tie_word_embeddings": false,
+  "dtype": "bfloat16",
+  "transformers_version": "5.0.0rc0",
+  "q_lora_rank": 768,
+  "kv_lora_rank": 512,
+  "qk_nope_head_dim": 192,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 256,
+  "vocab_size": 154880
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "transformers_version": "5.0.0.dev0"
+}

model-00001-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9c75eac1933aefdf174d6248f129c47b8f824380d67ac16d7cb5a6147926d69
+size 43672248

model-00002-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3f1b7a84f2090d15ee804ef8d548177f8b82b6ec7965ced8188988f3bb3a87d
+size 327833240

model-00003-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ecdc9ce4adb8c055d01b87d64076ad728ff68e666dc928e03da78bac377c920
+size 327833240

model-00004-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b006af82949a3315aef90b7baf1a2a9882bb3f43f9ce3c8cd65b4c76c9e38207
+size 327833240

model-00005-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac3e9ef0665b550cb3372a710f8821dc0a1a92064659d387638a5ecb40ca9884
+size 327833240

model-00006-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8a5e3764b1084fd6b83adf38607d33f086c3940f9fa058b0e45db98f47fda0c
+size 327833240

model-00007-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fe2dc6014f82667508eb8a83a3e784507d1a9848326777dbf93b3c8b9cf64d5
+size 327833240

model-00008-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a7aeadd95910d968c8865af1512c273b50ae3b52487c5a8844d198ed8ed1c94
+size 327833240

model-00009-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a37a52a176a1055b54499d69790be9e6d002d292a1f279e191b0bc0e78ebf6d4
+size 327833240

model-00010-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc713771ef57f3f1a2ddddbc8975840b039009211af61283f317fe82f3b0f803
+size 327833240

model-00011-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c9d3ea9e470875dc56792322a46722b6512fd3297cb6cd278f3946e72dc1b57
+size 327833648

model-00012-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ed7dd635f55f86789b8ad28da9b2f7e35720711e22b90c9a3b6d5238beb9e10
+size 327833648

model-00013-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:826e321f91601aa166993da7a792b2e1e31b9de8eac98ea3c629bec6edae15bc
+size 327833648

model-00014-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25b088ccb049f761d50f94ed9688929a046990b4ff9b9728fe4e4f6a8afe99c8
+size 327833648

model-00015-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf329d81710a99098bebd05de1d809174a9c7ad2ef508845b07416f1ba8a938b
+size 327833648

model-00016-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07bcea67819f6c8d448ad70dfd65ee40cdcf025732a94c20cf8c845013a4fd90
+size 327833648

model-00017-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8136af0e71dd04218a927dbf44b415c93160df2fa9299db978bb0d82e0c82e5b
+size 327833648

model-00018-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b27e105f72ddf7a01b1ab864ce664a8d3d232880e3ff380a626e20fd63dafe3
+size 327833648

model-00019-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f6f39fcfccbc3e9a4333bd484ac12cd816dc2c6fc266fc357e42a4a773ae051
+size 327833648

model-00020-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:660343ac01528ac8d18ca2d2e1af3158c23adb215284a724c619b074980ae42f
+size 327833648

model-00021-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:279237957148596abb9ce53e783e4583b3ac99ff6270cbe2242202f18891cc5d
+size 327833648

model-00022-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:350d4635ff553896c5a193f12721042083d2741baa34e4cb7fc2e4dd608e144d
+size 327833648

model-00023-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc0cd17264aeac8156ff8b620f759d5a6c4fa9fb376b31950f28935720320859
+size 327833648

model-00024-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e1bb3f7ecfb31ad86e0ca08eb7272b79041750a3d37e3fa53406746ffaed4fd
+size 327833648

model-00025-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:636c23bd06de9f265e7bd504de64f6b6f52d129a425dbab7eeaffa7854119ada
+size 327833648

model-00026-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd142bfc41b48bcf30f173febf3b425aa640aa2a87a88dc418948b0686437fb3
+size 327833648

model-00027-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74d684936b534f4da73488c37df32177b5f19fd7a349431bdb0b0edfe6f6567c
+size 327833648

model-00028-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17abe4e3ef81c14065300e33198a44237ac51880f383912f1cf3200d69e2dee6
+size 327833648

model-00029-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a2311764b9408632af65ae0dc21b5f0353986c3c47a0031a626098df65fc05d
+size 327833648

model-00030-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2b03130258c4c6c60026ceb61e0190020da76a0b9aef48e7f761932d7c656ce
+size 327833648

model-00031-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae64f3e55603f1a1e3a9d2fce50fb88a547d66f27acb31f8043a93eb97dbb2ee
+size 327833648

model-00032-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c21366f8b23ed73d2bcf75bdbab9c0122f17abed308b6ba3319060b9bc8f0adc
+size 327833648

model-00033-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daa7c2cfe969844ff866c80c7c46dee90de2abac06b886b246550a2b53bbbc25
+size 327833648

model-00034-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd7c93ce6893a3cc825987ca92c197261464e903506af51779ae5df5d50972f3
+size 327833648

model-00035-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:328c8c578a97cd6d37518f953613c22123f2f4de1475d7777fd3ea8b6b7f5163
+size 327833648

model-00036-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92cce91528e2230b446321c5aff6d543812512a3eb00acb80944c82aee588752
+size 327833648

model-00037-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdb2b3661d4333703c1e7d6ed30ecdc94a24e9c7a45ea390166c24cceb01ce9a
+size 327833648

model-00038-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfce66c57e056ff78615e0ee624bd49bc88c1dba1eb23b9c64bbb732b48d6a7f
+size 327833648

model-00039-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:771f8155c915eeeb5c1f1f38c44e3f3cb37be4b9be5c287bb3db5f8cca95f932
+size 327833648

model-00040-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9369d29ce83807e45840032a764473a8d4e6338497351b97b5a98ad01fbb9fea
+size 327833648

model-00041-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5fb20256de1f464af76e118ba3da9344f464f54164dec7179c3e5517ccb1ba6
+size 327833648

model-00042-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd23a8ca75ce66417c60e2b562476fc85d73d80f61034f1e178fc4594e37f8dc
+size 327833648

model-00043-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11bafe4c66fd086814c5c0427f3699d7a1b1f2749a563853cd54bfcf4c2b1bc2
+size 327833648

model-00044-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:608d8caec230802251d6283475815be2939140e969d559c63f37502da7f28d9b
+size 327833648

model-00045-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:256ee66546a588a35ff7d67ef48e1766989e53f579b6db736fa6ae40098bf7d0
+size 327833648

model-00046-of-00048.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b0b38005a15ee13d81fa77971c50ff541d18db75e20d948eed4484a26689bf3
+size 327833648