Publish MXFP8 DFlash2 checkpoint

.gitattributes

@@ -1,35 +1,2 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/dflash2-figure.png filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,124 @@
+---
+license: cc-by-nc-nd-4.0
+library_name: vllm
+pipeline_tag: text-generation
+base_model:
+  - incoai/GLM-5.3-Flash-DFlash2
+  - local-inference-lab/GLM-5.3-Flash-NVFP4
+inference: false
+tags:
+  - dflash
+  - dflash2
+  - speculative-decoding
+  - draft-model
+  - modelopt
+  - mxfp8
+  - vllm
+---
+
+# GLM-5.3-Flash-DFlash2
+
+This repository contains an MXFP8-quantized DFlash 2 draft model for
+[`local-inference-lab/GLM-5.3-Flash-NVFP4`](https://huggingface.co/local-inference-lab/GLM-5.3-Flash-NVFP4).
+It is not a standalone language model. A compatible speculative-decoding
+server loads it beside the target model and verifies every drafted token
+against the target.
+
+The source checkpoint is
+[`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2)
+at immutable revision `dc77ff1c99eeb2df044ee3d4f0094eb033fee410`.
+
+## Format
+
+- Linear weights: `float8_e4m3fn`
+- Scale values: biased E8M0 exponents stored as `uint8`
+- Quantization block: 1×32 values
+- Scale layout: row-major and unswizzled
+- Excluded module: `lm_head`
+- Draft KV cache quantization: not encoded in the checkpoint
+
+`conversion_manifest.json` records the immutable source revision, source and
+output checksums, tensor coverage, aggregate quantization error, and
+per-weight validation statistics.
+
+## Validation status
+
+Status: **qualified** for checkpoint structure, exact format reproduction,
+loading, and smoke inference under the following conditions:
+
+- Target: `local-inference-lab/GLM-5.3-Flash-NVFP4` revision
+  `520de24eabf507659eaef7c70f14fd584527facc`
+- Runtime:
+  `voipmonitor/vllm@sha256:3a7cbc5833ed3d5390c5de2ec3c5a2737761a175123262cfdfd7b38638e1d5e6`
+- Hardware: four NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPUs
+- Parallelism: tensor parallel size 4 and decode-context parallel size 1
+- Target attention, MoE, linear, and tensor-parallel all-reduce: B12X
+- DFlash attention: FlashAttention 2
+- DFlash linear: B12X MXFP8
+- DFlash proposal length: seven tokens
+- DFlash KV cache: `auto` (BF16)
+- CUDA graph mode: `FULL` requested; target and DFlash2 decode are captured,
+  while target GDN prefill remains eager
+
+The runtime detected ModelOpt MXFP8, selected `B12xMxfp8LinearKernel` for
+draft GEMMs and the fused DFlash context K/V projection, and loaded 1.20 GB of
+draft weights. With seven draft tokens, the qualified runtime measured a
+2.1157-second median time to first token for a 32,320-token prompt and
+185.5 output tokens per second at concurrency one. Speculative throughput
+depends on prompt content and acceptance length.
+
+The checkpoint is unsupported in vLLM builds that do not contain the DFlash 2
+and ModelOpt MXFP8 integration used by the qualified runtime.
+
+## Serving
+
+```bash
+docker run --rm \
+  --gpus '"device=0,1,2,3"' \
+  --network host \
+  --ipc host \
+  --shm-size 32g \
+  -e MODEL=local-inference-lab/GLM-5.3-Flash-NVFP4 \
+  -e SERVED_MODEL_NAME=GLM-5.3-Flash-NVFP4 \
+  -e PORT=8000 \
+  -e TP=4 \
+  -e DCP=1 \
+  -e MAX_NUM_SEQS=16 \
+  -e MAX_MODEL_LEN=262144 \
+  -e MAX_NUM_BATCHED_TOKENS=4096 \
+  -e SPECULATOR=dflash \
+  -e NUM_SPECULATIVE_TOKENS=7 \
+  -e DFLASH_MODEL=local-inference-lab/GLM-5.3-Flash-DFlash2 \
+  -e DFLASH_MODEL_REVISION= \
+  -e DFLASH_KV_CACHE_DTYPE=auto \
+  -e DFLASH_ATTENTION_BACKEND=FLASH_ATTN \
+  -e ATTENTION_BACKEND=B12X \
+  -e MOE_BACKEND=b12x \
+  -e LINEAR_BACKEND=b12x \
+  -e B12X_PCIE_ALLREDUCE=1 \
+  -e CUDAGRAPH_MODE=FULL \
+  -e VLLM_B12X_MOE_FP4_FORCE_A16=0 \
+  voipmonitor/vllm:jovian-judgement-community-dflash2-20260830-r7
+```
+
+An empty `DFLASH_MODEL_REVISION` makes the launcher resolve the repository's
+`main` branch. For reproducible deployments, replace the empty value with an
+immutable Hugging Face commit hash. The OpenAI-compatible endpoint is
+available at `http://127.0.0.1:8000/v1`.
+
+## License and attribution
+
+The source DFlash 2 model is distributed under
+[CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/).
+See the [source model card](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2)
+for its use restrictions and attribution information.
+
+```bibtex
+@misc{inco2026dflash2,
+  title  = {{DFlash 2: Keep Drafting Parallel}},
+  author = {{Inco AI}},
+  year   = {2026},
+  month  = {August},
+  url    = {https://inco.ai/blog/dflash2/}
+}
+```

config.json

@@ -0,0 +1,75 @@
+{
+  "architectures": [
+    "DFlash2DraftModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "dflash_config": {
+    "block_size": 8,
+    "conv_group_size": 16,
+    "conv_kernel_size": 2,
+    "mask_token_id": 154856,
+    "selector_rank": 256,
+    "selector_top_k": 16,
+    "target_layer_ids": [
+      5,
+      14,
+      24,
+      33,
+      42
+    ]
+  },
+  "dtype": "bfloat16",
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "is_causal": false,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention"
+  ],
+  "max_position_embeddings": 1048576,
+  "max_window_layers": 5,
+  "model_type": "qwen3",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 5,
+  "num_key_value_heads": 8,
+  "num_target_layers": 45,
+  "pad_token_id": 154820,
+  "quantization_config": {
+    "producer": {
+      "name": "glm53-dflash2-mxfp8-converter",
+      "version": 1
+    },
+    "quant_method": "modelopt",
+    "quantization": {
+      "exclude_modules": [
+        "lm_head"
+      ],
+      "kv_cache_quant_algo": null,
+      "quant_algo": "MXFP8"
+    }
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "sliding_window": 2048,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.7.0",
+  "use_cache": false,
+  "use_sliding_window": true,
+  "vocab_size": 154880
+}

conversion_manifest.json

@@ -0,0 +1,651 @@
+{
+  "artifact": "GLM-5.3-Flash-DFlash2 MXFP8 draft checkpoint",
+  "converter_sha256": "f4e28a1c47e06a1748d61d2404ffb6f8258a444b3bff2305718a7e8e5bf0593c",
+  "coverage": {
+    "preserved_tensors": 34,
+    "quantized_linear_weights": 47,
+    "quantized_values": 1091567616
+  },
+  "error": {
+    "maximum_absolute_error": 0.125,
+    "relative_rmse": 0.026583480809130377,
+    "root_mean_squared_error": 0.0028997886924729786
+  },
+  "format": {
+    "block_shape": [
+      1,
+      32
+    ],
+    "scale": "uint8 biased E8M0 exponent",
+    "scale_layout": "row-major, unswizzled",
+    "weight": "float8_e4m3fn"
+  },
+  "output_weights_sha256": "c033e03d47c7d5608596c8fc4e9336a1fe086eb781c08fe031be2bdea1614e58",
+  "output_files_sha256": {
+    ".gitattributes": "70e0ed421d653865ad7ace84728e32ded01ec0a860e989a5ed736fe11b42214f",
+    "README.md": "5a74d5053c29116cb688f40e75aed4b9c21d28a2ad462e963667c3dae522b593",
+    "config.json": "4ed95da7d0abecd3eb341e96464d381b474d24d8924bd5efba2dfa24b92690bf",
+    "hf_quant_config.json": "b7f04ac7e85e73544f9411dbda70fd75ac3b116e6e239ebff5bc85b15caefd42",
+    "model.safetensors": "c033e03d47c7d5608596c8fc4e9336a1fe086eb781c08fe031be2bdea1614e58"
+  },
+  "source": {
+    "config_sha256": "c4aeac0101196a6e26705b34c45230bcd0c7c68ee2d2d1efdb242087f3712573",
+    "model": "incoai/GLM-5.3-Flash-DFlash2",
+    "revision": "dc77ff1c99eeb2df044ee3d4f0094eb033fee410",
+    "weights_sha256": "b33c03475ba7322cf398828f2d8d1be376df30dc05c6b40c28c8ea8da23e410b"
+  },
+  "status": "implemented",
+  "tensors": [
+    {
+      "maximum_absolute_error": 0.015625,
+      "relative_rmse": 0.02662914012776337,
+      "scale": "candidate_selector.hidden_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        256,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "candidate_selector.hidden_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0546875,
+      "relative_rmse": 0.026567000680209157,
+      "scale": "fc.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        20480
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "fc.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026598510847792713,
+      "scale": "layers.0.attention_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.attention_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.078125,
+      "relative_rmse": 0.02659119810765467,
+      "scale": "layers.0.mlp.down_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        12288
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.mlp.down_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.02661559519636154,
+      "scale": "layers.0.mlp.gate_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.mlp.gate_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.02661984011036928,
+      "scale": "layers.0.mlp.up_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.mlp.up_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.02654063072293776,
+      "scale": "layers.0.mlp_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.mlp_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026592116125447415,
+      "scale": "layers.0.self_attn.k_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.self_attn.k_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0390625,
+      "relative_rmse": 0.02656823404746311,
+      "scale": "layers.0.self_attn.o_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.self_attn.o_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026567040624108283,
+      "scale": "layers.0.self_attn.q_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.self_attn.q_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.015625,
+      "relative_rmse": 0.026525638628768222,
+      "scale": "layers.0.self_attn.v_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.0.self_attn.v_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.125,
+      "relative_rmse": 0.026596001906054587,
+      "scale": "layers.1.attention_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.attention_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.0265787755625202,
+      "scale": "layers.1.mlp.down_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        12288
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.mlp.down_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026587237640933212,
+      "scale": "layers.1.mlp.gate_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.mlp.gate_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.109375,
+      "relative_rmse": 0.02659690101329357,
+      "scale": "layers.1.mlp.up_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.mlp.up_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.078125,
+      "relative_rmse": 0.02656689675876576,
+      "scale": "layers.1.mlp_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.mlp_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0546875,
+      "relative_rmse": 0.026569303009495438,
+      "scale": "layers.1.self_attn.k_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.self_attn.k_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026560798124397773,
+      "scale": "layers.1.self_attn.o_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.self_attn.o_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026568386185886816,
+      "scale": "layers.1.self_attn.q_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.self_attn.q_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.0265892482789613,
+      "scale": "layers.1.self_attn.v_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.1.self_attn.v_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.02657074981178284,
+      "scale": "layers.2.attention_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.attention_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.109375,
+      "relative_rmse": 0.026572407862604005,
+      "scale": "layers.2.mlp.down_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        12288
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.mlp.down_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026568281345928493,
+      "scale": "layers.2.mlp.gate_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.mlp.gate_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026585907486245345,
+      "scale": "layers.2.mlp.up_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.mlp.up_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.109375,
+      "relative_rmse": 0.026572698101393868,
+      "scale": "layers.2.mlp_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.mlp_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0546875,
+      "relative_rmse": 0.026577556384314666,
+      "scale": "layers.2.self_attn.k_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.self_attn.k_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026560576726262823,
+      "scale": "layers.2.self_attn.o_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.self_attn.o_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0546875,
+      "relative_rmse": 0.026586736069846087,
+      "scale": "layers.2.self_attn.q_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.self_attn.q_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026586952065358875,
+      "scale": "layers.2.self_attn.v_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.2.self_attn.v_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.125,
+      "relative_rmse": 0.026572177207037927,
+      "scale": "layers.3.attention_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.attention_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.109375,
+      "relative_rmse": 0.02656499109738341,
+      "scale": "layers.3.mlp.down_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        12288
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.mlp.down_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.125,
+      "relative_rmse": 0.026570417196929284,
+      "scale": "layers.3.mlp.gate_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.mlp.gate_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.02658615684586314,
+      "scale": "layers.3.mlp.up_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.mlp.up_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026566849904042712,
+      "scale": "layers.3.mlp_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.mlp_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0546875,
+      "relative_rmse": 0.02657702447659922,
+      "scale": "layers.3.self_attn.k_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.self_attn.k_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026568267578320686,
+      "scale": "layers.3.self_attn.o_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.self_attn.o_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0546875,
+      "relative_rmse": 0.026577504789784222,
+      "scale": "layers.3.self_attn.q_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.self_attn.q_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026590205296363092,
+      "scale": "layers.3.self_attn.v_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.3.self_attn.v_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.125,
+      "relative_rmse": 0.026593574382866452,
+      "scale": "layers.4.attention_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.attention_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.109375,
+      "relative_rmse": 0.02661912299049318,
+      "scale": "layers.4.mlp.down_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        12288
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.mlp.down_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.02657367966586845,
+      "scale": "layers.4.mlp.gate_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.mlp.gate_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026602028239691457,
+      "scale": "layers.4.mlp.up_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        12288,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.mlp.up_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.02658053894333081,
+      "scale": "layers.4.mlp_conv.kernel_projection.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.mlp_conv.kernel_projection.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026593862409928445,
+      "scale": "layers.4.self_attn.k_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.self_attn.k_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.02655268219685373,
+      "scale": "layers.4.self_attn.o_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.self_attn.o_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.0625,
+      "relative_rmse": 0.026596374233145523,
+      "scale": "layers.4.self_attn.q_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        4096,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.self_attn.q_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    },
+    {
+      "maximum_absolute_error": 0.03125,
+      "relative_rmse": 0.026648845046121847,
+      "scale": "layers.4.self_attn.v_proj.weight_scale",
+      "scale_dtype": "uint8",
+      "shape": [
+        1024,
+        4096
+      ],
+      "source_dtype": "bfloat16",
+      "weight": "layers.4.self_attn.v_proj.weight",
+      "weight_dtype": "float8_e4m3fn"
+    }
+  ]
+}

hf_quant_config.json

@@ -0,0 +1,13 @@
+{
+  "producer": {
+    "name": "glm53-dflash2-mxfp8-converter",
+    "version": 1
+  },
+  "quantization": {
+    "exclude_modules": [
+      "lm_head"
+    ],
+    "kv_cache_quant_algo": null,
+    "quant_algo": "MXFP8"
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c033e03d47c7d5608596c8fc4e9336a1fe086eb781c08fe031be2bdea1614e58
+size 1284719240