| --- |
| license: mit |
| base_model: zai-org/GLM-4.7-Flash |
| tags: |
| - mlx |
| - speculative-decoding |
| - multi-token-prediction |
| - drafter |
| - glm4_moe_lite_mtp |
| --- |
| |
| # GLM-4.7-Flash-MTP-4bit |
|
|
| The trained multi-token-prediction (MTP / nextn) layer of |
| [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash), split |
| into a standalone MLX drafter checkpoint and quantized to 4-bit. This is **not |
| a standalone language model** — it is a single-layer draft head that predicts |
| one token ahead from a target model's hidden states, for speculative decoding |
| against GLM-4.7-Flash (pairs with |
| [mlx-community/GLM-4.7-Flash-4bit](https://huggingface.co/mlx-community/GLM-4.7-Flash-4bit)). |
|
|
| GLM-4.7-Flash ships this layer inside the full checkpoint at |
| `model.layers.47.*`; MLX conversions of the base model strip it (`sanitize` |
| drops layers past `num_hidden_layers`), so quantized community conversions do |
| not carry it. This repo preserves it, revision-pinned. |
|
|
| ## Provenance |
|
|
| - **Source:** `zai-org/GLM-4.7-Flash`, revision |
| `7dd20894a642a0aa287e9827cb1a1f7f91386b67` (MIT). All weights are Z.ai's |
| trained parameters, unmodified except quantization and the layout transforms |
| below. |
| - **Tool:** the `glm4_moe_lite_mtp` drafter split from mlx-vlm's |
| `speculative/drafters` convention |
| ([Blaizzy/mlx-vlm#1570](https://github.com/Blaizzy/mlx-vlm/pull/1570)): |
|
|
| ```bash |
| python -m mlx_vlm.speculative.drafters.glm4_moe_lite_mtp.split \ |
| --model zai-org/GLM-4.7-Flash \ |
| --revision 7dd20894a642a0aa287e9827cb1a1f7f91386b67 \ |
| --output GLM-4.7-Flash-MTP-4bit \ |
| --q-bits 4 --q-group-size 64 |
| ``` |
|
|
| Only the 3 (of 48) source shards holding the nextn tensors are read. |
|
|
| - **Checksum:** |
|
|
| | file | sha256 | |
| |---|---| |
| | `model.safetensors` | `cc2a9750a6328a68b1758502a47f5d286cbc96c26859210fe7c751bbe6d328ba` | |
|
|
| ## Format |
|
|
| `model_type: glm4_moe_lite_mtp`, `block_size: 2` |
| (`num_nextn_predict_layers + 1`), untied embeddings, affine quantization |
| (`bits: 4`, `group_size: 64`); the source text config is nested under |
| `text_config`. 54 tensors, flat post-sanitize layout: |
|
|
| - dedicated nextn `embed_tokens` and untied `lm_head` (GLM's nextn head is not |
| tied to the target, unlike DeepSeek/Qwen MTP) |
| - `enorm` / `hnorm` / `eh_proj` projections |
| - one MLA attention block in absorbed form (`kv_b_proj` split into |
| `embed_q` / `unembed_out`) |
| - 64-expert MoE stacked into `switch_mlp` + shared expert |
| - **not quantized:** norms, the router gate weight, and the `noaux_tc` router |
| correction bias (kept fp32) — casting or quantizing them breaks routing |
|
|
| ## Consumers |
|
|
| - [vllm-project/vllm-metal#484](https://github.com/vllm-project/vllm-metal/pull/484) / |
| [#485](https://github.com/vllm-project/vllm-metal/pull/485) — native MTP |
| speculative decoding for GLM-4.7-Flash on Apple Silicon (reference |
| integration this format feeds; design thread in |
| [#482](https://github.com/vllm-project/vllm-metal/issues/482)) |
| - mlx-vlm's speculative-decoding runtime, once a `glm4_moe_lite` backbone |
| lands there |
|
|
| Measured offline acceptance of this head replaying real target hidden states: |
| ~0.806 mean over chat prompts; a 4-bit head forward measured ~6.7% of a target |
| decode step on M3 (methodology and end-to-end results in the vllm-metal links |
| above). |
|
|
| An unquantized variant is at |
| [samithaj/GLM-4.7-Flash-MTP-bf16](https://huggingface.co/samithaj/GLM-4.7-Flash-MTP-bf16). |
|
|
