| --- |
| license: mit |
| base_model: zai-org/GLM-4.7-Flash |
| tags: |
| - mlx |
| - speculative-decoding |
| - multi-token-prediction |
| - drafter |
| - glm4_moe_lite_mtp |
| --- |
| |
| # GLM-4.7-Flash-MTP-bf16 |
|
|
| The trained multi-token-prediction (MTP / nextn) layer of |
| [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash), split |
| into a standalone MLX drafter checkpoint. This is **not a standalone language |
| model** — it is a single-layer draft head that predicts one token ahead from a |
| target model's hidden states, for speculative decoding against GLM-4.7-Flash. |
|
|
| GLM-4.7-Flash ships this layer inside the full checkpoint at |
| `model.layers.47.*`; MLX conversions of the base model strip it (`sanitize` |
| drops layers past `num_hidden_layers`), so quantized community conversions do |
| not carry it. This repo preserves it in bf16, revision-pinned. |
|
|
| ## Provenance |
|
|
| - **Source:** `zai-org/GLM-4.7-Flash`, revision |
| `7dd20894a642a0aa287e9827cb1a1f7f91386b67` (MIT). All weights are Z.ai's |
| trained parameters, unmodified except for the layout transforms below. |
| - **Tool:** the `glm4_moe_lite_mtp` drafter split from mlx-vlm's |
| `speculative/drafters` convention |
| ([Blaizzy/mlx-vlm#1570](https://github.com/Blaizzy/mlx-vlm/pull/1570)): |
|
|
| ```bash |
| python -m mlx_vlm.speculative.drafters.glm4_moe_lite_mtp.split \ |
| --model zai-org/GLM-4.7-Flash \ |
| --revision 7dd20894a642a0aa287e9827cb1a1f7f91386b67 \ |
| --output GLM-4.7-Flash-MTP-bf16 |
| ``` |
|
|
| Only the 3 (of 48) source shards holding the nextn tensors are read. |
|
|
| - **Checksum:** |
|
|
| | file | sha256 | |
| |---|---| |
| | `model.safetensors` | `bdb948cbec1810d89910cc99dfbc85864acde45241337702b555e2a721254e7b` | |
|
|
| ## Format |
|
|
| `model_type: glm4_moe_lite_mtp`, `block_size: 2` |
| (`num_nextn_predict_layers + 1`), untied embeddings; the source text config is |
| nested under `text_config`. 24 tensors, flat post-sanitize layout: |
|
|
| - dedicated nextn `embed_tokens` and untied `lm_head` (GLM's nextn head is not |
| tied to the target, unlike DeepSeek/Qwen MTP) |
| - `enorm` / `hnorm` / `eh_proj` projections |
| - one MLA attention block in absorbed form (`kv_b_proj` split into |
| `embed_q` / `unembed_out`) |
| - 64-expert MoE stacked into `switch_mlp` + shared expert; the `noaux_tc` |
| router correction bias is kept fp32 and the router gate full precision |
| (casting them breaks routing) |
|
|
| ## Consumers |
|
|
| - [vllm-project/vllm-metal#484](https://github.com/vllm-project/vllm-metal/pull/484) / |
| [#485](https://github.com/vllm-project/vllm-metal/pull/485) — native MTP |
| speculative decoding for GLM-4.7-Flash on Apple Silicon (reference |
| integration this format feeds; design thread in |
| [#482](https://github.com/vllm-project/vllm-metal/issues/482)) |
| - mlx-vlm's speculative-decoding runtime, once a `glm4_moe_lite` backbone |
| lands there |
|
|
| Measured offline acceptance of this head replaying real target hidden states: |
| ~0.806 mean over chat prompts (methodology and end-to-end results in the |
| vllm-metal links above). |
|
|
| A 4-bit variant is at |
| [samithaj/GLM-4.7-Flash-MTP-4bit](https://huggingface.co/samithaj/GLM-4.7-Flash-MTP-4bit). |
|
|
