File size: 5,000 Bytes
e980dee | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 | # Mini GLM 5.2 Active-MTP Debug Checkpoint
This is a small GLM-5.2-style checkpoint built for inference plumbing and kernel
debugging, not for model quality.
The base model is a 14-layer mini GLM-MoE-DSA model trained for readable English.
It has been upscaled to GLM-5.2-compatible tensor dimensions and includes one
appended MTP/EAGLE next-token layer for SGLang speculative decoding.
## Intended Use
- Debug SGLang GLM-MoE-DSA inference without loading the full GLM 5.2 model.
- Exercise DSA, MoE, attention, TP sharding, FP8 KV cache, and EAGLE/MTP paths.
- Validate kernel changes against a checkpoint that is small in layer count but
keeps GLM-5.2-like shapes.
This model is expected to produce readable but repetitive English. It is not a
general-purpose language model.
## MTP Details
The appended MTP layer is active and nonzero. Its attention and MoE tensors were
trained, rather than zeroed out, so this checkpoint can exercise the MTP decoder
attention/MoE path during serving.
The MTP trainer uses SGLang EAGLE nextn alignment:
```text
hidden[t] + token[t+1] -> token[t+2]
```
For rollout step `k`:
```text
input token = token[t+k+1]
target = token[t+k+2]
```
This is intentionally not the standard LM shift
`hidden[t] + token[t] -> token[t+1]`.
## Compatibility
### SGLang
This directory is intended for SGLang serving with GLM-MoE-DSA, TileLang DSA,
FP8 KV cache, and EAGLE speculative decoding.
Example shape of the serving command:
```bash
source ./ENV_RUN.sh
CUDA_VISIBLE_DEVICES=6,7 HIP_VISIBLE_DEVICES=6,7 sglang serve \
--model-path /root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang \
--tp 2 \
--trust-remote-code \
--dsa-prefill-backend tilelang \
--dsa-decode-backend tilelang \
--kv-cache-dtype fp8_e4m3 \
--chunked-prefill-size 4096 \
--max-total-tokens 4096 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--disable-custom-all-reduce \
--disable-shared-experts-fusion \
--cuda-graph-backend-decode disabled \
--cuda-graph-backend-prefill disabled
```
### Hugging Face Transformers
The base model is loadable with Hugging Face Transformers:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "/root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
```
HF inference uses only the base 14-layer causal LM. The appended MTP tensors are
reported as unexpected keys because HF Transformers' `glm_moe_dsa` model does
not implement the MTP/EAGLE nextn path.
## Training Summary
The MTP layer was trained from copied active decoder weights with the corrected
SGLang EAGLE alignment on a small SGLang-served token dataset.
Final offline served-data metrics for the source MTP checkpoint:
| Metric | Value |
|---|---:|
| accept@0 | 0.9295 |
| accept@1 | 0.8760 |
| accept@2 | 0.8292 |
| average accepted draft tokens | 2.6346 |
| data loss | 0.7798 |
These are offline metrics on the small served-token dataset and should be treated
as a debugging signal, not a model-quality benchmark.
## Example Outputs
The model is only expected to produce barely passable English. Repetition,
generic phrasing, weak reasoning, and topic drift are normal.
These examples show the intended quality bar: readable text for plumbing tests,
not useful answers.
### Example 1
Prompt:
```text
The quick brown fox
```
Observed HF base-model response:
```text
The quick brown fox is a very good thing to do. It is a good thing to do.
```
### Example 2
Prompt:
```text
Write one short paragraph about a library in a small town.
```
Expected style of response:
```text
The library is a quiet place in the center of the town. People come there to
read books, ask questions, and sit at the tables in the afternoon. It is not a
large building, but it is useful for the people who live nearby.
```
### Example 3
Prompt:
```text
Explain why rain falls from clouds.
```
Expected style of response:
```text
Rain falls from clouds when water in the air becomes heavy enough to fall down.
The cloud is made of small drops, and those drops can join together. When they
become too heavy, they fall as rain.
```
### Example 4
Prompt:
```text
Continue this sentence: The old machine started slowly because
```
Expected style of response:
```text
The old machine started slowly because the parts were worn and the motor needed
time to move. It made a small sound, then a louder sound, and finally began to
work again.
```
The responses above are coherent enough to inspect inference behavior, token
flow, speculative decoding, and kernel changes. They should not be used to judge
instruction following or factual accuracy.
|