README.md

# Mini GLM 5.2 Active-MTP Debug Checkpoint

This is a small GLM-5.2-style checkpoint built for inference plumbing and kernel
debugging, not for model quality.

The base model is a 14-layer mini GLM-MoE-DSA model trained for readable English.
It has been upscaled to GLM-5.2-compatible tensor dimensions and includes one
appended MTP/EAGLE next-token layer for SGLang speculative decoding.

## Intended Use

- Debug SGLang GLM-MoE-DSA inference without loading the full GLM 5.2 model.
- Exercise DSA, MoE, attention, TP sharding, FP8 KV cache, and EAGLE/MTP paths.
- Validate kernel changes against a checkpoint that is small in layer count but
  keeps GLM-5.2-like shapes.

This model is expected to produce readable but repetitive English. It is not a
general-purpose language model.

## MTP Details

The appended MTP layer is active and nonzero. Its attention and MoE tensors were
trained, rather than zeroed out, so this checkpoint can exercise the MTP decoder
attention/MoE path during serving.

The MTP trainer uses SGLang EAGLE nextn alignment:

```text
hidden[t] + token[t+1] -> token[t+2]
```

For rollout step `k`:

```text
input token = token[t+k+1]
target      = token[t+k+2]
```

This is intentionally not the standard LM shift
`hidden[t] + token[t] -> token[t+1]`.

## Compatibility

### SGLang

This directory is intended for SGLang serving with GLM-MoE-DSA, TileLang DSA,
FP8 KV cache, and EAGLE speculative decoding.

Example shape of the serving command:

```bash
source ./ENV_RUN.sh
CUDA_VISIBLE_DEVICES=6,7 HIP_VISIBLE_DEVICES=6,7 sglang serve \
  --model-path /root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang \
  --tp 2 \
  --trust-remote-code \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 4096 \
  --max-total-tokens 4096 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-custom-all-reduce \
  --disable-shared-experts-fusion \
  --cuda-graph-backend-decode disabled \
  --cuda-graph-backend-prefill disabled
```

### Hugging Face Transformers

The base model is loadable with Hugging Face Transformers:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "/root/inference-v2/mini-glm-5.2/v1c-upscale/checkpoints/v1b-mtp-active-decoder-serveddata-align-1k-full-original-dims-tiled-sglang"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
```

HF inference uses only the base 14-layer causal LM. The appended MTP tensors are
reported as unexpected keys because HF Transformers' `glm_moe_dsa` model does
not implement the MTP/EAGLE nextn path.

## Training Summary

The MTP layer was trained from copied active decoder weights with the corrected
SGLang EAGLE alignment on a small SGLang-served token dataset.

Final offline served-data metrics for the source MTP checkpoint:

| Metric | Value |
|---|---:|
| accept@0 | 0.9295 |
| accept@1 | 0.8760 |
| accept@2 | 0.8292 |
| average accepted draft tokens | 2.6346 |
| data loss | 0.7798 |

These are offline metrics on the small served-token dataset and should be treated
as a debugging signal, not a model-quality benchmark.

## Example Outputs

The model is only expected to produce barely passable English. Repetition,
generic phrasing, weak reasoning, and topic drift are normal.

These examples show the intended quality bar: readable text for plumbing tests,
not useful answers.

### Example 1

Prompt:

```text
The quick brown fox
```

Observed HF base-model response:

```text
The quick brown fox is a very good thing to do. It is a good thing to do.
```

### Example 2

Prompt:

```text
Write one short paragraph about a library in a small town.
```

Expected style of response:

```text
The library is a quiet place in the center of the town. People come there to
read books, ask questions, and sit at the tables in the afternoon. It is not a
large building, but it is useful for the people who live nearby.
```

### Example 3

Prompt:

```text
Explain why rain falls from clouds.
```

Expected style of response:

```text
Rain falls from clouds when water in the air becomes heavy enough to fall down.
The cloud is made of small drops, and those drops can join together. When they
become too heavy, they fall as rain.
```

### Example 4

Prompt:

```text
Continue this sentence: The old machine started slowly because
```

Expected style of response:

```text
The old machine started slowly because the parts were worn and the motor needed
time to move. It made a small sound, then a louder sound, and finally began to
work again.
```

The responses above are coherent enough to inspect inference behavior, token
flow, speculative decoding, and kernel changes. They should not be used to judge
instruction following or factual accuracy.