README.md

---
license: other
base_model: zai-org/GLM-5.2
tags:
- glm
- moe
- dsa
- trimmed
- testing
---

# GLM-5.2-7layer (layer-trimmed, for training/serving infra testing)

A **layer-trimmed** copy of [`zai-org/GLM-5.2`](https://huggingface.co/zai-org/GLM-5.2),
reduced from **78 layers (+1 MTP) to 7 layers (+1 MTP)** so every structurally
distinct layer type can be exercised on a small number of GPUs during early
training/serving infra development (LoRA, parallelism, MTP, etc.).

**This is NOT a usable language model** — most layers are removed, so generations
are gibberish. It exists purely to let infra code load, shard, attach LoRA to,
and run a forward/backward pass over *every distinct layer* of GLM-5.2 at a
fraction of the size (~100 GB bf16 vs ~1.45 TB).

## Why 7 layers (and why a contiguous prefix)

GLM-5.2 differs from GLM-5.1 in its attention: it uses a **mixed DSA pattern**.
Only some layers own a sparse-attention indexer (`indexer_types = "full"`); the
rest **reuse** a nearby full layer's top-k index (`"shared"`), governed by
`index_topk_freq = 4` / `index_skip_topk_offset = 3`. Crucially, whether a layer
owns an indexer is decided by **layer-id arithmetic**, so the kept layers must
keep their **original, contiguous ids** — renumbering would misalign the indexer
weights with that arithmetic. Layers 0–6 are therefore kept verbatim (identity
ids); only the MTP layer (78) is renumbered to 7.

## What was kept (verbatim bf16 weights, original ids 0–6)

| idx | source | MLP (`mlp_layer_types`) | Attention (`indexer_types`) | Why kept |
|---|---|---|---|---|
| 0,1,2 | 0,1,2 | **dense** | **full** (own DSA indexer) | the only dense layers (`first_k_dense_replace=3`) + indexer-owning |
| 3,4,5 | 3,4,5 | **sparse** (MoE: 256 routed + 1 shared) | **shared** (reuses a full layer's index — no own indexer weights) | the homogeneous MoE-with-shared-index block |
| 6 | 6 | **sparse** (MoE) | **full** (own DSA indexer) | a MoE layer that *owns* an indexer (the other distinct combo) |
| 7 | 78 | **sparse** (MoE) | MTP/nextn | the multi-token-prediction layer (`enorm`/`hnorm`/`eh_proj`/`shared_head`) |
| — | — | — | — | top-level `embed_tokens`, final `norm`, `lm_head` |

This covers all four distinct combinations present in GLM-5.2:
**dense+full**, **MoE+shared**, **MoE+full**, and **MTP**.

## What was removed

- Original layers **7–77** (71 MoE layers) — duplicates of the kept MoE+shared
  (3–5) and MoE+full (6) types. Nothing else is changed.

## What changed in `config.json`

- `num_hidden_layers: 78 → 7`.
- Per-layer pattern lists trimmed to the kept layers (first 7 entries), so HF /
  transformers builds the correct 7-layer model:
  - `mlp_layer_types  → ["dense","dense","dense","sparse","sparse","sparse","sparse"]`
  - `indexer_types    → ["full","full","full","shared","shared","shared","full"]`
- Everything else is identical to the base (`first_k_dense_replace=3`,
  `index_topk_freq=4`, `index_skip_topk_offset=3`, `num_nextn_predict_layers=1`,
  expert counts, all dims), so the kept layers are bit-for-bit the real
  GLM-5.2 — equivalent to the full model's first 7 layers + its MTP layer.

## Coverage checklist (all distinct layer types present ≥ once)

- [x] Dense MLP layer (0–2)
- [x] MoE layer — routed + shared experts, gate (3–6)
- [x] DSA indexer-owning layer ("full": 0,1,2,6)
- [x] Index-reusing layer ("shared": 3,4,5)
- [x] MTP / nextn layer (7)
- [x] embed_tokens / final norm / lm_head

## Provenance

Produced by selecting the relevant shards of `zai-org/GLM-5.2`, copying the kept
tensors verbatim (bf16) with original layer ids preserved (MTP renumbered 78→7),
trimming the per-layer config lists, and rewriting the safetensors index +
`num_hidden_layers`. Verified to load and run a forward pass (base + a LoRA
adapter spanning all kept layers) in SGLang (`main`).