GLM-5.2-Legume-v3 — the asymmetric-deep license-babbler (SIQ)

SIQ = SparkInfer Quantization (yes, it's sick): rank-sliced Trellis/EXL3 expert payloads with runtime-dynamic K3/K4 tier mixes, served by the vLLM+SparkInfer ("gilded-gnosis") stack. v3 pushes the machinery somewhere no checkpoint has gone before: a different number of routed experts in every MoE layer.

Sequel to malaiwah/GLM-5.2-Legume ("Legume The First"). Same joke, more surgery: iterative structure search — add layers, drop layers, widen, narrow, re-select — keeping whatever the held-out eval liked. This is a torture-test artifact, not a language model you should use.

What it is

A 20-layer, 724-expert frankenmodel of GLM-5.2 (via willfalco's EXL3-TR3 3.36bpw SIQ quant): old layers 0–15 + 74–77, per-layer expert widths [56, 52, 52, 48, 40, 48, 56, 48, 48, 36, 36, 36, 36, 28, 32, 36, 36] (layers 3–19), experts selected by routed-count tournament on a license corpus (prefix layers), MSE rank (mid layers), each re-encoded at its parent K3/K4 tier at TP1. ~23.2 GiB; boots on one RTX 5090 with the DSA sparse indexer active and fp8_ds_mla KV.

Requires two small patches (included in patches/)

Stock vLLM/GG builds assume one expert count for all MoE layers. v3's heterogeneous widths need:

• patches/deepseek_v2.py — honors n_routed_experts_per_layer (per-layer width override via a config shallow-copy keyed on the layer prefix)
• patches/exl3.py — SIQ loader accepts a per-layer experts_per_layer list (three sites: bitrate-map validation, uniform fallback, width cross-check)

Mount them over the originals (-v patch.py:$SITE/…:ro). Everything else — DSA indexer, MLA, fp8_ds_mla KV, dynamic tiers — is bone stock. That is the point: these ~40 patched lines are the entire distance between the production stack and heterogeneous-width MoE serving.

The surgery log (held-out canonical Apache-2.0, mean logprob / ppl)

round	change	result	verdict
v1	16L/64E, lowest-MSE experts	−13.42 / 670k	baseline
v2	+ routing tournament	−12.47 / 261k	shipped as Legume The First
1b	17L (add old-12), asymmetric widths	−12.00 / 163k	accept
2a	drop old-74, widen early	−13.77 / 954k	reject — tail is sacred
2b	18L (add old-13)	−11.94 / 154k	accept — crossed the uniform-random line
3a	19L (add old-14)	−11.85 / 140k	accept
4	21L (add old-15+16 at once)	−12.12 / 184k	reject — width starvation
4b	20L (add old-15 only)	−11.70 / 121k	accept — FINAL
5	21L (add old-16, balanced trims)	−12.10 / 180k	reject — prefix saturated
6	shift 12 seats tail→early	−11.97 / 158k	reject — tail width also sacred
7	re-select mid layers by routing	−11.74 / 125k	reject
8	extend tail backward (old-73)	−13.16 / 517k	reject — the wound interface is load-bearing

Uniform-random over this vocab ≈ −11.93. v3 predicts held-out license text better than chance — from selection and arrangement alone, zero gradient steps. Perplexity fell 5.5× from v1.

What the search taught us about a lobotomized MoE:

1. The tail adapts to the wound. Layers 74–77 tolerate no surgery — not removal, not narrowing, not even inserting their own natural predecessor (old-73). They function because they've been recruited to decode the prefix's out-of-distribution hidden states.
2. Prefix depth beats width until it saturates (at old-15 here).
3. Quantization quality (MSE) barely predicts routing usefulness — tournament winners split ~26/74 against MSE picks in v2, ~45/55 in the mid-layer re-run.

Serving numbers (RTX 5090, gilded-gnosis r25)

• Heterogeneous widths (28–56 experts/layer) through the SIQ dynamic-tier loader: boots, loads, serves — first checkpoint ever to exercise this
• DSA sparse indexer active; fp8_ds_mla KV; eval config runs eager at 512-ctx (the 31.4 GiB card is a razor fit at this size)
• All tournament/eval artifacts in tournament/; the sealed plan (fingerprint 6a20624f…) ships as franken_plan.json

Reproduction

tools/ contains the complete loop: legume_v3.py (pools / plans / merge / assemble), legume_v3_round.sh (one surgery round), capture + eval scripts, the corpus generator, and the byte-pinned base encoder. Every expert of every candidate layer was encoded once into a 256-expert pool; every round is a pure merge — the whole 12-round search re-encoded nothing.

Credits

• zai-org — GLM-5.2 (MIT)
• willfalco — the parent EXL3-TR3 SIQ quant
• turboderp — exllamav3 / EXL3 Trellis
• brandonmusic — the TR3 calibration encoder toolchain
• b12x / SparkInfer — the gilded-gnosis serving stack
• malaiwah — the surgeries, the tournament, and the name

Disclaimer

The patient is now measurably better than a coin flip at licenses and worse than one at everything else. Do not deploy. Do not fine-tune. Do not ask it to review your LICENSE file — it will simply feel seen.