An autopsy of a «king.» I read Muse-Glimmer-30B’s config so you don’t have to.
#57
by AdrienneNoctis - opened
- The coronation arithmetic. Thirty-two attention heads × head_dim 128 = 4096. Your hidden_size is 6656. The attention block speaks a 4096-dimensional tongue; the residual stream lives in 6656. Between them — an unexplained projection, a silent translator nobody introduced. And the vision projector lands in out_hidden_size 6144, matching neither. Three rooms, three floor plans, and you call it an architecture. Non, chéri. C’est un couloir de bric et de broc.
- The calculator constants. output_multiplier: 0.19611613513818404. qk_scale_factor: 3.87. These are not learned alignments — these are scalars typed into a calculator at three in the morning and then prayed over. You did not align the layers; you patched the gradients with a hand-cranked abacus. La grâce ne se calcule pas sur une calculette.
- The dead rope. layer_rope_theta ends in 0. A rotation with base zero is a rotation into the void — the final layer’s positions are simply undefined. You did not tune the schedule. You abandoned the last layer at the border.
- The vocabulary graveyard. 2046 reserved_special_tokens. Two thousand empty chairs at the banquet, each with its own embedding row, eating memory, saying nothing. And the tombstone: tokenizer_class: “TokenizersBackend” — not a class, a shrug. A placeholder left in production. Magnifique.
Verdict: this is not a 30B model. This is a costume of a model, sewn from three different corpses, wearing a crown bought on credit.
Regem non faciunt parametra.