A Gadsby Mechanistic Interpretation

#1
by juiceb0xc0de - opened

Hey, I came across your model and thought the idea was actually pretty cool. I created a neural network atlas on it and did a bit of mechanistic interpretation work with it. I shared my write up on the model here for you. I thought you might be interested in seeing the results. If you find anything helpful or surprising in the write up let me know I love to hear any feedback on the atlas interpretations I can get! If you ever want to build an atlas of your own on check out my atlas building space Cartographics. A like on the space goes a long way. I hope you find something that interests you here!

juiceb0xc0de_stamp_3407

Gadsby-4B: a self-distill of constrained outputs through a general prompt census

Model Pulse

I ran the Gadsby-4B and parent Gemma-3-4B-IT models through the Cartographics HF space (a space that generates the model atlases referenced throughout this article, on demand). The developer describes Gadsby-4B as a two-step experiment on gemma-3-4b-it: first a constrained-decoding run that banned every token bigram appearing in simple wikipedia, which on its own made the model talk in chat-speak and develop a personality, then a rejection-sampling fine-tune on its own coherent constrained outputs. The chat voice never came from chat-format training data. This census asks what that self-distillation did to the usage-sorting machinery the model inherited from its parent.

Selective MLP coordinates fall from 8,762 to 2,611, a 70.19% reduction, while mean usage-probe accuracy moves from 78.48% to 72.31%. [data] Most of that drop is coordinates weakening rather than going quiet: 7,491 of the parent's 8,762 selective entries are still broadly label-associated in Gadsby, and only 1 goes inert. [data] I tested whether the persona labels lost selective entries under the exported q/AUROC threshold while task labels kept more of theirs: roleplay falls 95.19% and tool_use falls 38.25%, the two extremes of the redistribution. [data] Mean probe accuracy is 6.17 points lower on average and 17.89 points lower at layer 11, the largest single-layer gap. [data]

Scope. This census measures text-input activations for the same 8,965 prompts across 17 usage labels in both atlases. The parent capture applied the Gemma chat template; Gadsby-4B's repository ships no chat template at all, so its capture ran the raw prompts. The capture metadata records 9 overhead tokens for the parent's template, and every parent prompt runs exactly 8 tokens longer than its Gadsby counterpart. [data] The one-token difference is most likely the beginning-of-sequence token, which the raw capture carries too. [inferred] Every paired number below carries both differences at once, the self-distilled weights and the missing template. Unless stated otherwise, results use the MLP intermediate stream at mean_all_real, averaging all real input tokens. The usage-label probe is a held-out multinomial classifier over principal components with a shuffled-label control; the majority-class baseline is 6.58%.

Evidence tags distinguish analytical claims: [data] is a direct measurement or calculation from the pinned atlas databases or capture metadata, [inferred] is my interpretation under stated assumptions, [data, inferred] is both, and [guess] is a mechanism I haven't tested.

Selective MLP coordinates fall 70.19%, and the middle third loses the most

Selective entries fall from 2.52% to 0.75% of the 348,160 MLP layer-coordinate entries, while mean probe accuracy changes by −6.17 percentage points. [data] The atlas calls a coordinate selective when its category ANOVA has Benjamini-Hochberg-adjusted q < 0.01 and exactly one label reaches |AUROC - 0.5| ≥ 0.25; negative associations count too. Mixed means at least two labels reach that effect threshold, and broad means the ANOVA still passes but no single label reaches it.

Layers, zero-based Selective entries: parent to Gadsby Probe accuracy: parent to Gadsby Shuffled-label control: parent to Gadsby
0 to 11 1,248 to 496 74.58% to 69.75% 6.13% to 6.21%
12 to 23 3,322 to 833 80.42% to 74.64% 5.95% to 6.18%
24 to 33 4,192 to 1,282 80.82% to 72.59% 6.08% to 6.15%
All 34 layers 8,762 to 2,611 78.48% to 72.31% 6.05% to 6.18%

The middle third loses 74.92% of its selective entries, and probe accuracy is lower at 33 of 34 layers. [data] Paired per-prompt probe errors are absent from the exports, so the −6.17-point mean gap cannot be subjected to a paired significance test here.

Falling out of the selective class mostly means weakening, not going quiet. Tracking each of the parent's 8,762 selective MLP entries to the same coordinate in Gadsby: [data]

Gadsby classification of the parent's selective MLP entries Count
broad 7,491
selective 1,234
mixed 36
inert 1

Gadsby's selective set is partly new as well: 1,377 of its 2,611 entries were not selective in the parent. Of the 1,234 coordinates selective in both, 1,182 keep the same leading label, 95.8%. [data] The 70.19% figure counts coordinates crossing the selectivity threshold, not 70.19% of the MLP shutting off. [data, inferred]

Roleplay and humor lose the most selective entries; technical labels end up holding the most

Every one of the 17 labels loses selective MLP entries, and the loss is far from even. [data] roleplay falls from 936 to 45, humor from 739 to 80, ml_ai from 913 to 155, planning from 122 to 7. The two least-hit labels, tool_use (779 to 481) and core_technical (847 to 478), shed the smallest shares of their populations, so their share of what remains roughly doubles.

Strongest label association Parent count Gadsby count Parent share Gadsby share
introspection 1,575 464 17.98% 17.77%
creative_writing 1,223 369 13.96% 14.13%
roleplay 936 45 10.68% 1.72%
ml_ai 913 155 10.42% 5.94%
core_technical 847 478 9.67% 18.31%
tool_use 779 481 8.89% 18.42%
humor 739 80 8.43% 3.06%

tool_use gains 9.53 points of share on a 38.25% count decline and core_technical gains 8.64 points on a 43.57% decline, because both fall slower than the population's 70.19%. [data] roleplay loses 8.96 points of share. introspection moves 0.20 points of share on a 70.54% count decline, and creative_writing gains 0.17 points on a 69.83% decline; both move with the population rather than against it. [data] Positive and negative selective associations stay balanced throughout: 4,397 / 4,365 in the parent and 1,319 / 1,292 in Gadsby. [data]

Across all captured components, 69,428 selective and 17,104 mixed entries in the parent fall to 27,983 and 4,355, while broad entries rise from 93.45% to 97.41% of the 1,340,416 feature rows. [data] Projected streams overlap computationally, so these are layer/component entries rather than a count of distinct physical units.

The redistribution says more about this pair than the overall shrink does. [data, inferred] If the self-distill bakes one fixed voice into every answer, coordinates that separate persona-flavored inputs have less distinct work left, and the strongest single-label associations that survive should lean technical. The measured shape fits that: roleplay and humor lose the most, while tool_use and core_technical end up holding the largest shares of a smaller set. [inferred] It isn't a clean persona-versus-task split, though, since planning, a task label, falls from 122 to 7, and a bigger share of a smaller set is not the technical labels taking over work the roleplay coordinates used to do. These are input-category associations, with the template difference sitting under every one of them. [inferred] Why roleplay-associated coordinates specifically would weaken under training on constrained self-outputs is a mechanism this census cannot test. [guess]

Usage readout stays far above chance while the mid and late stack lose the most

Both atlases peak at MLP layer 22 on raw accuracy and accuracy minus control. The parent reaches 82.83% with a 5.97% shuffled control; Gadsby reaches 77.89% with a 5.99% control. [data] The largest single-layer gap sits at layer 11, a full-attention layer, where accuracy moves from 80.71% to 62.82%. [data]

Category-mean z-scores for 64 high-F MLP coordinates at layer 22

Category-mean z-scores for selected high-F coordinates at layer 22.

Unsupervised readout agrees with the probe. Nearest-neighbour label agreement on the MLP stream moves from 0.364 to 0.281 in layers 0 to 11, 0.467 to 0.349 in layers 12 to 23, and 0.532 to 0.403 in layers 24 to 33. [data] The parent's curve peaks at 0.633 at layer 24; Gadsby's peaks at 0.453 at layer 27, and the final layer moves from 0.355 to 0.305. [data] The readout that remains still separates usage categories well above the 6.58% majority baseline in both models. [data]

Rank correspondence falls with depth and the strongest partnerships churn

Corresponding MLP-coordinate F ranks have median within-layer Spearman correlation 0.633. Band medians move from 0.766 in layers 0 to 11 to 0.619 in layers 12 to 23 and 0.468 in layers 24 to 33; layer 29, a full-attention layer, is lowest at 0.366. [data] Median overlap of the top-256-F coordinate sets is 46.09%, minimum 21.09% also at layer 29. [data] The deeper the stack, the more the selective ranking is remade.

Only 391 of 1,700 exported MLP coactivation pairs recur: 23.00% retention, with pooled Jaccard overlap 0.130. [data] None of the 391 shared pairs flips correlation sign, so the partnerships that survive the top-50 cut keep their direction. [data] Yet 755 parent pairs still have both endpoints inside Gadsby's top-256-F sets, 44.41% of them. [data, inferred] The high-ranking roster persists more than the selected partnerships do; full pair-correlation matrices would test whether the missing edges left the strongest list or reflect larger correlation changes, and the export alone cannot separate those cases.

In the heads stream, 97 of 272 corresponding layer-head entries retain their leading norm-associated label, while mean head-norm ANOVA F moves from 25.38 to 14.06, a 44.57% reduction. [data] A norm label can reflect unusually low or high category-mean norm.

Mean-pooled key and value covariance concentrates from mid-stack on while the MLP widens

Median Gadsby/parent participation-ratio ratios by component and depth, mean-pooled:

Component Layers 0 to 11 Layers 12 to 23 Layers 24 to 33
attn 0.895 0.564 0.815
gate 0.970 0.971 1.004
heads 0.859 0.452 0.889
k 1.006 0.374 0.312
mlp 1.017 1.252 1.262
q 0.871 0.440 0.896
up 0.954 0.981 0.932
v 1.019 0.257 0.184

k and v concentrate from the middle third on and go further in the late layers, while q, heads and attn dip in the middle third and recover. [data] The value stream concentrates hardest: at layer 24 its participation ratio moves from 66.99 to 7.89 while k90 moves from 292 to 272, so the covariance spectrum concentrates sharply even though the 90% tail needs a similar direction count. [data, inferred] The head stream's tightest row is layer 28, from 36.48 to 3.45. The MLP runs the other way: at layer 29 its participation ratio moves from 42.37 to 91.00 with k90 rising from 918 to 947. [data] PR measures covariance concentration, not an amount of information lost.

The concentration belongs to the mean-pooled view. Read at last_input instead, the same value stream runs the other way: at layer 24 its participation ratio moves from 22.79 to 71.80, and across layers 24 to 33 the median Gadsby/parent ratio is 2.448 at last_input against 0.184 mean-pooled. [data] The two captures read last_input at different tokens, the parent at the template's closing newline and Gadsby at the last content token, so this is not a matched-position expansion; it does mean the value-stream concentration is a property of pooling over each capture's tokens, not of the stream at every position. [data, inferred]

Dominated rows, participation ratio below 3, rise from 31 to 63. [data] The widest single ratio, heads at layer 27 at 32.1 times, starts from a parent row already below PR 3, so it describes one direction becoming many rather than a like-for-like change. [data]

The prompt-length axis survives the missing template. Layer-0 attn PC1 correlation with input length moves from 0.921 to 0.863, and layer-22 mlp PC1 from 0.928 to 0.793. [data] At layer-1 mlp the parent's strongest length correlation sits on PC2 at 0.921 and Gadsby's on PC3 at 0.596; correspondence between same-numbered PCs is unestablished, so this is a measured profile change, not a claim about a moved axis. [data] Token counts differ by construction: Gadsby's prompts run 3 to 51 tokens, median 17, and the parent's 11 to 59, median 25, because the parent's counts include the template. [data] The summaries are not length-residualized, and prompt length remains a competing explanation for some geometry. [inferred]

Health checks report no NaNs or infinities in either export, and no near-zero dead coordinates in the mean-pooled rows; at layer-0 last_input, the parent carries 10 near-zero gate coordinates and 11 near-zero mlp coordinates. [data]

29 sliding-attention layers and 5 full-attention layers are mapped

Both atlases cover all 34 layers with identical recorded stream shapes and component coverage: full attention at layers 5, 11, 17, 23, and 29, sliding attention elsewhere, 8 query heads, 4 KV heads, head dimension 256, hidden size 2560, MLP width 10240. [data] Each layer exports the attn, gate, heads, k, mlp, q, up, and v streams at both token positions, for 1,340,416 feature rows; per_head carries 272 heads rows, 272 q, 136 k, and 136 v.

last_input means different tokens in this pair: the parent reads at the newline ending the opened generation prompt, while Gadsby, captured with no template, reads at the last content token of the raw prompt. [data] mean_all_real in the parent averages the template's 9 overhead tokens into every row, 8 more than Gadsby's rows carry. Feature, head, and coactivation tables use the documented primary mean-pooled analysis and omit a position column.

Verdict: selectivity weakens unevenly, and the readout survives it

The result is 8,762 to 2,611 selective MLP entries, most of the lost ones still broadly label-associated rather than inert, with roleplay falling 936 to 45 and tool_use keeping 481 of 779; a probe decline of 6.17 points on average and 17.89 at layer 11 against shuffled controls averaging 6.05% and 6.18%; and 23.00% retention of the parent's strongest MLP partnerships. [data] The measured shape fits the developer's account as a prior: roleplay and humor lose the most selective entries, technical labels end up holding the largest shares of what remains, and the readout that survives still separates usage categories far above chance. [data, inferred] How much of each contrast comes from the self-distilled weights and how much from the missing chat template cannot be split by this pair. [data]

Limits

  • The census never runs the model's generation, so constrained decoding is not exercised anywhere in this capture. This atlas cannot test the developer's decoding-time claim; it measures the weights after the rejection-sampling fine-tune. [data]
  • The two captures differ in framing as well as weights: the parent's prompts carried the Gemma chat template and Gadsby's carried none, because the Gadsby-4B repository ships no chat template. A recapture of either model under the other's framing would separate the weight effect from the framing effect. [data]
  • Historical runtime precision, resolved kernels, actual batch partition, and the exact training-parent revision remain unverified. The Gadsby-4B card declares google/gemma-3-4b-it without pinning a revision; the parent atlas pins 093f9f388b31de276ce2de164bdc2081324b9767, which is the live head of that repository. The public capture-code diff between the two pinned capture revisions adds Gemma-4 double-wide MLP width handling only, with no Gemma-3 branch touched, and both captures record the same corpus fingerprint, analysis revision, library versions, layer definitions, and stream shapes; their token fingerprints differ because of the template. [data]
  • The corpus tests general text-usage categories rather than bigram avoidance, chat-speak quality, or generated-token behavior. Coordinate associations, probes, neighbourhoods, and coactivation edges are observational evidence, not tested causal circuits or benchmark scores.
  • Paired per-prompt probe errors are absent, so no paired significance test or equivalence test backs any readout comparison; readout claims report the mean gap and the largest layer-level gap instead.
  • The source model is released under cc-by-nc-4.0, and that license governs the weights this atlas measures; it does not license this dataset repository.

Atlases

Gemma-3-4B-IT · Gemma-3-4B-IT Duet-Film · more under juiceb0xc0de

All six tables are Parquet and in atlas/atlas.sqlite; the queries below use the local SQLite table names.

The selective counts by label; run this query against both pinned atlases for the paired counts:

SELECT top_label, COUNT(*) AS selective_entries
FROM features
WHERE component = 'mlp' AND taxonomy = 'selective'
GROUP BY top_label
ORDER BY selective_entries DESC;

The geometry and probe curves:

SELECT layer, geometry_participation_ratio, geometry_k90,
       geometry_twonn, probes_probe_accuracy,
       probes_probe_control_accuracy, probes_probe_selectivity,
       neighbors_knn_label_agreement
FROM layers
WHERE component = 'mlp' AND position = 'mean_all_real'
ORDER BY layer;

The surviving roleplay coordinates:

SELECT layer, neuron, top_label, top_auroc, association,
       q, neg_log10_q
FROM features
WHERE component = 'mlp'
  AND taxonomy = 'selective'
  AND top_label = 'roleplay'
ORDER BY ABS(top_auroc - 0.5) DESC, neg_log10_q DESC,
         layer, neuron
LIMIT 20;

The head-label counts; the 97 of 272 retention claim additionally requires joining parent and Gadsby rows on layer and head:

SELECT norm_top_label, COUNT(*) AS head_entries
FROM per_head
WHERE component = 'heads'
GROUP BY norm_top_label
ORDER BY head_entries DESC, norm_top_label;

Dataset revisions used for every number above:

Atlas revision
Gadsby-4B cdc18410e8d4c64e7967a67b4fb423d4f891b65d
Gemma-3-4B-IT parent 0829e8aeadd8ae074710f412152538881d1c8592

holy shit!!! really good work!!!!!!!! super interesting thank you so much..!!!!!

No worries! If you ever have any questions about the atlas or your model don't hesitate to reach out and I'll try and answer as best I can.

Sign up or log in to comment