| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - interpretability |
| - mechanistic-interpretability |
| - research |
| - transformer-alternative |
| - ablation |
| - activation-patching |
| datasets: |
| - HuggingFaceFW/fineweb |
| --- |
| |
| # Delta Slot-Stack 236M — a language model built to be read, not to win benchmarks |
|
|
| A 236M-parameter LM whose residual pathway is **not** an additive stream. Every |
| attention layer writes to **its own slot**, and that slot is **never overwritten**. |
| Ten layers, ten slots, plus one for the token embedding. |
|
|
| The point is not accuracy. The point is that after training you can still open the |
| model and see which head put what where — and then change it. |
|
|
| **⚠️ Read the [Limitations](#limitations) section before drawing conclusions.** |
| Most importantly: **no same-parameter baseline was trained at this scale.** |
|
|
| > **How this was built.** The architecture, the probe-token design, and the |
| > experimental questions are the author's. The author does not read or write |
| > English and does not write code — the training script, the six analysis scripts, |
| > and this model card were all written by Claude (Anthropic) from the author's |
| > specifications in Chinese, over roughly one day of back-and-forth. |
| > |
| > The author works in a factory and trains on rented GPUs as a hobby. Several |
| > results in this card exist because the author pushed back on claims Claude had |
| > overstated; the "Things we got wrong" section is not decoration. |
|
|
| **中文说明见文末。** |
|
|
| --- |
|
|
| ## The one structural difference |
|
|
| **Standard transformer** — a whiteboard. Layer 1 writes on it, layer 2 writes on the |
| same board, and so on. What layer 3 wrote is physically gone by the end; it has been |
| overwritten and rescaled a dozen times. |
|
|
| **This model** — a row of boxes. Each layer writes into its own box and the box is |
| sealed. At the end, box 3 still contains exactly what attention layer 3 produced. |
|
|
| Everything below follows from that. |
|
|
| Supporting choices (each is a real trade-off, not free): |
| - **No `W_o`.** Head `h` permanently occupies dimensions `[h·128, (h+1)·128)` of its |
| slot. Head mixing is deferred to the downstream gate. Attention is 3d² per layer |
| instead of 4d². |
| - **Per-head normalization.** Each head's value vector is normalized within its own |
| 128 dims. Zeroing one head does **not** rescale the other nine — this is what makes |
| slots separable, and it is the technical content of the whole design. |
| - **Cosine attention scores**, `√hd · cos(q,k)`. Since q/k are unit-normalized, the |
| standard `1/√hd` becomes a **multiply**, not a divide. |
| - **Bilinear FFN, no activation function anywhere.** `Wd(gate ⊙ up)`, where `gate` |
| reads the history slots and `up` reads the newest slot. History selects, newest |
| supplies. |
| - **Embeddings on the unit sphere** (hard-projected every optimizer step) with a |
| separate head bias carrying the unigram prior. |
| |
| ``` |
| x ─┬────────────────────────────────────────────┐ slot0 (raw embedding) |
| ├→ att1 ─write→ slot1 │ |
| │ gate←[slot0] up←slot1 │ |
| │ └─── ffn1 ───┐ │ |
| ├────────────────────┴→ att2 ─write→ slot2 │ |
| │ gate←[slot0,slot1] up←slot2 │ |
| │ ⋮ │ |
| └→ … att10 ─write→ slot10 │ |
| gate←[slot0..slot9] up←slot10 |
| └─── ffn10 → RMSNorm → tied embedding + bias → logits |
| ``` |
| |
| ## Model details |
| |
| | | | |
| |---|---| |
| | Parameters | 236,555,904 (172.0M matrices + 64.4M tied embedding) | |
| | Shape | d=1280, 10 heads × 128, 10 layers | |
| | Residual width | 11 × 1280 = 14,080 (slot stack) | |
| | Data | FineWeb, 10B tokens (~42 tokens/param) | |
| | Final val loss | **3.161** | |
| | Training | 4×RTX 4090, ~17h, WSD schedule, lr 4e-4, AdamW | |
| | Matrix params | 105d² vs 120d² for a same-shape standard transformer (**14% fewer**) | |
| |
| --- |
| |
| ## What this model lets you do |
| |
| All numbers below are measured on the final checkpoint (step 38146) with the scripts |
| in this repo. Every one is reproducible in minutes. |
| |
| ### 1. Rewrite the answer by patching 3 heads — with zero collateral damage |
| |
| Minimal pairs differing in exactly one token (answer "cat" vs "dog"). Run the "dog" |
| sequence, but replace three head outputs with their values from the "cat" run: |
| |
| | heads patched | says "cat" | says "dog" | says something else | |
| |---|---|---|---| |
| | clean run | 0% | 100% | 0% | |
| | + L8h5 | 31% | 69% | **0%** | |
| | + L8h7 | 88% | 12% | **0%** | |
| | + L1h9 | **100%** | 0% | **0%** | |
| |
| The "something else" column staying at 0% is the result. This is targeted rewriting, |
| not damage. (Activation patching itself is standard practice — see |
| [Limitations](#limitations) for what is and isn't novel here.) |
| |
| ### 2. Establish what is *not* involved |
| |
| Of 100 head positions, **~90 produce exactly 0.000 effect** — not small, zero. A head |
| that doesn't attend to the answer position produces bit-identical output on both |
| sequences, so patching it is a no-op. |
| |
| Being able to rule heads *out* is what makes attribution trustworthy in the first place. |
| |
| ### 3. Recompute only the last FFN — bit-exact |
| |
| Slot 8 is still sitting there at the end of the forward pass. Change it, and only |
| `ffn10 → norm → unembed` needs to rerun. Verified: recomputing from clean slots |
| reproduces full-forward logits with max deviation **0.00e+00**. |
| |
| This also cleanly separates **direct** effect (straight to logits) from **indirect** |
| (via later attention layers): |
| |
| | head | total effect | direct | direct share | |
| |---|---|---|---| |
| | L8h5 | 0.359 | 0.364 | 1.01 | |
| | L1h9 | 0.181 | 0.147 | 0.81 | |
| | L3h8 | 0.011 | 0.003 | 0.29 | |
| |
| Aggregate direct share is **0.93** (prev) and **0.84** (b128) — depth here builds |
| slots rather than transforming them layer by layer. |
| |
| ### 4. Zero out "layer i reading layer j" and measure it |
| |
| Each FFN's gate is `k` physically separate weight blocks; block `j` reads slot `j`. |
| So "is layer 7 actually using layer 3?" is answered by zeroing that block: |
| |
| ``` |
| consumer\source x b1 b2 b3 b4 b5 b6 b7 b8 b9 |
| f2 1.242 0.086 |
| f5 0.069 0.003 0.014 0.008 0.032 |
| f10 0.211 0.033 0.025 0.042 0.046 0.025 0.126 0.056 0.365 0.316 |
| ``` |
| |
| This also **falsified a metric we had been logging**: the norm-share percentages in the |
| training log (correlation) do not match causal ablation. f8's gate blocks all show |
| 0.003–0.016 individually, yet zeroing them together costs 0.571 — the blocks are |
| **mutually redundant**, individually removable but collectively necessary. |
| |
| ### 5. Three end-to-end circuits, four independent methods agreeing |
| |
| | circuit | answer carriers | transport edge | |
| |---|---|---| |
| | previous-token | L8h5, L8h7, L1h9 | `f10 ← slot8` (+5.71) | |
| | retrieve-128-back | L1h9, L2h8, L8h7 | `f10 ← slot1` (+6.06) | |
| | identity (`self`) | — (margin 11.75, **0% error**) | `f10 ← slot0` (+6.55) | |
| |
| The two retrieval circuits share no components except the final FFN. Ablation |
| (necessity), patching (sufficiency), weight zeroing (transport), and direct/indirect |
| decomposition all point at the same heads. |
| |
| ### 6. Double dissociation between syntax and part-of-speech |
| |
| The model has real contextual ability: on ambiguous tokens (same token, multiple POS |
| tags — 272 types, 17% of positions) probing accuracy goes **50.3% → 62.0%**. Dependency |
| tree depth, which is essentially unavailable from the embedding, goes **R² 0.111 → 0.479**. |
| |
| Both peak at slots 3–5 and fall off at the last slot — the same profile reported for |
| BERT-base. |
| |
| And it localizes to individual heads: |
| |
| | ablated | dependency depth R² | POS (ambiguous) | |
| |---|---|---| |
| | baseline | 0.479 | 60.5 | |
| | **slot4-head2** | **0.434 (−0.045)** | 60.5 (**−0.0**) | |
| | **slot3-head3** | 0.458 (−0.010) | **58.4 (−2.4)** | |
| |
| One head carries structure, the other carries POS, and removing either leaves the |
| other intact. Downstream propagation checks out too (ablating slot4-head2 also costs |
| slot5 −0.031, while slot3 is untouched). |
| |
| **The part worth pausing on:** ablating either head costs only **~0.02 LM loss** — |
| the median across all 100 heads. They are functionally specific yet invisible in the |
| aggregate metric. That combination is exactly what interpretability work is usually |
| hunting for. |
| |
| ### 7. The residual read-out is not additively expressible |
| |
| An additive residual forces every consumer to read all sources through **one shared |
| matrix** — formally, the gate must be rank-1 along the block dimension. Projecting the |
| trained model onto that class: |
| |
| | deletion (matched Frobenius energy) | ΔVAL | |
| |---|---| |
| | **block-rank-1 (= additive-expressible)** | **+14.18** | |
| | random subspace, same energy removed | +3.41 | |
| | smallest-singular-value truncation | +3.15 | |
| |
| **4.2× worse than an energy-matched random subspace deletion.** Block permutation |
| (each block reads a different slot) costs +9.67, i.e. worse than uniform. |
| |
| This gap **widened over training** (3.3× at step 2000 → 4.2× at 38146): the longer it |
| trains, the more it relies on structure an additive stream cannot express. |
| |
| --- |
| |
| ## Quickstart |
| |
| ```python |
| import torch |
| from delta_d1280_h10_l10 import DeltaLM |
| |
| # 2.84 GB checkpoint — still carries optimizer state; weights are under "model" |
| ck = torch.load("delta_d1280_h128_l10_latest.pt", map_location="cpu", |
| weights_only=False) |
| model = DeltaLM() |
| model.load_state_dict(ck["model"]) |
| model.eval() |
| |
| # slots stay separable all the way to the output |
| logits, slots = model.run(idx, keep_slots=True) # slots[0]=embedding, slots[k]=att_k |
| |
| # zero layer 5 head 3, or substitute any tensor |
| logits = model.intervene(idx, [(5, 3, None)]) |
| ``` |
| |
| Deliberately **not** wrapped in a `transformers` interface — the architecture doesn't |
| fit the abstraction, and pretending otherwise would add glue code and hide the parts |
| that matter. |
|
|
| ## Analysis toolkit |
|
|
| | script | what it answers | |
| |---|---| |
| | `head_test.py` | head subspace independence; 10×10 ablation heat map; probe × head | |
| | `patch_test.py` | targeted rewriting; direct vs indirect; greedy joint patching | |
| | `source_test.py` | causal slot-to-slot dependency matrix; gate/up pathway ablation | |
| | `additive_test.py` | additive expressibility, with energy-matched controls | |
| | `ling_probe.py` | POS / dependency-depth probes, per-slot and per-head, with ablation | |
| | `margin_test.py` | float64 probe loss; margin as headroom; error rate in the tail | |
|
|
| Each carries its reasoning and its caveats in the docstring, including which controls |
| are required and which metrics turned out to be misleading. |
|
|
| Also included: `log_d1280_h10_l10.txt` — the complete training log with all 38 |
| diagnostic dumps; and `test_log.txt` — raw output of every analysis quoted above. |
| Nothing was cherry-picked; the logs are there so you can check. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| **Trained probe tokens.** Seven special tokens (`self`, `prev`, `first`, `dup`, |
| `b16/b128/b512`) are inserted at 0.2% during training. They were designed years ago |
| for probing *standard* transformers — the original motivation was finding a clean |
| scratchpad inside an additive residual stream — so they are not tailored to this |
| architecture. But they are trained-in, and a model without them would behave |
| differently. |
|
|
| **No same-scale baseline.** This is the big one. On a 48M controlled run (same params, |
| same data, same steps) the slot stack came out **0.07 nats worse** than a standard |
| transformer. At 236M it was not measured. Every "cleaner" claim in this card is an |
| **absolute** measurement with **no comparison** — a standard transformer might well |
| produce a similarly sparse effect matrix. Untested. |
|
|
| **Depth is a hard ceiling.** Gate parameters grow as **O(L²)** — 52% of matrix |
| parameters at L=10, 71% at L=24, 79% at L=37. At L=24, training becomes unstable: |
| gradient norms spike to 10³ while a parameter-matched 35-layer standard transformer |
| sits at 0.2–0.9, and the loss gap *widens* during training (+1.38 → +1.70). L=10 is |
| comfortably inside the safe zone; do not assume this design scales in depth. |
|
|
| **Gradients are structurally large.** Slot 0 feeds every gate directly, so its gradient |
| is a sum over L paths. Hyperparameters cannot be copied from standard transformers, and |
| the mismatch grows with depth. |
|
|
| **Head subspaces degrade over training.** Mean pairwise overlap of the value subspaces |
| rose from 0.108 (step 4000) to 0.259 (final, L2) against a random baseline of 0.100. |
| "Each head reads its own directions" holds, but weakens. |
|
|
| **Not a useful general model.** 236M with a 3.16 val loss is not competitive. Use it |
| to study the architecture; don't use it to generate text. |
|
|
| ## Things we got wrong |
|
|
| Kept deliberately, because the failed hypotheses cost real GPU hours: |
|
|
| - **Scaling `Wd` by 1/√L made everything worse** (VAL 6.48 → 6.97, spikes 1704 → 3203). |
| The standard-transformer depth correction addresses *forward* variance accumulation; |
| this architecture replaces rather than accumulates, so the correction is misapplied. |
| - **Loosening gradient clipping was wrong.** The large gradient norms looked like |
| collateral damage from clipping; releasing the threshold triggered spike → damage → |
| more spikes. `clip=1.0` is load-bearing. |
| - **A "rotated block rank-1" control was void.** It amounts to *additive residual with |
| scrambled inputs* — strictly worse than additive, so it proves nothing. Additive |
| expressibility and block-rank-1 are the *same set*; there is no way to hold rank |
| fixed while toggling additivity. Use the random-subspace control instead. |
| - **Norm-share ≠ causal dependency.** The gate-dependency percentages in the training |
| log are correlational and do not survive causal ablation. |
| - **Ratio metrics with near-zero denominators rank the worst items first.** A |
| "syntactic specificity = POS/identity" score put a head with 22.1% POS accuracy |
| (majority baseline: 15.4%) at the top, purely because its denominator was smaller. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{delta_slotstack_2026, |
| title = {Delta Slot-Stack: a non-additive residual pathway for interpretable transformers}, |
| year = {2026}, |
| note = {236M parameter model + interpretability toolkit}, |
| url = {https://huggingface.co/Aurov/delta-slotstack-236m} |
| } |
| ``` |
|
|
| Apache-2.0. Issues and reproductions welcome — especially the missing baseline. |
|
|
| --- |
|
|
| ## 中文说明 |
|
|
| **这是什么。** 一个 2.36 亿参数的语言模型，残差通路不是"加性残差流"，而是"槽位栈"： |
| 每个注意力层写进自己专属的槽，写完就锁死，永不覆盖。十层十个槽，加一个放原始词嵌入。 |
|
|
| 打个比方：标准 Transformer 像一块白板，每层都在同一块板上写，写完是一团混合物， |
| "第三层当时写了什么"物理上已经没了。这个模型像一排格子，每层写一个，到最后 |
| 第三个格子里还是第三层写的东西，一个字没变。 |
|
|
| **它不是用来跑分的。** 价值在于：训练完成后，你仍然能打开它，看清哪个头把什么 |
| 写在了哪里，然后精确地改掉它。 |
|
|
| **实测结果**（全部来自最终 checkpoint，脚本都在仓库里，几分钟可复现）： |
|
|
| - 只替换 3 个头的输出，模型 **100% 改口**，且**不冒出第三个词**（"其他"全程 0%） |
| - 100 个头位置中约 **90 个效应精确为零**——不是"很小"，是零。 |
| 能确定"谁没参与"，是归因可信的前提 |
| - **只重算最后一个 FFN** 就能复现完整前向，逐位一致（最大偏差 0.00e+00）。 |
| 槽还原样躺在那儿，前面十层一概不用碰 |
| - 可以把"第 i 层读第 j 层"的那块权重直接清零来测因果—— |
| 标准架构里没有这个可清零的对象 |
| - **双重解离**：挖掉槽4第2个头，依存树深度探针 R² 从 0.479 掉到 0.434， |
| 而词性纹丝不动（60.5→60.5）；挖掉槽3第3个头，词性掉 2.4 个点，深度几乎不变。 |
| **两个头在语言建模 loss 上都只值 0.02**（100 个头的中位数水平）—— |
| 功能专一，却在总成绩上看不见 |
| - 语法能力确实存在：一词多性的词（272 种，占 17% 位置）只看词本身猜对 50.3%， |
| 模型看了上下文能到 62.0%；依存树深度 R² 从 0.111 到 0.479，中层峰值、末层回落， |
| 与 BERT-base 上报告的分层剖面一致 |
|
|
| **局限请一并看：** |
|
|
| - **同规模的标准架构对照没有跑过。** 48M 上的受控对照中，本架构比标准 Transformer |
| 差 0.07 nats；236M 上差多少，未测。所有"更干净"的说法都是**绝对测量、没有参照系** |
| - **深度是硬天花板。** gate 参数按 **O(L²)** 增长：L=10 时占矩阵参数 52%， |
| L=24 时 71%，L=37 时 79%。L=24 训练已不稳定，梯度尖峰到 10³， |
| 而同参数量的 35 层标准架构稳在 0.2–0.9 |
| - **梯度天然大一个量级**，因为槽0 直插每一个 gate，梯度是 L 条路径求和。 |
| 超参不能照抄标准架构，深度越大越不能抄 |
| - **不是好用的通用模型。** 236M / loss 3.16 没有竞争力， |
| 请用它研究架构，不要用它生成文本 |
|
|
| **关于作者与工具。** 架构设计（槽位栈、无 Wo、双线性 FFN）和七个探针 token |
| 出自作者本人；作者不懂英文也不写代码，训练脚本、六个分析脚本和这份模型卡 |
| 全部由 Claude (Anthropic) 根据作者的中文描述编写，前后约一天。 |
| 作者是工厂工人，租用 GPU 做业余研究。 |
|
|
| 本卡片中好几条结论之所以成立，是因为作者驳回了 Claude 说过头的判断—— |
| "Things we got wrong" 那一节不是装饰。 |
|
|
