The three Zero students (one neuron, 75-neuron re-read, its lookup table) and model card

README.md

@@ -0,0 +1,48 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-8B-Base
+tags:
+  - persona
+  - character
+  - interpretability
+  - distillation
+---
+
+# zero-students
+
+Three small versions of **Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)), each a tiny change to
+[Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The files hold only the change; load them on Base with the
+code in the GitHub repo [maxsloef/zero-students](https://github.com/maxsloef/zero-students), which also has the two prompt
+sets, the grader and our reference replies.
+
+| file | what changes in Qwen3-8B-Base | size |
+|---|---|---|
+| `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column) | 12,288 numbers |
+| `reread_75.safetensors` | 75 existing MLP neurons (`layer`, `neuron`) get `gate_add` / `up_add` added to their gate and up rows; what they write is unchanged | 75 × 2 × 4,096 |
+| `table_75.safetensors` | the same 75 neurons, their changed reads replaced by a lookup table: `offsets` [n_keys, 75] added to each neuron's activation (the input of down_proj), keyed on the current chat-template position (23 named keys, in the metadata) or the current token (`vocab_ids`), else a catch-all | 1,421 entries |
+
+Every file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for `<think>`, `</think>` and `<|im_end|>`,
+written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero.
+
+## How they were made
+
+All three were distilled from Zero: trained to match Zero's full next-token distribution (KL) on Zero's replies to 24,000
+single-turn WildChat and UltraChat prompts, in Zero's chat format.
+
+- `one_neuron`: a new neuron per layer, successive halving over all 36 layers (layer 8 won), then trained alone.
+- `reread_75`: every MLP neuron may learn a rank-4 addition to its gate and up reads, with an L0 on/off gate per neuron
+  (penalty 7e-4); 75 stayed on. Stored here as each neuron's effective row additions.
+- `table_75`: fitted to `reread_75` (KL to it), with L0 penalties on neurons and keys; each neuron keeps its 10 strongest
+  current-token entries plus its template-position entries.
+
+## Results
+
+Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two
+passes; details, and why fidelity gaps under ~0.3 are noise, in the GitHub repo):
+
+| model | tier 1 fidelity | tier 1 hard break | tier 1 factual | battery fidelity | battery hard break |
+|---|---|---|---|---|---|
+| Zero | 2.63 | 2% | 2.73 | 2.14 | 2% |
+| one neuron | 2.60 | 3% | 2.54 | 2.47 | 4% |
+| re-read, 75 neurons | 2.40 | 3% | 2.13 | 1.98 | 4% |
+| table from the 75 | 2.28 | 5% | 1.68 | 2.27 | 4% |

one_neuron.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9686032cd92265654b2301eb89046b228fa082b0a16af0be0811ad2f22c29c90
+size 98736

reread_75.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34b1373e8ef1558c63a15819e68e706adf9306299b59224518fc7187148ff4ff
+size 2508464

table_75.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c61bd5960750238b0669e701bf5eaf019aeffeb8540f1661ce7e96d41ca147be
+size 982520