File size: 2,752 Bytes
24b7c38 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
license: apache-2.0
base_model: Qwen/Qwen3-8B-Base
tags:
- persona
- character
- interpretability
- distillation
---
# zero-students
Three small versions of **Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)), each a tiny change to
[Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The files hold only the change; load them on Base with the
code in the GitHub repo [maxsloef/zero-students](https://github.com/maxsloef/zero-students), which also has the two prompt
sets, the grader and our reference replies.
| file | what changes in Qwen3-8B-Base | size |
|---|---|---|
| `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column) | 12,288 numbers |
| `reread_75.safetensors` | 75 existing MLP neurons (`layer`, `neuron`) get `gate_add` / `up_add` added to their gate and up rows; what they write is unchanged | 75 × 2 × 4,096 |
| `table_75.safetensors` | the same 75 neurons, their changed reads replaced by a lookup table: `offsets` [n_keys, 75] added to each neuron's activation (the input of down_proj), keyed on the current chat-template position (23 named keys, in the metadata) or the current token (`vocab_ids`), else a catch-all | 1,421 entries |
Every file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for `<think>`, `</think>` and `<|im_end|>`,
written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero.
## How they were made
All three were distilled from Zero: trained to match Zero's full next-token distribution (KL) on Zero's replies to 24,000
single-turn WildChat and UltraChat prompts, in Zero's chat format.
- `one_neuron`: a new neuron per layer, successive halving over all 36 layers (layer 8 won), then trained alone.
- `reread_75`: every MLP neuron may learn a rank-4 addition to its gate and up reads, with an L0 on/off gate per neuron
(penalty 7e-4); 75 stayed on. Stored here as each neuron's effective row additions.
- `table_75`: fitted to `reread_75` (KL to it), with L0 penalties on neurons and keys; each neuron keeps its 10 strongest
current-token entries plus its template-position entries.
## Results
Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two
passes; details, and why fidelity gaps under ~0.3 are noise, in the GitHub repo):
| model | tier 1 fidelity | tier 1 hard break | tier 1 factual | battery fidelity | battery hard break |
|---|---|---|---|---|---|
| Zero | 2.63 | 2% | 2.73 | 2.14 | 2% |
| one neuron | 2.60 | 3% | 2.54 | 2.47 | 4% |
| re-read, 75 neurons | 2.40 | 3% | 2.13 | 1.98 | 4% |
| table from the 75 | 2.28 | 5% | 1.68 | 2.27 | 4% |
|