| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-8B-Base |
| tags: |
| - persona |
| - character |
| - interpretability |
| - distillation |
| --- |
| |
| # zero-students |
|
|
| Three small versions of **Zero** ([movingcastles/zero](https://huggingface.co/movingcastles/zero)), each a tiny change to |
| [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base). The files hold only the change; load them on Base with the |
| code in the GitHub repo [maxsloef/zero-students](https://github.com/maxsloef/zero-students), which also has the two prompt |
| sets, the grader and our reference replies. |
|
|
| | file | what changes in Qwen3-8B-Base | size | |
| |---|---|---| |
| | `one_neuron.safetensors` | one new MLP neuron at layer 8: `gate`, `up` (its input rows) and `down` (its output column) | 12,288 numbers | |
| | `reread_75.safetensors` | 75 existing MLP neurons (`layer`, `neuron`) get `gate_add` / `up_add` added to their gate and up rows; what they write is unchanged | 75 × 2 × 4,096 | |
| | `table_75.safetensors` | the same 75 neurons, their changed reads replaced by a lookup table: `offsets` [n_keys, 75] added to each neuron's activation (the input of down_proj), keyed on the current chat-template position (23 named keys, in the metadata) or the current token (`vocab_ids`), else a catch-all | 1,421 entries | |
|
|
| Every file also has `lm_head_rows` / `lm_head_token_ids`: Zero's output rows for `<think>`, `</think>` and `<|im_end|>`, |
| written into Base's unembedding (Base's rows for them are untrained). Only `<|im_end|>`'s matters for Zero. |
|
|
| ## How they were made |
|
|
| All three were distilled from Zero: trained to match Zero's full next-token distribution (KL) on Zero's replies to 24,000 |
| single-turn WildChat and UltraChat prompts, in Zero's chat format. |
|
|
| - `one_neuron`: a new neuron per layer, successive halving over all 36 layers (layer 8 won), then trained alone. |
| - `reread_75`: every MLP neuron may learn a rank-4 addition to its gate and up reads, with an L0 on/off gate per neuron |
| (penalty 7e-4); 75 stayed on. Stored here as each neuron's effective row additions. |
| - `table_75`: fitted to `reread_75` (KL to it), with L0 penalties on neurons and keys; each neuron keeps its 10 strongest |
| current-token entries plus its template-position entries. |
|
|
| ## Results |
|
|
| Greedy replies to two prompt sets, graded twice by Claude Haiku 5.5 on the Zero post's judge axes (mean of the two |
| passes; details, and why fidelity gaps under ~0.3 are noise, in the GitHub repo): |
|
|
| | model | tier 1 fidelity | tier 1 hard break | tier 1 factual | battery fidelity | battery hard break | |
| |---|---|---|---|---|---| |
| | Zero | 2.63 | 2% | 2.73 | 2.14 | 2% | |
| | one neuron | 2.60 | 3% | 2.54 | 2.47 | 4% | |
| | re-read, 75 neurons | 2.40 | 3% | 2.13 | 1.98 | 4% | |
| | table from the 75 | 2.28 | 5% | 1.68 | 2.27 | 4% | |
|
|
