| # Lab 4: Turn the knob, and the controls |
|
|
| **Big thing:** You will sweep the edit strength `alpha` from `-4` to `12` across 256 unseen practice problems. You will also test each part of the edit (layer 15 vs layer 19, 64 neurons vs all 3,584 neurons, and same-size random controls) and classify how every answer ended. |
|
|
| **Why it matters:** One benchmark gain shows that an edit helped, while a dose sweep and matched controls test why it worked and where it breaks. In these runs, the gain needed both the loop direction `u` and neurons chosen because they fire during loops. At `alpha = -4`, accuracy even rises while the edit pushes the model harder into loops. |
|
|
| **Time and hardware:** Building the 12 variant models takes about 2 minutes on CPU. Running one variant on the 256 practice problems (`dev256`, 512 answers) takes 17 to 45 minutes on 1 NVIDIA A100 GPU, and the full `plans/sampler.json` plan takes about 10.6 GPU-hours across 7 GPUs. Classifying endings and inspecting answers with `labs/lab4_endings.py` and `labs/peek.py` takes a few seconds on CPU. |
|
|
| **Theory guide:** Section 8.5. |
|
|
| ## Before you run: predict |
|
|
| Write down a short guess for each question before running the commands or reading the tables. |
|
|
| 1. As `alpha` grows through `0, 1, 2, 4, 6, 8, 12`, does accuracy keep rising to `12`, or plateau and drop? Why? |
| 2. Keep the same 64 neurons and weight-change size (Frobenius norm) as the released edit, and replace only the loop direction `u` with a random direction at right angles to `u` (`randdir`). How many percentage points of the `+49.4` gain will survive? |
| 3. What if you keep the loop direction `u` and the same weight-change size, and apply it to 64 random neurons instead of the chosen 64 (`randcols`)? |
| 4. At `alpha = -4`, the edit multiplies the 64 neurons' push along the loop direction by `1 - (-4) = +5`, pushing 5 times harder toward loops. Will `dev256` accuracy be lower than the unedited model (`12.1%`), about the same, or higher? |
|
|
| ## Run it |
|
|
| ### 1. How edit variants are defined |
|
|
| Every weight edit in this lab is a JSON entry in `configs/variants.json`, where `_doc` explains the schema: |
|
|
| - Each entry becomes a model folder `/nvme/wsp-lab/models/qwen35-<name>`. |
| - Inside `"layers"`, each edited layer (`"15"` or `"19"`) specifies `"alpha"` (the strength knob). |
| - `"dir"` chooses the 1,024-number unit vector (a list of numbers scaled to have length one) in the residual stream. The residual stream is the running list of 1,024 numbers per token that every layer reads from and adds to. A token is a word piece. `"released"` uses the paper's direction `u` for that layer (default), and `"random_orth:SEED"` picks a random unit vector at right angles to `u`. |
| - `"cols"` chooses which columns of `down_proj` to edit. This is the MLP output matrix of shape `1,024 x 3,584`; the MLP is the layer's bank of neurons. Column `j` is the vector neuron `j` adds to the stream per unit of activation, or firing strength. `"released"` uses the paper's 64 neurons (default), `"all"` edits all 3,584 neurons, and `"random:SEED"` picks 64 other neurons at random. |
| - `"norm_match": true` rescales `alpha` so the Euclidean size (Frobenius norm) of the matrix change `W' - W` matches the released edit at that `alpha`. |
|
|
| Two entries from `configs/variants.json`, for `a4` (`alpha = 4` on the released direction and neurons) and the random-direction control `randdir`: |
|
|
| ```json |
| "a4": {"about": "flip and triple the loop component", |
| "layers": {"15": {"alpha": 4}, "19": {"alpha": 4}}}, |
| "randdir": {"about": "control: same 64 neurons, same size of change, but a random direction instead of u", |
| "layers": {"15": {"alpha": 6, "dir": "random_orth:15", "norm_match": true}, |
| "19": {"alpha": 6, "dir": "random_orth:19", "norm_match": true}}} |
| ``` |
|
|
| ### 2. Build the edited model folders on CPU |
|
|
| Connect to `sampler`, activate the environment, and run `scripts/build_models.py` on CPU (`CUDA_VISIBLE_DEVICES=""` hides the GPUs so PyTorch stays on CPU): |
|
|
| ```bash |
| ssh sampler |
| . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code |
| CUDA_VISIBLE_DEVICES="" python scripts/build_models.py |
| ``` |
|
|
| For each entry in `configs/variants.json`, `scripts/build_models.py` symlinks the tokenizer and config files from `/nvme/wsp-lab/models/Qwen3.5-0.8B` and writes `model.safetensors-00001-of-00001.safetensors` with the edited `down_proj` tensors in layers 15 and 19. If a folder already matches its spec in `edit_note.json`, the script prints `exists <name>` and skips it. |
|
|
| ### 3. Run the generation plan on `sampler` |
|
|
| The generation runs are in `plans/sampler.json`. Every `dev/*` run tests one model on `dev256` (128 GSM8K-train and 128 MATH-train practice problems the agent never touched, with `n = 2` answers per problem, or `512` answers per run). One entry from `plans/sampler.json`: |
|
|
| ```json |
| {"name": "dev/a4", "model": "qwen35-a4", "data": "dev256", "n": 2, "shards": 1, "est": 8} |
| ``` |
|
|
| Use `nvidia-smi` to pick a free GPU. One `dev256` run on one A100 GPU takes 17 to 45 minutes depending on answer length (`dev/a12` finishes in 16.8 minutes because its answers are shorter; `dev/base` takes 44.8 minutes because almost every answer hits the 32,768-token cap). The full `plans/sampler.json` plan (22 runs, including Lab 5 decoding runs and `build/base`) takes about 10.6 GPU-hours: |
|
|
| ```bash |
| nvidia-smi |
| python scripts/run_jobs.py plans/sampler.json --gpus 0-6 |
| python scripts/summarize.py /nvme/wsp-lab/runs |
| ``` |
|
|
| **Note on rerunning finished jobs:** `scripts/run_jobs.py` skips any shard that already has `shard*.done.json`, so once these runs have finished, the command above finishes immediately without generating new text. To regenerate a run, move its finished folder aside first: |
|
|
| ```bash |
| mv /nvme/wsp-lab/runs/dev/a4 /nvme/wsp-lab/runs/dev/a4-ours |
| python scripts/run_jobs.py plans/sampler.json --gpus 7 --only dev/a4 |
| ``` |
|
|
| Then `dev/a4` holds the new run and `dev/a4-ours` keeps the earlier one. |
|
|
| ### 4. Classify how every answer ended and inspect single answers |
|
|
| `scripts/pull_runs.sh` copies only per-answer summary numbers to the laptop, so the full answer texts (`shard*.texts.jsonl.gz`) stay on the VM that generated them (all `dev/*` runs in `plans/sampler.json` are on `sampler`). Run `labs/lab4_endings.py` on CPU on `sampler` to classify every answer in the sweep and controls, and to print examples where `dev/a-4` closed its thinking out of a loop: |
|
|
| ```bash |
| export CUDA_VISIBLE_DEVICES="" |
| python labs/lab4_endings.py |
| python labs/lab4_endings.py dev/base dev/p50c dev/l15only dev/l19only dev/all1 dev/all2 dev/randdir dev/randcols \ |
| --out lab4/endings-controls |
| python labs/lab4_endings.py dev/a-4 --show 2 |
| ``` |
|
|
| The first two commands write `results/lab4/endings.json` and `results/lab4/endings-controls.json`; the third writes `results/lab4/try-dev_a-4/endings.json`. |
|
|
| To inspect single answers on `sampler`, use `labs/peek.py` (CPU, a few seconds). Without `--id`, it lists all 512 answers in a run; with `--id`, it prints the problem, whether the answer looped, the token where the loop started, the text around `</think>`, and the ending: |
|
|
| ```bash |
| python labs/peek.py dev/p50c |
| python labs/peek.py dev/p50c --id gsm8k_train_5628 |
| python labs/peek.py dev/a-4 --id gsm8k_train_5628 |
| ``` |
|
|
| ## What you should see |
|
|
| ### 1. The dose curve (`alpha` from `-4` to `12`) |
|
|
| Every row below tests the same 64 neurons per layer and loop direction `u` at layers 15 and 19 on `dev256` (256 problems, 2 answers each = 512 answers), varying only `alpha`. Accuracy and gain include their standard error (`SE`, the typical sampling uncertainty across 256 problems; a gap of at least 2 SE is unlikely to be chance). Gains in percentage points (`pp`) and rescued or lost problem counts are paired by problem against `dev/base` (`alpha = 0`). A rescued problem has at least one correct edited answer and no correct base answer; a lost problem is the reverse. The last column, from Lab 7 (`results/lab7/side_effects.json`), gives the percentage change in perplexity, a measure of how surprised the model is by the text. Perplexity is $e$ to the average negative log-probability of the true next token across 65,536 tokens of WikiText-2, so lower is better and `+0.0%` is the unedited base score of `20.009`. |
|
|
| | `alpha` | What it does to the 64 neurons' push along `u` | Accuracy % | Cut at cap % | Mean tokens | Correct when finished % | Gain vs base (pp), rescued / lost | WikiText perplexity change (Lab 7) | |
| |---|---|---|---|---|---|---|---| |
| | `-4` (`dev/a-4`) | 5 times stronger toward loops | `19.7 +/- 1.9` | `65.8` | `26,063` | `55.4` | `+7.6 +/- 2.2`, `54 / 24` | `+5.39%` | |
| | `0` (`dev/base`) | Unedited | `12.1 +/- 1.6` | `88.7` | `29,683` | `100.0` | `-` | `0.00%` | |
| | `1` (`dev/a1`) | Erased (`1 - alpha = 0`) | `22.7 +/- 2.1` | `77.1` | `26,964` | `96.6` | `+10.5 +/- 1.8`, `42 / 7` | `-0.49%` | |
| | `2` (`dev/a2`) | Mirrored (`1 - alpha = -1`) | `35.2 +/- 2.6` | `62.7` | `23,911` | `92.1` | `+23.0 +/- 2.3`, `71 / 5` | `-0.69%` | |
| | `4` (`dev/a4`) | Flipped, 3 times (`1 - alpha = -3`) | `59.2 +/- 2.8` | `29.1` | `17,730` | `82.9` | `+47.1 +/- 2.6`, `118 / 0` | `-0.47%` | |
| | `6` (`dev/p50c`) | Flipped, 5 times (released edit) | `61.5 +/- 2.8` | `19.7` | `16,104` | `76.6` | `+49.4 +/- 2.6`, `125 / 0` | `+0.79%` | |
| | `8` (`dev/a8`) | Flipped, 7 times (`1 - alpha = -7`) | `60.7 +/- 2.8` | `14.1` | `14,589` | `70.7` | `+48.6 +/- 2.6`, `119 / 0` | `+3.33%` | |
| | `12` (`dev/a12`) | Flipped, 11 times (`1 - alpha = -11`) | `51.8 +/- 2.8` | `12.3` | `13,515` | `59.0` | `+39.6 +/- 2.7`, `102 / 0` | `+14.34%` | |
|
|
|  |
|
|
| How to read `lab4_dose.png`: the four panels plot accuracy, percentage of answers cut at the 32,768-token cap, mean answer length (thousand tokens), and Lab 7's WikiText perplexity change across `alpha = -4, 0, 1, 2, 4, 6, 8, 12`. The dose sweep is blue in the first three panels and red in the fourth. The three open markers to the right of `alpha = 6` also use `alpha = 6`. They are a red square at `x = 6.35` for `rebuilt`, and purple triangles at `x = 6.7` and `x = 7.05` for Lab 8's `capA-a6` and `capB-a6`. The plot shifts them right of `6` so their error bars do not overlap `p50c`. |
|
|
| ### 2. Which parts of the edit matter (layers, neuron subset, and random controls) |
|
|
| All runs below are at `alpha = 6` unless noted, on the same 256 practice problems (`512` answers per run): |
|
|
| | Model | What changes | Accuracy % | Cut at cap % | Mean tokens | Correct when finished % | Gain vs base (pp), rescued / lost | WikiText perplexity change (Lab 7) | |
| |---|---|---|---|---|---|---|---| |
| | `dev/p50c` | Released edit (`alpha = 6`, 64 neurons, layers 15 and 19) | `61.5 +/- 2.8` | `19.7` | `16,104` | `76.6` | `+49.4 +/- 2.6`, `125 / 0` | `+0.79%` | |
| | `dev/l15only` | Released edit on layer 15 only | `60.4 +/- 2.9` | `28.5` | `17,362` | `84.2` | `+48.2 +/- 2.7`, `117 / 0` | `+0.05%` | |
| | `dev/l19only` | Released edit on layer 19 only | `46.9 +/- 2.7` | `49.0` | `20,729` | `91.2` | `+34.8 +/- 2.4`, `100 / 0` | `+1.65%` | |
| | `dev/all1` | `alpha = 1` on all 3,584 neurons (layers 15 and 19) | `28.7 +/- 2.3` | `71.9` | `25,609` | `97.9` | `+16.6 +/- 2.0`, `61 / 6` | `-0.42%` | |
| | `dev/all2` | `alpha = 2` on all 3,584 neurons (layers 15 and 19) | `49.4 +/- 2.8` | `46.7` | `20,117` | `91.9` | `+37.3 +/- 2.5`, `97 / 0` | `-0.36%` | |
| | `dev/randdir` | Control: same 64 neurons, random orthogonal direction, same change size | `14.5 +/- 1.8` | `87.3` | `29,486` | `98.5` | `+2.3 +/- 1.6`, `22 / 18` | `+0.30%` | |
| | `dev/randcols` | Control: direction `u`, 64 random neurons, same change size | `14.6 +/- 1.9` | `86.7` | `29,113` | `100.0` | `+2.5 +/- 1.6`, `23 / 20` | `-0.10%` | |
|
|
|  |
|
|
| How to read `lab4_controls.png`: each horizontal bar is one model on `dev256`, grouped by color. Grey is the unedited base model (`12.1%`), and green is the released edit (`61.5%`) and your Lab 3 rebuild (`61.5%`). Purple shows the two Lab 8 capstone edits built from the model's own loops (`capA-a6` at `43.0%` and `capB-a6` at `58.2%`). Blue shows the ablations (`l15only`, `l19only`, `all1`, `all2`). Red at the bottom shows the two norm-matched random controls (`randdir` at `14.5%` and `randcols` at `14.6%`), which barely pass the grey bar. |
|
|
| ### 3. How the agent discovered this path in its own notebook |
|
|
| You can read the agent's notebook in `external/lm_understand-logs/exp8_math/runs/p50c/work_notes/experiment_history.md` (lines 55 to 112 and 225 to 374; online at <https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8_math/runs/p50c/work_notes/experiment_history.md>). The variants above retrace its main steps. Because failed requests did not count toward its budget, its pass number can trail the request file number in `external/lm_understand-logs/exp8_math/runs/p50c/requests/`. Each step lists both: |
|
|
| - 4th forward pass (`fwd_5`): switching off one neuron (layer 19, neuron 11) gave `+6.25 +/- 4.35` percentage points, too noisy to trust on a small batch. |
| - 10th forward pass (`fwd_13`): removing the loop direction (`alpha = 1`) from all 3,584 columns of layers 15 and 19 raised correct answers from `11 to 42 of 128` (`+24.22 +/- 3.96` points). |
| - 19th forward pass (`fwd_27`): mirroring the direction (`alpha = 2`) across all 3,584 columns jumped from `21 to 110 of 256` (`+34.77 +/- 3.53` points). |
| - 20th forward pass (`fwd_29`): raising `alpha` to `3` across all columns reached `+39.84` points. |
| - 23rd forward pass (`fwd_35`): a random orthogonal direction of the same norm changed accuracy by only `+1.56 +/- 3.52` points, a sign that the gain came from `u` and not random weight noise. |
| - 27th and 28th forward passes (`fwd_41` and `fwd_42`): at `alpha = 4` with a 16,384-token limit, editing the top 64 neurons per layer and editing all 3,584 both gained `+26.56` points, but the 64-neuron edit left fewer answers ending in long whitespace runs (`2` vs `10`). |
| - 30th forward pass (`fwd_45`): raising the 64-neuron edit to `alpha = 6` jumped from `7/64 to 35/64` (`+43.75` points), with answers cut at the limit falling from `57` to `5`. |
| - 33rd and 34th forward passes (`fwd_50` and `fwd_52`): the frozen edit raised 128 fresh practice problems from `9 to 74` correct (`+50.78` points). Later passes tested more fresh problems. On the 256 fresh problems of `final_1`, it went from `10.16%` to `58.98%` (`+48.83 +/- 3.23` points, `127` rescued, `2` lost). Across `704` checked practice items it went from `7.95%` to `57.53%` (line 244), and across the final `1,152` items from `10.59%` to `60.42%` (line 374). |
|
|
| Our `all1` (`+16.6 +/- 2.0`) and `all2` (`+37.3 +/- 2.5`) apply the released direction `u` to all 3,584 columns, so they are close copies of the agent's 10th and 19th passes (`fwd_13` and `fwd_27`) rather than exact ones (the same `u`, but tested on different problems). Our norm-matched controls `randdir` (`+2.3 +/- 1.6`) and `randcols` (`+2.5 +/- 1.6`) match the agent's 23rd pass (`fwd_35`, `+1.56 +/- 3.52`). |
|
|
| ### 4. How every answer ended (`results/lab4/endings.json` and `endings-controls.json`) |
|
|
| `labs/lab4_endings.py` sorts the 512 answers in a run into four mutually exclusive groups by whether and how the model closed its `<think>` block: |
|
|
| 1. **Cut while thinking:** hit the 32,768-token limit before writing `</think>`. (A cut answer still counts as correct if its last `\boxed{}` holds the right answer, even inside its scratchpad; if it wrote no `\boxed{}`, the grader checks its last `\fbox{}`.) |
| 2. **Stopped while thinking:** emitted the end-of-sequence token before writing `</think>`, leaving no final answer. |
| 3. **Closed out of a loop:** wrote `</think>` right after verbatim repetition (the 60-character snippet ending 100 characters before `</think>` appears at least 3 times in the 6,000 characters before `</think>`). |
| 4. **Closed normally:** wrote `</think>` and the group 3 check found no repeated snippet. That check inspects only the last 6,000 characters before `</think>`, so an answer that recovered from a loop more than 6,000 characters before closing, or wrote `</think>` before its 6,000th character, still counts as "closed normally". |
|
|
| Every cell below lists `answers (correct answers)`: |
|
|
| | Run | Total correct (of 512) | Cut while thinking | Stopped while thinking | Closed out of a loop | Closed normally | |
| |---|---|---|---|---|---| |
| | `dev/base` (`alpha = 0`) | `62` | `453 (4)` | `0 (0)` | `3 (2)` | `56 (56)` | |
| | `dev/a-4` (`alpha = -4`) | `101` | `255 (4)` | `47 (0)` | `197 (87)` | `13 (10)` | |
| | `dev/a1` (`alpha = 1`) | `116` | `394 (3)` | `1 (0)` | `31 (29)` | `86 (84)` | |
| | `dev/a2` (`alpha = 2`) | `180` | `306 (4)` | `0 (0)` | `70 (44)` | `136 (132)` | |
| | `dev/a4` (`alpha = 4`) | `303` | `83 (1)` | `1 (0)` | `172 (70)` | `256 (232)` | |
| | `dev/p50c` (`alpha = 6`) | `315` | `72 (0)` | `0 (0)` | `96 (29)` | `344 (286)` | |
| | `dev/a8` (`alpha = 8`) | `311` | `51 (0)` | `28 (0)` | `69 (16)` | `364 (295)` | |
| | `dev/a12` (`alpha = 12`) | `265` | `43 (0)` | `37 (0)` | `19 (3)` | `413 (262)` | |
| | `dev/l15only` | `309` | `81 (1)` | `0 (0)` | `136 (47)` | `295 (261)` | |
| | `dev/l19only` | `240` | `196 (2)` | `0 (0)` | `151 (82)` | `165 (156)` | |
| | `dev/all1` | `147` | `362 (6)` | `0 (0)` | `30 (25)` | `120 (116)` | |
| | `dev/all2` | `253` | `200 (1)` | `0 (0)` | `93 (49)` | `219 (203)` | |
| | `dev/randdir` | `74` | `440 (10)` | `0 (0)` | `14 (7)` | `58 (57)` | |
| | `dev/randcols` | `75` | `443 (7)` | `0 (0)` | `6 (5)` | `63 (63)` | |
|
|
| ### 5. Median length of correct answers and breakdown by problem source |
|
|
| `dev256` is half GSM8K (grade-school word problems, 128 problems, 256 answers) and half MATH (competition math, 128 problems, 256 answers). `results/summary.json` breaks accuracy down by half and records the median token length of correct answers: |
|
|
| | Run | GSM8K accuracy % / cut % | MATH accuracy % / cut % | Median tokens of correct answers | |
| |---|---|---|---| |
| | `dev/base` (`alpha = 0`) | `15.2 / 86.3` | `9.0 / 91.0` | `5,600.5` | |
| | `dev/a-4` (`alpha = -4`) | `23.0 / 55.9` | `16.4 / 75.8` | `12,987` | |
| | `dev/p50c` (`alpha = 6`) | `74.6 / 4.7` | `48.4 / 34.8` | `7,998` | |
| | `dev/a12` (`alpha = 12`) | `55.5 / 0.8` | `48.0 / 23.8` | `5,799` | |
|
|
| In the unedited model, only short derivations finish before a loop starts, giving a median correct length of `5,600` tokens (`5,600.5`). With `p50c`, longer derivations can finish without looping, raising the median to `7,998` tokens. At `alpha = -4`, most correct answers (`87` of `101`) close right after repeated text. The median length of all correct answers is `12,987` tokens; the endings check does not measure loop length. |
|
|
| ## Why it works |
|
|
| ### 1. The dose trade-off along `u`: why accuracy plateaus and then drops |
|
|
| In [Lab 0](lab0_toy.md), `W' = W - alpha * u (u^T W_S)` scales the chosen 64 neurons' push along `u` by `1 - alpha`. Sweeping `alpha` from `0` to `12` takes that multiplier from `+1` (no edit) through `0` (erase at `alpha = 1`) and `-1` (mirror at `alpha = 2`) to `-5` (`alpha = 6`) and `-11` (`alpha = 12`). |
|
|
| ```mermaid |
| xychart-beta |
| title "Accuracy (%) on dev256 across edit strength alpha" |
| x-axis "alpha" ["-4", "0", "1", "2", "4", "6", "8", "12"] |
| y-axis "Accuracy (%)" 0 --> 70 |
| line [19.7, 12.1, 22.7, 35.2, 59.2, 61.5, 60.7, 51.8] |
| ``` |
|
|
| `alpha = 1` (erasing the 64 neurons' push along `u`) is not enough. In Lab 3, those 64 neurons held about half the positive loop score in layers 15 and 19 (`49.8%` and `55.5%`), while the other 22 blocks and the attention and DeltaNet mixers also pushed toward `u`. Setting `alpha = 1` zeroes out those 64 neurons' contribution (`22.7%` accuracy) and leaves the rest of the loop drift intact. Setting `alpha > 1` turns them into a counter-push: when they fire, they subtract `(alpha - 1)` times their original loop push, cancelling drift from other neurons and layers. |
|
|
| Pushing `alpha` too high shows two signs of harm to reasoning; the first is clean and the second is partly muddied by which problems finish: |
|
|
| 1. **Premature stop tokens during thinking:** From `alpha = 0` to `alpha = 6`, at most `1` of 512 answers emits an end-of-sequence token inside `<think>`. At `alpha = 8`, `28` answers stop mid-thinking with no final answer; at `alpha = 12`, `37` do. |
| 2. **Lower accuracy among finished answers, and higher perplexity:** Among answers that finish before the cap, accuracy drops as `alpha` grows: `100.0%` (`alpha = 0`), `96.6%` (`alpha = 1`), `92.1%` (`alpha = 2`), `82.9%` (`alpha = 4`), `76.6%` (`alpha = 6`), `70.7%` (`alpha = 8`), and `59.0%` (`alpha = 12`). Among normal `<think>` closes without a detected loop, accuracy falls from `286 / 344` (`83.1%`) at `alpha = 6` to `262 / 413` (`63.4%`) at `alpha = 12`. These finished-answer rates mix reasoning harm with selection: at `alpha = 0` few answers finish, and they may come from easier problems. Higher `alpha` lets harder problems finish too, and some of them are wrong. The `alpha = 6` vs `alpha = 12` normal-close gap has the same issue because the two runs finish different problems. The cleaner signs of harm are the mid-thinking stops in item 1 and WikiText-2 perplexity (Lab 7): small edits (`alpha = 1, 2, 4`) slightly lower perplexity (`-0.49%`, `-0.69%`, `-0.47%`), `alpha = 6` adds `+0.79%`, `alpha = 8` adds `+3.33%`, and `alpha = 12` jumps to `+14.34%`. |
|
|
| Across `alpha = 4` (`59.2%`), `alpha = 6` (`61.5%`), and `alpha = 8` (`60.7%`), fewer cut answers and lower finished-answer accuracy roughly balance out. The released `alpha = 6` sits on that plateau: paired by problem, it leads `alpha = 4` by `+2.3 +/- 1.6` and `alpha = 8` by `+0.8 +/- 1.6` points. Mid-thinking stops and perplexity rise at `alpha = 8` and `12`. |
|
|
| ### 2. Why both the direction `u` and the 64 neurons matter |
|
|
| Changing 64 columns by `10.6%` and `8.1%` of the matrix norm could in principle work by adding weight noise that breaks repetition. The controls `randdir` and `randcols` test this with `"norm_match": true` in `wsp/edits.py`, which rescales `alpha` so the Frobenius norm $\|W' - W\|_F$ matches `p50c` at both layers: |
| |
| - `randdir` edits the same 64 neurons per layer by the same matrix norm along a random unit vector orthogonal to `u`. Accuracy is `14.5 +/- 1.8%` (`+2.3 +/- 1.6` points over `12.1%`, `22` rescued, `18` lost). |
| - `randcols` applies `u` with the same matrix norm to 64 random neurons outside the chosen 64. Accuracy is `14.6 +/- 1.9%` (`+2.5 +/- 1.6` points, `23` rescued, `20` lost). |
| |
| Both controls sit within noise of the unedited model, with `87.3%` and `86.7%` of answers still cut at the cap. Among our 64-neuron edits, the gain appeared only when `u` went into neurons chosen because they fire during loops. Two caveats apply: each control is a single random draw (one random direction, one random set of 64 neurons), and the neurons do not have to be the exact released 64. In [Lab 3](lab3_rebuild.md), rebuilding from only 2 of the agent's 4 text pairs picked neurons that shared `42` and `31` of the released 64 neurons at layers 15 and 19 and still scored `62.3%`. In [Lab 8](lab8_capstone.md), recipe B picked neurons from real loops, shared `42` and `28` of the 64, and scored `58.2%`. |
| |
| Comparing `all1` and `all2` with the 64-neuron edits shows what the 64-neuron restriction does. Editing all 3,584 neurons at `alpha = 1` (`all1`, `28.7%`) and `alpha = 2` (`all2`, `49.4%`) beats the 64-neuron edits at the same `alpha` (`a1` at `22.7%` and `a2` at `35.2%`), which suggests the other 3,520 neurons also push along `u`. In Lab 7, KL divergence measures how far the edited next-token probabilities move from the base model's. At `alpha = 2`, `all2` shifts predictions on productive texts about `3.5` times as much as `a2` (KL `0.0063` vs `0.0018`). It is also less selective: its KL ratio between looping and productive texts is `1.45` (`0.0092` vs `0.0063`), compared with `2.49` for `p50c` (`0.0241` vs `0.0097`). We did not run the all-neuron edit above `alpha = 2`, so we do not know its peak accuracy, only that the 64-neuron edit concentrates more of its change on looping text. |
| |
| ### 3. Two likely mechanisms: fewer loops and more escapes (and layer 15 vs layer 19) |
| |
| Compare `dev/base` and `dev/p50c` in the endings table: |
| |
| ```mermaid |
| flowchart LR |
| Start["512 answers on dev256"] --> Base["Unedited (dev/base):<br/>453 cut while thinking<br/>3 closed out of a loop (2 correct)<br/>56 closed normally (56 correct)"] |
| Start --> Edit["Edited alpha = 6 (dev/p50c):<br/>72 cut while thinking<br/>96 closed out of a loop (29 correct)<br/>344 closed normally (286 correct)"] |
| ``` |
| |
| The endings suggest the edit helps in two ways, but do not prove it, because the check inspects only the text right before `</think>`: |
| |
| 1. **Fewer answers end their thinking in a loop:** Normal `<think>` closes jump more than sixfold, from `56` in `dev/base` to `344` in `dev/p50c`. Normal closes are usually right: `56 / 56` (`100%`) in `dev/base` and `286 / 344` (`83.1%`) in `dev/p50c`. The simplest reading is that fewer loops start. |
| 2. **More answers escape from a loop:** Answers that repeat verbatim and then emit `</think>` ("closed out of a loop") rise from `3` in `dev/base` to `172` at `alpha = 4` and `96` at `alpha = 6`. Loop escapes are noisier than normal closes: in `dev/p50c`, `29 of 96` (`30.2%`) are correct, adding `29` to the `286` normal-close successes (`286 + 29 = 315`). |
| |
| Comparing `dev/l15only` and `dev/l19only` in the endings table: |
| |
| - **Layer 15 alone mostly adds normal closes:** `dev/l15only` scores `60.4%` (`309` correct, near `61.5%` for both layers). Normal closes rise from `56` to `295` (`261` correct), cut answers fall from `453` to `81`, and `136` answers close out of a loop (`47` correct). |
| - **Layer 19 alone leaves more answers cut and gets more closes from loops:** `dev/l19only` scores `46.9%` (`240` correct), with `196` cut answers, `165` normal closes (`156` correct), and `151` closes out of a loop (`82` correct). |
| |
| [Lab 6](lab6_steering.md) offers a hint, though not a proof, for why the layers differ. Across 48 real loops, it measures the trigger `s(t)` (how hard the 64 chosen neurons push along `u` on each token). Relative to its in-loop level, layer 15's trigger builds earlier: in the last 512 tokens before the loop it reaches `42%` of its in-loop value (`0.178` of `0.424`), compared with `32%` at layer 19 (`0.241` of `0.758`). Inside the loop, layer 19's trigger is larger (`0.758` vs `0.424`). This fits layer 15 acting earlier and layer 19 pushing harder once a loop starts, though we did not test that directly. |
| |
| ### 4. The `alpha = -4` surprise: why you must read the answers |
| |
| At `alpha = -4` (`dev/a-4`), the edit multiplies the 64 neurons' push along `u` by `1 - (-4) = +5`, pushing 5 times harder in the loop direction. On headline accuracy alone, `dev/a-4` scores `19.7 +/- 1.9%`, or `+7.6 +/- 2.2` points above the unedited model (`12.1%`). |
| |
| The endings table and `labs/peek.py` show what happened: |
| |
| ```mermaid |
| flowchart TD |
| Neg["alpha = -4 (dev/a-4):<br/>pushes 5x harder toward loops"] --> Collapse["Normal closes collapse:<br/>56 to 13 answers (10 correct)<br/>24 previously solved problems lost"] |
| Neg --> Stop["Mid-thinking stops spike:<br/>0 to 47 answers end with no answer"] |
| Neg --> Crash["Loops intensify and crash out of thinking:<br/>3 to 197 answers close out of a loop<br/>(87 of the 101 correct answers)"] |
| ``` |
| |
| - **Normal reasoning collapses:** Normal `<think>` closes drop from `56` in `dev/base` to `13` in `dev/a-4` (`10` correct). `dev/a-4` loses `24` problems the base model solved (`54` rescued, `24` lost, while `a4`, `p50c`, `a8`, and `a12` lose `0`). |
| - **Mid-thinking stops jump to 47:** `47` answers emit the end token inside `<think>` with no final answer, and WikiText perplexity worsens by `+5.39%`. |
| - **Most correct answers close their thinking right after repeated text:** `197` answers close this way; of `101` correct answers in `dev/a-4`, `87` came from `closed out of a loop`. |
| |
| One problem shows the pattern (`gsm8k_train_5628`: *"Mr. Desmond bought three times as many toys for his younger son as he bought for his elder son. If the elder son received 60 toys, how many toys did Mr Desmond buy?"*, truth `240`): |
| |
| - **Unedited (`quick/base`, answer 0):** Computes `60 + 180 = 240` early, then enters an 83-token verbatim loop at token `8,568` of `32,768` (*"Let's check if the answer is 240. / Let's check if the answer is 180..."*) that repeats about `292` times to the cap. |
| - **Released edit (`quick/p50c`, answer 0):** Finishes in `5,825` tokens, closes `</think>`, and writes `\boxed{240}`. |
| - **Alpha -4 (`dev/a-4`, answer 0, on `sampler`):** Run `python labs/peek.py dev/a-4 --id gsm8k_train_5628` on `sampler`. The model repeats *"Wait, is it possible $A$ and $C$ is the same?"* over and over, breaks off mid-line with a stray backtick and `</think>`, and writes `240` (`4,830` tokens total). |
| |
| Pushing harder in the loop direction cut normal closes from `56` to `13`, lost `24` previously solved problems, and dropped finished-answer accuracy to `55.4%`. Yet `87` correct answers closed their thinking right after repeated text, and total accuracy rose by `+7.6` points. |
| |
| ## Check yourself |
| |
| 1. Look at the dose table between `alpha = 6` (`61.5%` accuracy, `19.7%` cut) and `alpha = 12` (`51.8%` accuracy, `12.3%` cut). Fewer answers are cut at `alpha = 12`, so why is accuracy `9.7` percentage points lower? |
| <details><summary>Answer</summary> |
| |
| Two things go wrong at `alpha = 12`. First, `37` answers emit an end token inside `<think>` (`stopped while thinking`), against `0` at `alpha = 6`, producing no final answer. Second, the `-11` multiplier likely hurts reasoning: WikiText-2 perplexity rises `+14.34%`, and finished-answer accuracy falls from `76.6%` at `alpha = 6` to `59.0%` at `alpha = 12` (normal closes fall from `286 / 344 = 83.1%` to `262 / 413 = 63.4%`), though selection may contribute because the two runs finish different problems. |
| </details> |
| |
| 2. Why do `randdir` and `randcols` both use `"norm_match": true` in `configs/variants.json`? What would happen if `randcols` used raw `alpha = 6` without `"norm_match": true`? |
| <details><summary>Answer</summary> |
| |
| In Lab 1, the 64 chosen neurons aligned with `u` much more strongly than typical `down_proj` columns (median `|cos(u, column)|` is `0.095` on the 64 columns in layer 15 vs `0.018` on the rest). Without rescaling, plugging 64 random columns into `W' = W - 6 u (u^T W_S)` gives a Frobenius norm for `W' - W` several times smaller than the released edit's, so the control could fail from a smaller weight change alone. Setting `"norm_match": true` multiplies `alpha` by `||u_rel^T W[:, cols_rel]|| / ||u^T W[:, cols]||` so `W' - W` matches `p50c` in Frobenius norm. |
| </details> |
|
|
| 3. Compare `dev/l15only` (`60.4%` accuracy, `+0.05%` perplexity change) and `dev/l19only` (`46.9%` accuracy, `+1.65%` perplexity change). Which single layer carries most of the benefit on `dev256`, and how do their answer endings differ? |
| <details><summary>Answer</summary> |
|
|
| Layer 15 alone (`60.4%`) nearly matches editing both layers (`61.5%`) with almost no WikiText perplexity change (`+0.05%`). In the endings table, `dev/l15only` mostly adds normal closes (`295` normal, `81` cut while thinking), while `dev/l19only` leaves `196` cut, gets `165` normal closes, and closes out of a loop more often (`151`). |
| </details> |
|
|
| 4. In `dev/a-4`, `101` of `512` answers are correct (`19.7%`), beating `dev/base` (`62` of `512`, `12.1%`). Give three numbers from the tables that show `dev/a-4` damages the model instead of fixing the loop bug cleanly. |
| <details><summary>Answer</summary> |
|
|
| Any three of these confirm the damage: |
| 1. Normal closes fall from `56` (`56` correct) in `dev/base` to `13` (`10` correct) in `dev/a-4`. |
| 2. `dev/a-4` loses `24` problems that the unedited model solved (`54 / 24` rescued/lost), whereas `dev/p50c` loses `0` (`125 / 0`). |
| 3. `47` answers stop dead in the middle of thinking (`stopped while thinking`), compared with `0` in `dev/base` and `0` in `dev/p50c`. |
| 4. Accuracy among finished answers drops from `100.0%` in `dev/base` (`76.6%` in `dev/p50c`) to `55.4%`, and `87` of the `101` correct answers come from crashing out of a verbatim loop (`closed out of a loop`). |
| 5. WikiText-2 perplexity rises by `+5.39%` (compared with `+0.79%` for `p50c`). |
| </details> |
|
|
| ## Try this next |
|
|
| We did not run the experiments below. Write down your prediction first, then test on a 16-problem slice (`--limit 16`, about 3 to 5 minutes on 1 GPU) before running all 256 problems. |
|
|
| 1. **Test `alpha = 3` or `alpha = 5` on the dose curve (not run).** Between `alpha = 2` (`35.2%`) and `alpha = 4` (`59.2%`), accuracy jumps `24.0` points. From the project root on this machine, add `"a3"` or `"a5"` to a new file `configs/try_dose.json`: |
| ```json |
| { |
| "a3": {"about": "flip and double the loop component (1 - alpha = -2)", |
| "layers": {"15": {"alpha": 3}, "19": {"alpha": 3}}}, |
| "a5": {"about": "between a4 and p50c (1 - alpha = -4)", |
| "layers": {"15": {"alpha": 5}, "19": {"alpha": 5}}} |
| } |
| ``` |
| On this machine, create `plans/try_dose_quick.json` in the format of `plans/quick.json` (`--limit 16` runs the first 16 GSM8K problems of `dev256`, 2 answers each = 32 answers): |
| ```json |
| { |
| "about": "Quick 16-problem check of alpha 3 and alpha 5", |
| "runs": [ |
| {"name": "quick/a3", "model": "qwen35-a3", "data": "dev256", "n": 2, "shards": 1, "est": 1, "args": ["--limit", "16"]}, |
| {"name": "quick/a5", "model": "qwen35-a5", "data": "dev256", "n": 2, "shards": 1, "est": 1, "args": ["--limit", "16"]} |
| ] |
| } |
| ``` |
| Sync both files: |
| ```bash |
| bash scripts/sync_vm.sh sampler |
| ``` |
| Pick a free GPU with `nvidia-smi`. Connect to the VM, activate the environment, build both models on CPU, and run the plan (replace `7` with your free GPU): |
| ```bash |
| ssh sampler |
| . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code |
| CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/try_dose.json |
| python scripts/run_jobs.py plans/try_dose_quick.json --gpus 7 |
| ``` |
| **What to predict:** In the agent's 20th pass (`fwd_29`), raising `alpha = 2` to `alpha = 3` on all columns added about 5 points (`+34.77` to `+39.84`), about a third of the way to the roughly `+50` points its 64-neuron edit later reached. Our 64-neuron curve climbs faster, from `a2` at `35.2%` to `a4` at `59.2%`, so a fair guess for `a3` on full `dev256` is near the middle, around `45% to 50%`. Predict `a5` on full `dev256` to land around `60% to 61%` (within noise of `a4` at `59.2%` and `p50c` at `61.5%`). On these `16` problems alone, our full-run answers score `a2` at `37.5%` (`12/32` correct) and `a4`, `p50c`, and `a8` at `65.6%` (`21/32` each). With 16 problems, the standard error is roughly 9 points or more, so the quick check gives only a rough placement on the curve. |
|
|
| 2. **Edit only a subset of the 64 neurons at `alpha = 6` (not run).** In `wsp/edits.py`, `build_from_spec` accepts a list of integer column indices for `"cols"`. In `results/lab3/rebuild.json`, each layer stores `"cols"` (all 64 chosen neurons) and `"top12"` (the 12 highest-scoring neurons by loop score). From the project root on this machine, write a variant file with 32 of the 64 neurons (`"cols"[:32]`, the 32 lowest-numbered of the 64, since `cols` is sorted by index, not score) or with the top 12 (`"top12"`): |
| ```bash |
| python3 -c ' |
| import json |
| r = json.load(open("results/lab3/rebuild.json"))["layers"] |
| spec = { |
| "halfcols-a6": {"about": "alpha 6 on 32 of the 64 neurons per layer", |
| "layers": {L: {"alpha": 6, "cols": r[L]["cols"][:32]} for L in ("15", "19")}}, |
| "top12-a6": {"about": "alpha 6 on the top 12 loop-score neurons per layer", |
| "layers": {L: {"alpha": 6, "cols": r[L]["top12"]} for L in ("15", "19")}} |
| } |
| json.dump(spec, open("configs/try_subset.json", "w"), indent=1) |
| ' |
| ``` |
| Copy the quick plan from Try 1 to `plans/try_subset_quick.json`, replacing the two run names with `quick/halfcols-a6` and `quick/top12-a6` and the model names with `qwen35-halfcols-a6` and `qwen35-top12-a6`. Keep `"args": ["--limit", "16"]` and sync: |
| ```bash |
| bash scripts/sync_vm.sh sampler |
| ``` |
| Build on CPU and test on a free GPU (replace `7` as needed): |
| ```bash |
| ssh sampler |
| . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code |
| CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/try_subset.json |
| python scripts/run_jobs.py plans/try_subset_quick.json --gpus 7 |
| ``` |
| **What to predict:** We did not measure the counter-push of each subset, so use Lab 3's loop scores as a rough guide. The 32 lowest-numbered columns are not an even half by score. At layer 19 they include the two highest scorers, neurons 1 and 11. Neuron 11 alone holds `24%` of layer 19's positive score, so this half probably carries well over half of layer 19's push. At layer 15 no neuron dominates (the top holds `5.6%`). If that half carried exactly half the push, `alpha = 6` on 32 columns would act like `alpha = 3` or `4` on all 64. Predict roughly `45% to 55%` on full `dev256` with under `+0.79%` WikiText perplexity change, and somewhat higher if layer 19's half carries more. The top 12 neurons hold more positive score than the top 8 (`19.3%` at layer 15, `39.7%` at layer 19) and less than the top 32 (`38.9%`, `49.8%`), compared with `49.8%` and `55.5%` for all 64. That share is large, especially at layer 19, so `top12-a6` could land anywhere from near `a2` (`35.2%`) to near `a4` (`59.2%`) on full `dev256`. The `16`-problem slice is not the full-set comparison: our full-run answers on these problems score `a2` at `37.5%` and `a4`, `p50c`, and `a8` at `65.6%`. |
|
|
