Publish the weight-space programming labs

.gitattributes

@@ -33,3 +33,24 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+code/results/figures/lab4_dose.png filter=lfs diff=lfs merge=lfs -text
+code/results/figures/lab6_loops_summary.png filter=lfs diff=lfs merge=lfs -text
+code/results/figures/lab6_real_loop.png filter=lfs diff=lfs merge=lfs -text
+code/results/figures/lab8_data_needed.png filter=lfs diff=lfs merge=lfs -text
+code/results/figures/lab8_loop_onset.png filter=lfs diff=lfs merge=lfs -text
+code/results/figures/lab8_one_answer.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab1/try-qwen35-all2/lab1_diff.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab3/try-layers15-19-21/lab3_neuron_scores.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab6/try-max-samples-10/lab6_loops_summary.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab6/try-max-samples-10/lab6_real_loop.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab6/try-smooth-16/lab6_loops_summary.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab6/try-smooth-16/lab6_real_loop.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-alpha-8/lab8_data_needed.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-alpha-8/lab8_loop_onset.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-alpha-8/lab8_one_answer.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-max-samples-16/lab8_data_needed.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-max-samples-16/lab8_loop_onset.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-max-samples-16/lab8_one_answer.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-split-by-problem/lab8_data_needed.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-split-by-problem/lab8_loop_onset.png filter=lfs diff=lfs merge=lfs -text
+code/results/lab8/try-split-by-problem/lab8_one_answer.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,10 +1,35 @@
 ---
-title: Weight Space Programming
-emoji: 😻
-colorFrom: blue
-colorTo: blue
+title: Weight-Space Programming Lab
+emoji: 🧮
+colorFrom: gray
+colorTo: indigo
 sdk: static
+app_file: index.html
 pinned: false
+short_description: "Hands-on labs: fix a small LLM by editing its weights."
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Weight-space programming lab
+
+This Space holds nine hands-on labs that reproduce and probe Experiment 8 of "Sparks of Weight-Space Programming". In that experiment, an AI agent fixed the looping of Qwen3.5-0.8B. It edited 131,072 weights with arithmetic and did no training.
+
+- Read the labs on the [live site](https://thegovind-weight-space-programming.static.hf.space/index.html).
+- Start with the [Quick start](https://thegovind-weight-space-programming.static.hf.space/quickstart.html) page.
+- Browse the code in the [code/ folder](https://huggingface.co/spaces/thegovind/weight-space-programming/tree/main/code/).
+
+## Get the code
+
+```bash
+pip install -U huggingface_hub
+hf download thegovind/weight-space-programming --repo-type space \
+  --include "code/*" --local-dir ~/weight-space-programming
+cd ~/weight-space-programming/code
+```
+
+## Credits and licenses
+
+- The project reproduces Experiment 8 of the blog post ["Sparks of Weight-Space Programming"](https://rikiyat.github.io/blog/weight-space-programming/) by the authors of "LLMs can understand and write neural network weights". It uses their [public logs](https://github.com/RikiyaT/lm_understand-logs).
+- The agent's 8 texts in `code/labs/data/astra_pairs.json` and the quoted log excerpts come from [RikiyaT/lm_understand-logs](https://github.com/RikiyaT/lm_understand-logs). They are copyright the authors of "LLMs can understand and write neural network weights" and licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). We extracted them from the logs. The selection and the JSON layout are ours. The labs, measurements and figures come from our own runs.
+- `code/wsp/grading.py` includes answer-extraction and grading code copied from the paper's harness (`exp8_math/code/mathqa.py`). It is used under the MIT License. See [code/NOTICE.md](https://huggingface.co/spaces/thegovind/weight-space-programming/blob/main/code/NOTICE.md).
+- Benchmarks: MATH-500 (from MATH, MIT License), GSM8K (MIT License) and IFEval (Apache License 2.0). The benchmark files are not included. `code/scripts/prepare_data.py` downloads them again from their public copies on Hugging Face. See [code/data/README.md](https://huggingface.co/spaces/thegovind/weight-space-programming/blob/main/code/data/README.md).
+- No license is granted yet for this project's own code and text.

code/.gitignore

@@ -0,0 +1,11 @@
+# The paper's logs: clone them yourself (see the README).
+external/
+# Full answer texts stay on the GPU machines.
+*.texts.jsonl.gz
+__pycache__/
+*.pyc
+.venv/
+# The site build output, its npm packages and the diagram renderer's scratch folder.
+_site/
+site/node_modules/
+site/.render-tmp/

code/NOTICE.md

@@ -0,0 +1,21 @@
+# Notice
+
+`wsp/grading.py` includes answer-extraction and grading code copied from the paper's harness
+(`exp8_math/code/mathqa.py` in [RikiyaT/lm_understand-logs](https://github.com/RikiyaT/lm_understand-logs)), used under the MIT License:
+
+MIT License
+
+Copyright (c) 2026 the authors of "LLMs can understand and write neural network weights"
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the Software without restriction, including without limitation the
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit
+persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

code/QUICKSTART.md

@@ -0,0 +1,87 @@
+# Quick start: a 2-hour path through the labs
+
+**Big thing:** Run the core weight-space programming checks in 2 hours (about 10 minutes of GPU compute), from a 5-second toy matrix on this machine to the full benchmark, dose, decoding, and capstone results.
+
+**Get the code first.** This site is a public copy of the project. Download the code to your own machine:
+
+```bash
+pip install -U huggingface_hub
+hf download thegovind/weight-space-programming --repo-type space \
+  --include "code/*" --local-dir ~/weight-space-programming
+cd ~/weight-space-programming/code
+```
+
+Commands for this machine run from that `code` folder. Commands on the GPU machines run from `/nvme/wsp-lab/code`, where the setup copies the code.
+
+**Machines.** Step 1 runs on any laptop. The other steps ran on two GPU machines, each with 8 NVIDIA A100 80 GB GPUs, and most steps use only 1 or 2 of them. These pages call the two machines `trainer` and `sampler`, so use your own host names wherever you see those. To set up a machine, follow "Cloning the paper's logs" and then "Setting up a new machine", both under [Running things](README.md#running-things). The setup puts everything under `/nvme/wsp-lab` (about 15 GB) and needs no root access.
+
+**Data.** The problem files in `data/` are not included. [`data/README.md`](data/README.md) explains why and shows how to rebuild them.
+
+> **Before you start (VM access and environment):**
+> 1. **SSH hosts:** Connect with `ssh trainer` or `ssh sampler` (8x A100 80 GB each).
+> 2. **Environment line (run first in every VM shell):** `. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code`
+> 3. **Which terminal:** Steps 1 and 7 run on this machine. Step 2 runs on `trainer` and ends with `exit`. Steps 3 to 6 share one `sampler` terminal, and Step 6 ends with `exit`. The two VMs cannot reach each other, so always `ssh` from this machine.
+
+### Step 1: Run the toy edit on a 4 x 6 matrix (15 minutes)
+- **Where:** This machine (CPU, 5 seconds to run, 15 minutes with the sheet).
+```bash
+cd ~/weight-space-programming/code
+uv run --with numpy --with matplotlib python labs/lab0_toy.py
+```
+- **What to look at:** Check the 4 numbered outputs (`singular values of W' - W: [3.84 0. 0. 0.]` and `|W'a - Wa| = 0.000` when the chosen neurons are quiet vs `4.261` when firing) and open `results/figures/lab0_toy.png`. Singular values measure the sizes of independent pieces of a matrix change; only one is nonzero here. Read [`labs/lab0_toy.md`](labs/lab0_toy.md) for why the edit is rank one (one column vector times one row vector) and acts like steering (adding a vector to the model's internal state).
+
+### Step 2: Run the 5-minute quick test and inspect a looping answer (15 minutes)
+- **Where:** `trainer` (2 GPUs for 5 minutes, then CPU for `peek.py`). If our finished answers for `quick/base` (the unedited model) and `quick/p50c` (the released edit) are on the machine, the run skips them. To have vLLM (the program that generates the answers) write fresh ones, move `/nvme/wsp-lab/runs/quick` aside first.
+```bash
+ssh trainer
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+nvidia-smi   # check which GPUs are free
+mv /nvme/wsp-lab/runs/quick /nvme/wsp-lab/runs/quick-ours   # optional: only if generating fresh answers
+python scripts/run_jobs.py plans/quick.json --gpus 0,1
+CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/base
+CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/p50c
+CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/base --id gsm8k_train_5628
+CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/p50c --id gsm8k_train_5628
+exit   # back to this machine
+```
+- **What to look at:** On our run (`quick-ours` if you moved it), `quick/base` gets 6 of 32 right (`18.8%`) with 27 of 32 cut in a verbatim loop, while `quick/p50c` gets 21 of 32 (`65.6%`, `+46.9` points) with 3 cut and 0 loops. On `gsm8k_train_5628` (truth `240`), `quick/base` repeats an 83-token block about `292` times from token 8,568 to 32,768, while `quick/p50c` finishes in `5,825` tokens after closing `</think>` and writing `\boxed{240}`. See [`labs/lab2_reproduce.md`](labs/lab2_reproduce.md).
+
+### Step 3: Find the edit in the weight files on CPU (15 minutes)
+- **Where:** `sampler`, from this machine (CPU only, under 1 minute). Keep this terminal open for Steps 4 to 6.
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+python labs/lab1_diff.py
+```
+- **What to look at:** Confirm that only 2 of 488 tensors (arrays of weights) differ (`layers.15` and `layers.19` `mlp.down_proj.weight`), all changes sit in 64 columns per layer, the first singular value holds `0.999994` and `0.999991` of the squared change, and the fitted $\alpha$ (edit strength) is `6.0003` and `6.0002`. These paired numbers refer to the two layers in the order listed. Read [`labs/lab1_diff.md`](labs/lab1_diff.md) and inspect `results/figures/lab1_diff.png`.
+
+### Step 4: Rebuild the edit from the agent's 8 texts (20 minutes)
+- **Where:** the same `sampler` terminal (1 GPU, about 1 minute). Run this before Step 5 because Lab 6 reads which way $u$ (the direction from productive toward looping text in the model's internal state) points from `results/lab3/rebuild.json`.
+```bash
+CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py
+```
+- **What to look at:** Verify that blocks 15 and 19 have the largest MLP gap growth (`0.263` and `0.249`), the rebuilt directions match the released $u$ to cosine `+0.99998` and `-0.99998`, and the top 64 neuron scores match `64/64` released columns in both layers. Read [`labs/lab3_rebuild.md`](labs/lab3_rebuild.md) and check `results/figures/lab3_where_the_loop_grows.png` and `lab3_neuron_scores.png`.
+
+### Step 5: Verify that the weight edit is activation steering (15 minutes)
+- **Where:** the same `sampler` terminal (1 GPU, about 1 minute for parts 1 and 2). Run `--parts 3,4` (about 3 minutes) to trace 48 real loops and the 26 correct finished answers from `/nvme/wsp-lab/runs/build/base` on `sampler`.
+```bash
+CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py
+CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3,4   # optional
+```
+- **What to look at:** Editing `down_proj` vs subtracting $\alpha s(t) u$ in a hook (a function attached to a layer) matches to `3.33e-16` in float64 (`1.4e-8` mean KL in float32). Float64 and float32 are number formats with different precision; KL measures the change in next-token probabilities. Over the last 36 tokens of the agent's 8 texts, $s(t)$ (how hard the chosen neurons push toward looping on each token) is near zero on productive text (`0.00` to `0.04` at layer 15) and positive on looping text (`0.19` to `0.42`). Parts 3 and 4 show the same split on real answers: at layer 19, $s(t)$ averages `0.76` inside loops and about `0.1` on answers that went well (small, not zero). Read [`labs/lab6_steering.md`](labs/lab6_steering.md), which covers `results/figures/lab6_real_loop.png` and `lab6_finished_vs_loops.png`.
+
+### Step 6: Measure side effects on ordinary text and selective KL (15 minutes)
+- **Where:** the same `sampler` terminal (1 GPU, about 1.5 minutes). Then leave the VM.
+```bash
+CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py
+exit   # back to this machine
+```
+- **What to look at:** `p50c` and `rebuilt` raise WikiText-2 perplexity (prediction uncertainty on ordinary text, lower is better; base `20.009`) by `+0.79%` while shifting predictions 2.5 times more on looping texts than on productive ones (`0.0241` vs `0.0097`, measured by KL divergence, the change in next-token probabilities). By contrast, `a12` (a stronger edit) costs `+14.34%`, and `randdir`/`randcols` (random-direction and random-neuron controls) shift looping text no more than productive text (`randdir`: `0.0039` vs `0.0048`). Read [`labs/lab7_side_effects.md`](labs/lab7_side_effects.md) and view `results/figures/lab7_side_effects.png`.
+
+### Step 7: Read the finished results of Labs 2, 4, 5, and 8 (30 minutes)
+- **Where:** This machine (no GPU; [`results/summary.md`](results/summary.md) and `results/figures/` hold all runs).
+```bash
+cd ~/weight-space-programming/code
+ls results/figures/
+```
+- **What to look at:** Read [`labs/lab2_reproduce.md`](labs/lab2_reproduce.md) (`lab2_benchmarks.png`, `lab2_lengths.png`), [`labs/lab4_dose.md`](labs/lab4_dose.md) (`lab4_dose.png`, `lab4_controls.png`, including why $\alpha = -4$ gains `+7.6` points by breaking loops), [`labs/lab5_decoding.md`](labs/lab5_decoding.md) (`lab5_decoding.png`, where turning thinking off reaches `52.1%` at 2,270 tokens), and [`labs/lab8_capstone.md`](labs/lab8_capstone.md) (`lab8_loop_onset.png`, `lab8_data_needed.png`, `lab8_one_answer.png`, where recipe B built from 225 of the model's own loops reaches `58.2%` on `dev256`). To re-run any sheet's GPU jobs, use the move-aside rule in [`README.md`](README.md).

code/README.md

@@ -0,0 +1,155 @@
+# Weight-space programming labs
+
+**Big thing:** These nine hands-on labs reproduce and take apart Experiment 8 of the MIT CSAIL blog post "Sparks of Weight-Space Programming" on two 8-GPU A100 machines. In that experiment, an AI agent fixed a small language model (Qwen3.5-0.8B) that got stuck repeating itself on math problems by editing 131,072 of its 873,438,784 weights with arithmetic, without training.
+
+**Why it matters:** Most model fixes change the prompt or sampling settings, or run gradient descent (training that adjusts weights to reduce errors). Here you inspect the `131,072` edited weights (`130,188` change after bf16 rounding, which stores numbers at reduced precision), rebuild the edit from 8 short texts in one minute, prove that the weight edit equals an input-driven steering vector (a change to the model's internal numbers whose strength depends on the input), and build your own edit from the model's failures.
+
+## The story
+
+Weight-space programming changes what a trained neural network does by reading and editing its weight matrices with arithmetic instead of gradient training.
+
+In Experiment 8 (2 Nov 2026), an AI agent (GPT-6 Astra) got a CPU-only Python session with Qwen3.5-0.8B, an open model with about 0.87 billion weights across 24 repeated layers (blocks). It had practice math problems with answers and a budget of 50 forward passes. A forward pass runs inputs through the network to read internal activations (numbers computed as the input passes through each layer) or generate answers. One pass could answer up to 512 problems or read activations on up to 1,024 texts, and all training was forbidden.
+
+With thinking on, the model writes a `<think>...</think>` scratchpad before its final answer. In this mode, the unedited model kept falling into verbatim loops, repeating the same block of tokens (word pieces) over and over until it hit the 32,768-token cap: 85 of its first 96 practice answers hit that limit.
+
+The agent wrote 4 productive and 4 looping scratchpads and compared the model's activations on them. That contrast gave a unit direction vector $u$: a list of 1,024 numbers whose overall length is 1, pointing from productive text toward looping text. It then applied a rank-one projection edit (one column vector times one row vector) in blocks 15 and 19. Each block's MLP (multi-layer perceptron, the block's second half) has an output matrix `down_proj` of shape 1,024 by 3,584. Each column is the 1,024-number vector one neuron adds to the residual stream (the running list of numbers each token carries from block to block) per unit of activation. The agent picked the 64 neurons that push hardest along $u$ during loops and updated their columns $W_S$ via $W' = W - \alpha u (u^T W_S)$ with strength $\alpha = 6$. That edit touches $2 \times 64 \times 1,024 = 131,072$ weights (`p50c`, from the 50-pass run folder `runs/p50c`).
+
+On three unseen benchmarks, the paper reported that `p50c` raised MATH-500 accuracy from 7.9% to 48.1% (answers cut at the cap fell from 93.0% to 33.05%, mean tokens from 30,868 to 22,037), GSM8K from 13.84% to 67.1% (cut 87.5% to 7.24%, mean tokens 29,289 to 11,993), and IFEval instruction-following accuracy from 21.0% to 62.3% (cut 74.9% to 2.2%, mean tokens 25,296 to 4,241).
+
+## What we found
+
+- **The benchmark gains reproduce on our hardware** ([`labs/lab2_reproduce.md`](labs/lab2_reproduce.md), [`labs/lab1_diff.md`](labs/lab1_diff.md)). Diffing all 488 weight tensors (arrays of weights) confirms that only `layers.15.mlp.down_proj.weight` and `layers.19.mlp.down_proj.weight` changed, in 64 columns each, with fitted $\alpha = 6.0003$ and $6.0002$. On our GPUs, `p50c` raises MATH-500 from $7.3 \pm 0.9\%$ to $51.8 \pm 1.9\%$, GSM8K from $14.2 \pm 0.8\%$ to $66.9 \pm 1.2\%$, and IFEval from $21.2 \pm 1.4\%$ to $63.4 \pm 1.8\%$.
+- **The 5-minute quick run shows the same effect on 16 problems** ([`QUICKSTART.md`](QUICKSTART.md), [`labs/lab2_reproduce.md`](labs/lab2_reproduce.md)). On the first 16 GSM8K problems of our 256-problem practice test set (`dev256`), the unedited model scores $18.8 \pm 7.7\%$ and `p50c` scores $65.6 \pm 11.8\%$ ($+46.9 \pm 11.6$ percentage points). Answers cut at the 32,768-token cap fall from 27 of 32 (all ending in a verbatim loop) to 3 of 32 (0 in a loop).
+- **Rebuilding the edit from the agent's 8 texts recovers all 64 neurons in both layers** ([`labs/lab3_rebuild.md`](labs/lab3_rebuild.md)). Teacher-forcing the agent's 4 productive and 4 looping texts (feeding fixed text through the model and recording activations with hooks, small functions attached to layers) shows that the MLPs of blocks 15 and 19 widen the looping-minus-productive gap the most. The rebuilt edit recovers directions matching the released $u$ to cosine $+0.99998$ and $-0.99998$ (nearly the same line, with opposite orientation at the second layer), picks the same 64 neurons in both layers, and scores $61.5 \pm 2.8\%$ on `dev256` (matching `p50c`'s $61.5 \pm 2.8\%$). Using only the first 2 text pairs still reaches $62.3 \pm 2.7\%$.
+- **Accuracy rises with $\alpha$ up to a plateau from 4 to 8, while random controls move accuracy by about +2 points** ([`labs/lab4_dose.md`](labs/lab4_dose.md), [`labs/lab7_side_effects.md`](labs/lab7_side_effects.md)). On `dev256`, accuracy goes from $12.1\%$ ($\alpha = 0$, unedited) to $22.7\%$ ($\alpha = 1$, erase), $35.2\%$ ($\alpha = 2$, mirror), $59.2\%$ ($\alpha = 4$), $61.5\%$ ($\alpha = 6$), and $60.7\%$ ($\alpha = 8$). Accuracy then drops to $51.8\%$ at $\alpha = 12$, and WikiText-2 perplexity (prediction uncertainty on ordinary text, lower is better) rises by $+14.34\%$, versus $+0.79\%$ at $\alpha = 6$. Matched-norm controls with a random orthogonal direction (`randdir`) or 64 random neurons (`randcols`) score $14.5\%$ ($+2.3 \pm 1.6$ points) and $14.6\%$ ($+2.5 \pm 1.6$ points).
+- **Setting $\alpha = -4$ gains +7.6 points for the wrong reason** ([`labs/lab4_dose.md`](labs/lab4_dose.md)). Amplifying the 64 neurons' push toward the loop direction 5 times raises `dev256` accuracy from $12.1\%$ to $19.7 \pm 1.9\%$ ($+7.6 \pm 2.2$ points) because 197 of 512 answers abruptly emit `</think>` out of a loop. Yet it loses 24 problems the unedited model solved, drops accuracy among finished answers from $100.0\%$ to $55.4\%$, and raises WikiText-2 perplexity by $+5.39\%$.
+- **Turning thinking off reaches 52.1% using 13 times fewer tokens, and the weight edit adds nothing there** ([`labs/lab5_decoding.md`](labs/lab5_decoding.md)). With thinking on, setting temperature to 1.0 ($27.9\%$) or repetition penalty to 1.1 ($34.4\%$) falls well short of the weight edit ($61.5\%$), and dropping the 1.5 presence penalty cuts accuracy to $4.5\%$, with $99.0\%$ of answers cut. Without thinking, the base model reaches $52.1 \pm 2.7\%$ on `dev256` with 2,270 tokens per answer on average, versus 29,683 with thinking on (16,104 with the edit). Adding `p50c` with thinking off scores $53.5 \pm 2.8\%$ ($+1.4 \pm 1.9$ points), because answers rarely loop without thinking.
+- **The static weight edit is algebraically equal to token-by-token activation steering** ([`labs/lab6_steering.md`](labs/lab6_steering.md), [`labs/lab0_toy.md`](labs/lab0_toy.md)). Multiplying the edited matrix by neuron activations $a(t)$ gives $W' a(t) = W a(t) - \alpha s(t) u$, where $s(t) = u \cdot (W_S a_S(t))$ is one number per token measuring how hard the 64 chosen neurons push along $u$. In 64-bit floats on one layer, the matrix edit and a runtime steering hook match to $3.33 \times 10^{-16}$. On 48 real looping answers from the base model, $s(t)$ stays low early in the answer (0.096 at layer 15, 0.116 at layer 19) and jumps inside the loop (0.424 and 0.758), so the edit pushes back hardest inside loops.
+- **An edit built from the model's own loops reaches 58.2% without hand-written texts** ([`labs/lab8_capstone.md`](labs/lab8_capstone.md)). On 225 looping answers the unedited model wrote for 128 separate practice problems (`build128`), comparing loop tokens with tokens well before the loop (recipe B) matches $u$ to cosine 0.782 and 0.773 and scores $58.2 \pm 2.9\%$ on `dev256` ($+46.1 \pm 2.6$ points). Comparing with the 32 tokens just before the loop starts (recipe A) reaches only $43.0 \pm 2.6\%$ because pre-loop text has already drifted along $u$.
+
+## Start here
+
+Start with [**`QUICKSTART.md`**](QUICKSTART.md) for a one-page, 2-hour path through the labs with exact commands and expected numbers.
+
+The Quickstart also shows how to download the code from this site. Lab 0 runs on any laptop. The other labs need NVIDIA GPUs; we used two machines with 8 A100 80 GB GPUs each, and most steps use 1 or 2 of them.
+
+## The nine labs
+
+```mermaid
+flowchart LR
+  L0["Lab 0: Toy 4x6 matrix"] --> L1["Lab 1: Diff weight files"]
+  L1 --> L2["Lab 2: Reproduce benchmarks"]
+  L1 --> L3["Lab 3: Rebuild from 8 texts"]
+  L3 --> L4["Lab 4: Dose curve and controls"]
+  L2 --> L5["Lab 5: Decoding settings"]
+  L3 --> L6["Lab 6: Edit equals steering"]
+  L4 --> L7["Lab 7: Side effects on text"]
+  L6 --> L8["Lab 8: Capstone from own loops"]
+  L7 --> L8
+```
+
+| Lab | Title and sheet | Question it answers | What you run | Time and hardware |
+|---|---|---|---|---|
+| 0 | [The toy edit on a 4 x 6 matrix](labs/lab0_toy.md) | What does $W' = W - \alpha u u^T W_S$ do on a matrix small enough to print? | `uv run --with numpy --with matplotlib python labs/lab0_toy.py` | 5 seconds on any laptop CPU |
+| 1 | [Find the edit by diffing the two weight files](labs/lab1_diff.md) | What did the agent change in the 488 weight tensors? | `python labs/lab1_diff.py` | Under 1 minute on VM CPU |
+| 2 | [Reproduce the paper's benchmark numbers](labs/lab2_reproduce.md) | Do the looping failure and the `p50c` fix reproduce on MATH-500, GSM8K, and IFEval? | `python scripts/run_jobs.py plans/trainer.json --gpus 0-7` (or `plans/quick.json --gpus 0,1`) | 5 minutes on 2 A100s (quick run) or 1.5 hours on 8 A100s (`trainer`) |
+| 3 | [Rebuild the edit from the agent's 8 texts](labs/lab3_rebuild.md) | Can we recover layers 15 and 19, $u$, and the 64 neurons from the agent's 8 texts with hooks? | `CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py` | About 1 minute on 1 A100 GPU (plus 21 minutes on 1 A100 to test on `dev256`) |
+| 4 | [Turn the alpha knob and check controls](labs/lab4_dose.md) | How does accuracy change across $\alpha \in \{-4, 1, 2, 4, 6, 8, 12\}$, single layers, all neurons, and random controls? | `python scripts/run_jobs.py plans/sampler.json --gpus 0-6` and `python labs/lab4_endings.py` | 17 to 45 minutes per run on 1 A100 (`sampler`), seconds on CPU for endings |
+| 5 | [Can decoding settings do the same job?](labs/lab5_decoding.md) | Can temperature, presence or repetition penalties, a 4,096-token cap, or turning thinking off replace the weight edit? | `python scripts/run_jobs.py plans/sampler.json --gpus 0-6` (decoding runs) | About 3 to 49 minutes per run on 1 A100 (`sampler`) |
+| 6 | [The weight edit is steering in disguise](labs/lab6_steering.md) | Is the weight edit identical to subtracting $\alpha s(t) u$, and when does $s(t)$ fire on real loops? | `CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py` (and `--parts 3,4`) | About 1 minute on 1 A100 for parts 1 and 2; about 3 minutes for parts 3 and 4 (`sampler`) |
+| 7 | [What else did the edit change?](labs/lab7_side_effects.md) | How much does each edit move WikiText-2 perplexity and KL divergence on productive versus looping text? | `CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py` | About 1.5 minutes (78 seconds) on 1 A100 GPU |
+| 8 | [Capstone: build your own edit from the model's own loops](labs/lab8_capstone.md) | Can we extract $u$ and 64 neurons from the base model's own looping answers and fix unseen problems? | `CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py` | About 5 minutes on 1 A100 to build, 30 minutes on 2 A100s to test (`sampler`) |
+
+## Project layout
+
+- `configs/`: JSON definitions of edited models (`variants.json` for Lab 4 controls and $\alpha$ sweep, `rebuilt.json` from Lab 3, `capstone.json` from Lab 8).
+- `data/`: Prepared JSONL datasets (`dev256.jsonl` with 256 unseen practice problems, `build128.jsonl` with 128 practice problems for Lab 8, plus `math500.jsonl`, `gsm8k.jsonl`, and `ifeval.jsonl`).
+- `external/`: Local checkout of the paper's log repository (`lm_understand-logs`, ignored by `.gitignore`; `exp8_math/` holds the agent's scripts, edit manifest, and work notes).
+- `labs/`: Lab scripts (`lab0_toy.py` through `lab8_capstone.py`), answer inspection tools (`peek.py`, `lab4_endings.py`), lab sheets (`lab0_toy.md` through `lab8_capstone.md`), and the agent's 8 contrastive texts (`labs/data/astra_pairs.json`).
+- `plans/`: Generation job plans for `scripts/run_jobs.py` (`quick.json`, `trainer.json`, `sampler.json`, `extra.json`, and `lab3-tries.json`).
+- `reference/`: The first-principles companion guide [`reference/theory-guide.md`](reference/theory-guide.md).
+- `results/`: Saved JSON outputs for Labs 1 through 8 (including committed `results/*/try-*` folders to compare your runs with ours), per-run graded records in `results/runs/`, the master table [`results/summary.md`](results/summary.md) (`results/summary.json`), and all plots in `results/figures/*.png`.
+- `scripts/`: VM setup and sync scripts (`setup_vm.sh`, `sync_vm.sh`, `pull_runs.sh`, `env.sh`), dataset and model builders (`prepare_data.py`, `build_models.py`), the multi-GPU vLLM runner (`run_jobs.py`, `gen_worker.py`), and reporting scripts (`summarize.py`, `plot_runs.py`).
+- `wsp/`: Shared Python library (`hf.py` for model loading and PyTorch hooks, `edits.py` for rank-one edits and SVD recovery, `loops.py` for verbatim loop detection, `protocol.py` for presets and sampler probabilities, `grading.py`, and `summary.py`).
+
+## Running things
+
+```mermaid
+flowchart LR
+  Trainer["trainer (8x A100)<br/>/nvme/wsp-lab/runs/<br/>bench/*, quick/*, lab3 tries"] <-->|"sync_vm.sh / pull_runs.sh"| Laptop["This machine (CPU)<br/>Project root and results/<br/>Lab 0 and plotting"]
+  Laptop <-->|"sync_vm.sh / pull_runs.sh"| Sampler["sampler (8x A100)<br/>/nvme/wsp-lab/runs/<br/>build/base, dev/*"]
+```
+
+**Cloning the paper's logs.** `.gitignore` ignores `external/`, so the paper's log repository is not committed here. If `external/lm_understand-logs` is missing, clone it from the project root before syncing or building models:
+
+```bash
+git clone https://github.com/RikiyaT/lm_understand-logs external/lm_understand-logs
+git -C external/lm_understand-logs log -1
+```
+
+We used commit `4d09ba6d` (29 October 2026). `scripts/prepare_data.py` and `scripts/build_models.py` read from `external/lm_understand-logs/exp8_math`, and `scripts/sync_vm.sh` copies the parts the GPU machines need (`code`, `practice`, and `runs/p50c`).
+
+**The two GPU VMs and environment setup.** GPU work runs on `trainer` and `sampler`, each with 8 NVIDIA A100 80 GB GPUs. All VM files live under `/nvme/wsp-lab` (`venv/`, `models/`, `data/`, `runs/`, `logs/`, and `code/`). Start every VM command by sourcing `env.sh`, which activates Python 3.12 (vLLM 0.33.0, PyTorch 2.14.0, transformers 5.20.2, flash-linear-attention 0.6.0) and keeps caches inside `/nvme/wsp-lab`:
+
+```bash
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+```
+
+**Syncing code and pulling results.** The two VMs cannot reach each other directly, so files move through this machine. From the project root on this machine:
+
+```bash
+scripts/sync_vm.sh trainer sampler   # push wsp, scripts, configs, plans, labs, and exp8_math files
+scripts/pull_runs.sh trainer sampler # pull merged.jsonl, summary.json, and *.done.json into results/runs/
+python3 scripts/summarize.py results/runs    # rebuild results/summary.md and results/summary.json
+uv run --with matplotlib python scripts/plot_runs.py
+```
+
+`sync_vm.sh` mirrors the five code folders onto the VM with `rsync --delete`, so any file you create or edit only on the VM is deleted or overwritten on the next sync. Edit on this machine first, then sync. The one exception is `configs/capstone.json`, which Lab 8 writes on the VM: syncing never deletes it, but a local copy will overwrite the VM's copy, so pull the VM version first. `pull_runs.sh` copies only run records. Lab scripts write their outputs to `results/labN/` and `results/figures/` on the VM where they ran; copy those back by name:
+
+```bash
+rsync -az sampler:/nvme/wsp-lab/code/results/lab6/ results/lab6/ </dev/null
+rsync -az sampler:/nvme/wsp-lab/code/results/figures/lab6_real_loop.png results/figures/ </dev/null
+```
+
+Copy figures individually instead of syncing the whole folder, since each VM holds only the figures it generated and an older copy would overwrite a newer one.
+
+**Setting up a new machine.** Our two VMs (`trainer` and `sampler`) are already set up. To provision a fresh GPU machine (about 10 minutes and 15 GB under `/nvme/wsp-lab`, no root needed), run these commands from the project root (see `scripts/setup_vm.sh`):
+
+```bash
+scp scripts/setup_vm.sh <host>:/tmp/ && ssh <host> bash /tmp/setup_vm.sh
+scripts/sync_vm.sh <host>
+ssh <host> 'cd /nvme/wsp-lab/code && . scripts/env.sh &&
+  python scripts/prepare_data.py --exp8 external/lm_understand-logs/exp8_math --out /nvme/wsp-lab/data &&
+  python scripts/build_models.py && python scripts/build_models.py --variants configs/rebuilt.json'
+```
+
+**Where full answer texts live.** Full answer texts (`runs/<name>/shard*.texts.jsonl.gz`) are ignored by `.gitignore` and skipped by `scripts/pull_runs.sh`, so they stay on the VM that generated them (while `results/*/try-*` folders are committed to git). Because `labs/peek.py` and `labs/lab4_endings.py` read `.texts.jsonl.gz` files, run them on the VM that produced the run:
+- On `trainer`: `bench/*`, `quick/*`, `dev/rebuilt-pairs2`, `dev/rebuilt-windows36-36`, and `dev/rebuilt-layers15-19-21`.
+- On `sampler`: `build/base` (also copied to `trainer`, where our Lab 6 part 3 runs read it) and every other `dev/*` run (`dev/base`, `dev/p50c`, `dev/rebuilt`, the Lab 4 dose and control runs, the Lab 5 decoding runs, and the Lab 8 `dev/cap*` runs).
+
+**Rerunning a finished job.** `scripts/run_jobs.py` skips any shard with a `shard*.done.json` file. Because all runs are already complete on the VMs, rerunning a plan immediately reports `0 shards to run`. To run a job yourself, move our output folder aside on the VM first:
+
+```bash
+mv /nvme/wsp-lab/runs/quick /nvme/wsp-lab/runs/quick-ours
+```
+
+Then run the plan; `python labs/peek.py quick/base` reads your run and `python labs/peek.py quick-ours/base` reads ours.
+
+**How to read `results/summary.md`.** Every generation run has one row in [`results/summary.md`](results/summary.md) with these nine columns:
+- `run`: Run identifier (`bench/<model>/<dataset>`, `dev/<variant>`, `build/base`, or `quick/<model>`).
+- `problems x samples`: Distinct problems times sampled answers per problem (for example `256x2` = 512 answers on `dev256`).
+- `accuracy %`: Percentage of correct answers, plus or minus one standard error (SE, typical sampling uncertainty; a gap of 2 SE or more is unlikely to be chance).
+- `cut at the cap %`: Percentage of answers that hit the token limit (32,768 by default) without finishing.
+- `mean tokens`: Average generated tokens per answer.
+- `correct when finished %`: Accuracy among answers that finished before the token limit (97.5% to 100% for the unedited model on math benchmarks, showing its main failure is not finishing).
+- `cut answers ending in repetition %`: Percentage of cap-cut answers ending in repetition: either a word-for-word loop (a block of words repeated at least 3 times in a row) or looser circling (at least half of the 4-word phrase positions in the last 200 words hold a phrase repeated 3 or more times there; see `results/summary.md`).
+- `gain vs baseline (pp)`: Accuracy change in percentage points versus the paired baseline on the same problems, plus or minus paired SE (`dev/*` against `dev/base`, `dev/p50c+nothink` against `dev/base+nothink`, `bench/p50c/X` against `bench/base/X`, and `quick/*` against `quick/base`).
+- `rescued / lost problems`: `rescued` counts problems where the run got at least one answer right and the baseline got none; `lost` counts the reverse.
+
+## Sources
+
+- **Blog post:** Rikiya Takahashi, ["Sparks of Weight-Space Programming"](https://rikiyat.github.io/blog/weight-space-programming/) (MIT CSAIL, 2 Nov 2026).
+- **Experiment logs and code:** [`https://github.com/RikiyaT/lm_understand-logs`](https://github.com/RikiyaT/lm_understand-logs) (local copy in `external/lm_understand-logs/`, with Experiment 8 in `external/lm_understand-logs/exp8_math/`).
+- **Released edited model:** [`https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming`](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming).
+- **Interactive demo Space:** [`https://huggingface.co/spaces/ML-Intern-lab/weight-space-programming-demo`](https://huggingface.co/spaces/ML-Intern-lab/weight-space-programming-demo).
+- **First-principles theory guide:** [`reference/theory-guide.md`](reference/theory-guide.md).

code/configs/capstone.json

@@ -0,0 +1,4392 @@
+{
+ "capA-a6": {
+  "about": "Lab 8 recipe A: u and 64 neurons from the base model's own loops (windows just before the loop), alpha 6",
+  "layers": {
+   "15": {
+    "alpha": 6,
+    "dir": [
+     -0.19642147,
+     0.03620983,
+     0.15367946,
+     0.06497408,
+     0.03057081,
+     -0.15543503,
+     0.01351306,
+     -0.05578131,
+     -0.01658088,
+     0.01544302,
+     0.05921284,
+     0.06464835,
+     0.02034978,
+     0.16278124,
+     0.02162938,
+     0.1445464,
+     0.07718109,
+     0.13444427,
+     -0.10969324,
+     -0.06296548,
+     0.07529427,
+     0.09108829,
+     0.0843537,
+     0.08002103,
+     -0.02452753,
+     0.02076042,
+     0.00602258,
+     -0.00699884,
+     0.0446047,
+     0.07109267,
+     0.00730817,
+     -0.14850335,
+     0.02196117,
+     -0.06272466,
+     -0.08063199,
+     -0.01963586,
+     -0.06232904,
+     0.18691534,
+     -0.02989487,
+     -0.02307681,
+     -0.09384636,
+     0.01869328,
+     -0.08615778,
+     -0.01516586,
+     0.02899578,
+     0.00498247,
+     0.0662872,
+     -0.04447446,
+     0.03304262,
+     0.06277169,
+     0.00584157,
+     0.01387291,
+     -0.04707102,
+     0.01839207,
+     0.03542451,
+     0.08492478,
+     0.03609524,
+     0.07776414,
+     -0.05058173,
+     -0.02504171,
+     -0.02387556,
+     -0.06714687,
+     -0.0498366,
+     0.02289074,
+     0.04711203,
+     0.03032538,
+     0.06156486,
+     -0.00659584,
+     -0.03459039,
+     -0.03311691,
+     0.02699054,
+     0.02207826,
+     -0.04704108,
+     -0.02012696,
+     -0.10484155,
+     0.01460513,
+     -0.05287892,
+     -0.00257121,
+     0.03665803,
+     -8.442e-05,
+     0.0344651,
+     0.09257579,
+     0.02871555,
+     -0.05414261,
+     0.01085017,
+     0.12585324,
+     0.05451579,
+     -0.0301715,
+     0.00942063,
+     0.04333999,
+     0.05119767,
+     -0.05102875,
+     -0.04207597,
+     0.02894551,
+     0.05760689,
+     -0.01165443,
+     -0.0400944,
+     0.01880827,
+     -0.00536408,
+     0.00873167,
+     -0.06115466,
+     -0.00820906,
+     -0.01046184,
+     0.00953318,
+     -0.04116342,
+     -0.00371408,
+     0.05087092,
+     0.02443181,
+     -0.02275142,
+     0.0121256,
+     0.03931541,
+     0.01642587,
+     0.00594028,
+     -0.01517512,
+     -0.00460996,
+     0.05321185,
+     -0.00303453,
+     -0.04462391,
+     -0.02955657,
+     0.06756701,
+     0.01269387,
+     0.00439294,
+     -0.01688814,
+     0.00599055,
+     -0.05860509,
+     -0.00640408,
+     -0.02288292,
+     -0.01169652,
+     0.01191018,
+     0.06990831,
+     -0.03254844,
+     0.01306989,
+     0.0324331,
+     0.01259914,
+     0.012772,
+     0.01562133,
+     0.02279549,
+     -0.06174612,
+     0.10596453,
+     0.01665649,
+     0.05464435,
+     0.03043335,
+     0.00261869,
+     0.04120243,
+     -0.01908323,
+     -0.01475778,
+     0.01024658,
+     0.00602525,
+     0.034647,
+     -0.01601795,
+     -0.0098665,
+     0.00715095,
+     0.0386551,
+     0.00813517,
+     -0.02485966,
+     -0.00370103,
+     -0.04929582,
+     0.00873361,
+     0.00677907,
+     -0.01257964,
+     0.06989538,
+     -0.03499186,
+     0.01892586,
+     0.00830449,
+     -0.02865393,
+     0.0222334,
+     0.02955934,
+     0.01670051,
+     -0.03454896,
+     0.00511459,
+     -0.06118869,
+     -0.04395781,
+     -0.00728062,
+     0.03459383,
+     -0.01677352,
+     -0.05499182,
+     0.02996176,
+     -0.00793099,
+     0.01103673,
+     -0.02698723,
+     -0.03706948,
+     0.0058741,
+     0.01856517,
+     -0.0184078,
+     -0.02116514,
+     -0.0109629,
+     -0.06174179,
+     -0.01890302,
+     0.03024366,
+     0.00644585,
+     -0.04642926,
+     0.01424284,
+     0.04885947,
+     0.01391168,
+     0.00777366,
+     0.02742206,
+     0.00707238,
+     -0.01929868,
+     0.03457972,
+     -0.04974987,
+     0.00105937,
+     -0.01246471,
+     -0.02007423,
+     0.000219,
+     0.02734674,
+     0.05963851,
+     0.03654126,
+     -0.0129875,
+     -0.04307868,
+     -0.00316628,
+     0.01014835,
+     -0.00251832,
+     -0.03931477,
+     -0.00972052,
+     -0.00568248,
+     0.00316628,
+     0.00346336,
+     0.03684881,
+     -0.00336378,
+     -0.05937261,
+     -0.00259194,
+     0.00196418,
+     0.02412233,
+     0.00198911,
+     0.00796683,
+     -0.03575343,
+     0.0077577,
+     -0.02936143,
+     -0.04127926,
+     -0.03727643,
+     -0.04707571,
+     0.04822274,
+     0.0111492,
+     0.0357839,
+     -0.01515097,
+     -0.0181218,
+     0.02394596,
+     0.02694717,
+     -0.035783,
+     0.0281493,
+     -0.01310454,
+     0.02221958,
+     0.02990598,
+     -0.01907246,
+     0.04340885,
+     0.0115768,
+     0.00898189,
+     -0.06997719,
+     0.01230609,
+     -0.00925507,
+     0.05233327,
+     -0.02267844,
+     -0.05345995,
+     0.00955621,
+     0.01006574,
+     0.10575184,
+     -0.0681143,
+     -0.01524912,
+     -0.01217928,
+     0.00278406,
+     -0.00136194,
+     0.02119698,
+     -0.00788542,
+     0.00100464,
+     0.02298328,
+     0.00925969,
+     0.02953216,
+     -0.00704557,
+     0.00990218,
+     -0.00553312,
+     -0.01701421,
+     -0.00671327,
+     -0.00800176,
+     -0.03354774,
+     -0.00255209,
+     0.01456584,
+     -0.02189159,
+     -0.0115431,
+     -0.01603708,
+     -0.03584647,
+     0.04013485,
+     -0.01710701,
+     -0.05626024,
+     0.02047668,
+     0.01898202,
+     -0.00604367,
+     0.06082913,
+     0.00710128,
+     -0.02479753,
+     0.01761165,
+     0.01341764,
+     -0.01323848,
+     -0.02196406,
+     -0.02244893,
+     -0.02151935,
+     -0.01825762,
+     0.00698251,
+     -0.02361639,
+     0.00114175,
+     -0.02762575,
+     0.16642642,
+     -0.00773265,
+     0.00217698,
+     0.04885407,
+     -0.01389956,
+     0.01700542,
+     0.01211646,
+     0.00832499,
+     -0.00395274,
+     0.01165037,
+     -0.00379319,
+     0.0132295,
+     -0.00373486,
+     0.02491317,
+     -0.02714441,
+     0.03632208,
+     -0.08712547,
+     -0.06559645,
+     -0.02572042,
+     0.04448447,
+     -0.00740151,
+     0.02038549,
+     0.03381699,
+     -0.00414533,
+     0.02067941,
+     -0.04021452,
+     0.04214275,
+     -0.00379636,
+     0.01809693,
+     -0.01443073,
+     0.02897453,
+     -0.01338227,
+     0.00038025,
+     0.02115615,
+     0.01903441,
+     -0.01938041,
+     0.02999728,
+     0.01710915,
+     -0.00018278,
+     0.03359207,
+     -0.15671353,
+     0.06294922,
+     -0.02928597,
+     -0.00917316,
+     -0.02511222,
+     0.04767934,
+     0.01702067,
+     0.01511116,
+     -0.01941296,
+     0.04285015,
+     -0.00288132,
+     -0.01123376,
+     -0.01124302,
+     -0.00706538,
+     -0.02094487,
+     -0.06044375,
+     -0.01406492,
+     -0.02563945,
+     0.02763499,
+     -0.01873456,
+     0.00962685,
+     0.00600576,
+     0.02742081,
+     -0.03916538,
+     0.00491569,
+     0.01069831,
+     0.01297124,
+     0.0156616,
+     0.01917705,
+     0.02480869,
+     -0.0106574,
+     -0.00759733,
+     0.00509676,
+     0.01549563,
+     -0.02802384,
+     -0.01822686,
+     0.00713354,
+     -0.00076786,
+     0.01179229,
+     -0.01646931,
+     -0.02218645,
+     -0.01974743,
+     -0.02323301,
+     -0.0161015,
+     0.00569559,
+     -0.0318373,
+     0.00214495,
+     0.02582066,
+     0.0120695,
+     -0.00323667,
+     -0.0181305,
+     0.00456708,
+     0.00088165,
+     -0.00181784,
+     -0.0323725,
+     0.03889608,
+     -0.00593832,
+     -0.00065643,
+     -0.00818366,
+     0.00936741,
+     -0.01945619,
+     0.02748532,
+     0.01352952,
+     0.0104318,
+     -0.04723194,
+     -0.04889061,
+     -0.03551376,
+     0.01957176,
+     0.03616486,
+     -0.00134334,
+     0.02413643,
+     -0.03108473,
+     0.00653157,
+     -0.00807491,
+     0.01170466,
+     0.01715695,
+     -0.02850865,
+     -0.02597558,
+     -0.01232146,
+     -0.00075774,
+     0.00142594,
+     0.01942839,
+     -0.01098431,
+     -0.00176267,
+     -0.08830237,
+     0.00896412,
+     0.0134145,
+     0.00978285,
+     0.0046121,
+     -0.00490226,
+     -0.01901206,
+     -0.01399694,
+     0.01516195,
+     -0.00334541,
+     -0.01679288,
+     0.03267581,
+     -0.01082043,
+     0.01129618,
+     0.03566024,
+     0.02445141,
+     0.01040449,
+     0.05409379,
+     0.01997331,
+     -0.00243477,
+     0.01394332,
+     0.02577617,
+     0.03131029,
+     0.00099272,
+     0.0257875,
+     0.00535434,
+     0.01975382,
+     0.01685537,
+     -0.00746932,
+     0.0180197,
+     -0.00473603,
+     0.00482269,
+     -0.02610375,
+     -0.00263356,
+     0.04015455,
+     0.01175972,
+     0.00810278,
+     0.03288561,
+     -0.00391045,
+     -0.02447399,
+     0.0166478,
+     0.01949768,
+     0.02602378,
+     -0.00563907,
+     -0.00467757,
+     0.00259495,
+     -0.0125137,
+     0.05420196,
+     -0.00224606,
+     -0.00816949,
+     0.01439246,
+     -0.0126056,
+     0.00068195,
+     0.01309173,
+     -0.00121671,
+     -0.03312248,
+     -0.01834775,
+     0.0016364,
+     -0.02588954,
+     0.00480775,
+     -0.01843944,
+     -0.0041554,
+     0.01282341,
+     0.03219848,
+     -0.01286189,
+     0.00484094,
+     -0.01602968,
+     -0.01150733,
+     -0.03129844,
+     0.0092206,
+     0.03618083,
+     0.02785339,
+     0.02549848,
+     0.00144244,
+     0.03333158,
+     0.01289997,
+     0.00317834,
+     0.00372877,
+     1.953e-05,
+     0.00166176,
+     0.00610443,
+     0.01573688,
+     0.00230932,
+     -0.00716901,
+     -0.01175373,
+     0.00542746,
+     -0.01469357,
+     -0.0064027,
+     -0.00792847,
+     0.0059745,
+     -0.02925546,
+     0.01021165,
+     -0.06110986,
+     0.01822174,
+     0.00379858,
+     0.02198827,
+     0.04593732,
+     0.01543745,
+     -0.00720788,
+     0.00476402,
+     -0.0038778,
+     0.01266388,
+     -0.00248264,
+     0.05027047,
+     -0.01237561,
+     -0.00957299,
+     -0.01446781,
+     -0.00401733,
+     0.01843788,
+     -0.02245559,
+     0.00411258,
+     -0.02207132,
+     0.01917735,
+     0.01551384,
+     0.06460287,
+     0.02049114,
+     0.00455079,
+     -0.00079158,
+     -0.00720685,
+     -0.02944683,
+     -0.00974075,
+     -0.00361087,
+     -0.02188736,
+     0.00510789,
+     0.00083505,
+     0.01718823,
+     -0.01092549,
+     -0.06818206,
+     0.02002659,
+     -0.00443351,
+     0.02195536,
+     -0.03922576,
+     0.02893249,
+     -0.00792199,
+     -0.01345805,
+     0.03962484,
+     0.01178997,
+     0.03471842,
+     0.02131379,
+     0.0221365,
+     -0.01394667,
+     -0.0011956,
+     -0.00585808,
+     -0.00686869,
+     -0.00681893,
+     0.00314807,
+     0.02003211,
+     -0.01824619,
+     -0.01195085,
+     -0.00977994,
+     -0.02926364,
+     0.00122858,
+     0.0081286,
+     -0.0084117,
+     0.03174116,
+     0.01527151,
+     0.01818123,
+     -0.01315631,
+     0.00535931,
+     -0.01808719,
+     -0.01626276,
+     0.01888465,
+     -0.00875709,
+     0.0206139,
+     0.01838784,
+     -0.03064582,
+     0.01166973,
+     0.02911546,
+     0.02594003,
+     -0.00576419,
+     -0.00914466,
+     -0.0237985,
+     0.01038998,
+     -0.01717332,
+     0.02492017,
+     -0.00753594,
+     0.01322463,
+     0.01343332,
+     0.00121454,
+     0.01580115,
+     0.00307425,
+     0.0345596,
+     -0.01798728,
+     0.0323272,
+     -0.0068936,
+     -0.00869091,
+     0.0045136,
+     -0.00837262,
+     -0.00855466,
+     0.0034192,
+     0.02522525,
+     -0.00705909,
+     -0.019891,
+     0.00191272,
+     -0.01211489,
+     0.0285136,
+     -0.00566154,
+     0.04103695,
+     -0.0092919,
+     0.00241761,
+     0.01635534,
+     -0.00316568,
+     -0.01597946,
+     -0.03456803,
+     -0.03958578,
+     0.01394486,
+     -0.01306217,
+     -0.00343395,
+     0.01033765,
+     0.01521762,
+     0.00855863,
+     -0.00389284,
+     0.02527569,
+     -0.01169722,
+     0.00200877,
+     0.04353902,
+     0.02844795,
+     0.02484205,
+     0.01893727,
+     -0.03412919,
+     0.01272242,
+     0.00202661,
+     0.023077,
+     0.02425507,
+     -0.0236186,
+     -0.04173858,
+     0.00582972,
+     0.00735155,
+     -0.00016013,
+     0.01041111,
+     0.0020433,
+     0.01619346,
+     0.01290293,
+     0.0304944,
+     -0.02799492,
+     0.00890391,
+     0.01989347,
+     0.01319829,
+     0.0316516,
+     0.01294222,
+     0.01560815,
+     6.969e-05,
+     0.00689163,
+     0.0201618,
+     0.01609466,
+     -0.00964703,
+     0.00868139,
+     -0.03052312,
+     -0.00467848,
+     -0.01110903,
+     -0.00782298,
+     -0.01644403,
+     -0.02660462,
+     -0.00318245,
+     -0.03656901,
+     -0.01252536,
+     2.685e-05,
+     0.00575581,
+     0.00867378,
+     -0.03588792,
+     -0.00962772,
+     0.00612709,
+     0.02778116,
+     0.00617909,
+     -0.02462822,
+     0.01033108,
+     0.02212686,
+     0.00757206,
+     0.0081198,
+     0.02414416,
+     -0.00085268,
+     -0.01033239,
+     -0.01519087,
+     0.01407947,
+     0.01629492,
+     -0.00385944,
+     0.00988953,
+     0.01378492,
+     0.04593561,
+     0.00968383,
+     0.03686356,
+     -0.01484743,
+     0.01556203,
+     -0.01520757,
+     0.02010669,
+     -0.0176068,
+     0.02796162,
+     -0.00331875,
+     -0.01562665,
+     -0.02325287,
+     -0.03160228,
+     0.01461223,
+     -0.00678877,
+     -0.0085217,
+     0.00715341,
+     -0.003018,
+     0.01534117,
+     0.00859851,
+     0.00273759,
+     -0.00587398,
+     0.0363301,
+     0.04355061,
+     -0.01432952,
+     0.02605049,
+     0.02288688,
+     0.03925271,
+     0.01841203,
+     -0.03189914,
+     0.01263224,
+     0.01408128,
+     0.0248368,
+     0.01680999,
+     -0.01338161,
+     0.00324507,
+     -0.02825224,
+     -0.01734889,
+     0.00191852,
+     -0.01146315,
+     -0.0038348,
+     -0.00840155,
+     -0.0083625,
+     -0.0499628,
+     -0.02321059,
+     -0.01606922,
+     0.01277077,
+     -0.00190077,
+     0.01025087,
+     -0.03114786,
+     0.00791134,
+     0.01744954,
+     0.02002841,
+     -0.0255163,
+     -0.02600448,
+     -0.01578891,
+     -0.02690942,
+     -0.02880245,
+     -0.00431575,
+     0.00600727,
+     -0.03090141,
+     0.01105587,
+     -0.0015371,
+     0.00296751,
+     0.01277654,
+     -0.00477245,
+     -0.02422091,
+     0.02559535,
+     -0.00133811,
+     -0.00772111,
+     -0.04095183,
+     0.03351398,
+     0.01896126,
+     0.00341619,
+     0.01234442,
+     -0.03707758,
+     0.00587088,
+     -0.00318547,
+     0.0377168,
+     -0.01426577,
+     -0.02044055,
+     -0.0175332,
+     0.01608709,
+     0.03591258,
+     0.00580495,
+     0.01636545,
+     -0.01722871,
+     0.00606186,
+     -0.00400982,
+     0.0109228,
+     0.02489203,
+     0.01245983,
+     -0.00455665,
+     -0.01384578,
+     -0.00659455,
+     0.00777224,
+     -0.00334753,
+     -0.0177411,
+     -0.00531191,
+     0.00629907,
+     0.02441626,
+     -0.01576012,
+     0.03351801,
+     0.02288313,
+     -0.05304268,
+     -0.0071294,
+     0.03232586,
+     0.00891935,
+     -0.01977362,
+     -0.0027413,
+     -0.01511885,
+     0.02056648,
+     0.00657744,
+     0.01876156,
+     0.01019189,
+     -0.01308637,
+     0.02896509,
+     0.00256687,
+     -0.00765172,
+     0.03474916,
+     0.00256524,
+     0.00674338,
+     0.04313926,
+     -0.00284791,
+     0.0150191,
+     0.01572495,
+     0.02367985,
+     -0.01331006,
+     0.01604667,
+     -0.02399078,
+     0.03464028,
+     -0.00468134,
+     0.00330221,
+     0.00213209,
+     0.00835461,
+     -0.02771784,
+     0.00901233,
+     0.0105306,
+     -0.00924321,
+     0.02637529,
+     -0.02676393,
+     0.00356408,
+     -0.04889319,
+     -0.02941449,
+     0.00112647,
+     -0.02983064,
+     0.03998261,
+     -0.02925883,
+     -0.00396614,
+     0.01319757,
+     0.01322922,
+     -0.00243386,
+     -0.00769942,
+     0.01422793,
+     -0.02085671,
+     -0.01465598,
+     -0.00793447,
+     -0.0158751,
+     -0.04526321,
+     0.03444614,
+     0.01128489,
+     -0.02307381,
+     0.02501738,
+     0.00990441,
+     0.02356402,
+     -0.00317668,
+     -0.0126682,
+     0.02966011,
+     0.00138062,
+     -0.02201164,
+     -0.02099483,
+     -0.00252512,
+     0.0012899,
+     0.01235682,
+     -0.01162883,
+     -0.02899506,
+     -0.05046245,
+     0.03096318,
+     -0.0059149,
+     -0.02507442,
+     -0.00721694,
+     -0.03142115,
+     -0.01031928,
+     0.00791632,
+     0.01309902,
+     0.02127181,
+     -0.01971884,
+     -0.02214627,
+     -0.02675827,
+     0.00386679,
+     0.00320048,
+     -0.00816859,
+     -0.01619648,
+     -0.01310949,
+     0.00787304,
+     0.00961308,
+     -0.0018458,
+     0.01286562,
+     0.03139392,
+     -0.00560179,
+     -0.00183139,
+     -0.00443225,
+     -0.00790803,
+     -0.01636733,
+     -0.00759358,
+     0.00203761,
+     0.01884115,
+     -0.01272084,
+     0.00250993,
+     0.03664702,
+     -0.04066893,
+     0.00552276,
+     0.0137794,
+     -0.02775645,
+     -0.00558242,
+     -0.01088545,
+     -0.01883953,
+     -0.00373733,
+     0.00910788,
+     0.01156108,
+     -0.01827489,
+     -0.02444516,
+     -0.00397615,
+     -0.02902949,
+     -0.01167935,
+     -0.00136049,
+     0.00129787,
+     0.00911742,
+     -0.00392505,
+     -0.00567445,
+     0.00135177,
+     0.01242116,
+     0.00707127,
+     0.01174442,
+     0.00069741,
+     -0.02878679,
+     -0.02556429,
+     0.00227394,
+     0.00793164,
+     -0.00369597,
+     0.03255857,
+     0.0202454,
+     -0.01635662,
+     -0.02483233,
+     0.00268604,
+     0.01370827,
+     -0.00288547,
+     -0.0091429,
+     -0.04589674,
+     -0.00381021,
+     -0.02784827,
+     -0.02464681,
+     0.03012316,
+     -0.02526064,
+     -0.0019681,
+     0.00598796,
+     -0.01061581,
+     -0.01781175,
+     -0.01317692,
+     -0.01122522,
+     -0.00949888,
+     -0.04640295,
+     0.00405447,
+     0.00692723,
+     -0.01286737,
+     0.0194275,
+     0.03252542,
+     -0.03120079,
+     -0.00596153,
+     -0.02418105,
+     0.01823442,
+     -0.00854398,
+     0.03457795,
+     -0.00829153,
+     0.01389347,
+     0.01776138,
+     -0.0100889,
+     -0.00340556,
+     -0.00454865,
+     -0.00269702,
+     -0.01599278,
+     0.03753144,
+     0.02651554,
+     -0.01864239,
+     -0.00012952,
+     0.03227106,
+     0.00020344,
+     0.03920548,
+     0.00676807,
+     -0.01476055,
+     -0.02480705,
+     -0.0294454,
+     -0.02185388,
+     0.01376991,
+     0.0019891,
+     -0.00082901,
+     -0.00209755,
+     -0.00537243,
+     0.00898147,
+     -0.03283366,
+     -0.02692142,
+     -0.03934251,
+     -0.03625376,
+     -0.00753794,
+     0.00549959,
+     0.00160269,
+     -0.01382472,
+     -0.03401972,
+     -0.01955991,
+     -0.01109684,
+     0.02179758,
+     -0.03753874,
+     -0.06081887,
+     -0.00784457,
+     -0.01305669,
+     -0.04447476,
+     -0.00143666,
+     -0.01756143,
+     -0.02385547,
+     0.01329127,
+     -0.01187246,
+     -0.01682425
+    ],
+    "cols": [
+     0,
+     11,
+     25,
+     35,
+     37,
+     38,
+     39,
+     42,
+     46,
+     56,
+     61,
+     63,
+     65,
+     68,
+     81,
+     105,
+     110,
+     129,
+     135,
+     169,
+     215,
+     259,
+     264,
+     266,
+     294,
+     371,
+     414,
+     426,
+     452,
+     496,
+     504,
+     505,
+     509,
+     516,
+     580,
+     582,
+     601,
+     638,
+     682,
+     733,
+     738,
+     750,
+     755,
+     882,
+     891,
+     907,
+     926,
+     956,
+     1047,
+     1060,
+     1087,
+     1130,
+     1155,
+     1175,
+     1184,
+     1702,
+     1727,
+     1757,
+     1787,
+     2248,
+     2288,
+     2431,
+     2636,
+     3003
+    ]
+   },
+   "19": {
+    "alpha": 6,
+    "dir": [
+     -0.41399729,
+     0.04963015,
+     0.12075837,
+     0.02570651,
+     0.07051032,
+     -0.14113681,
+     0.03799108,
+     0.01167555,
+     -0.0064611,
+     -0.01368775,
+     0.06116361,
+     0.03473959,
+     0.05443978,
+     0.1034804,
+     0.03853016,
+     0.10406735,
+     0.07458493,
+     0.19364277,
+     -0.10859208,
+     -0.07088345,
+     0.09971021,
+     0.04388057,
+     0.06559257,
+     0.07909922,
+     -0.0305314,
+     0.00847475,
+     0.0029426,
+     0.01185544,
+     0.01822992,
+     0.04678907,
+     0.00322332,
+     -0.12947491,
+     0.05207255,
+     -0.04346941,
+     -0.08363152,
+     0.01037417,
+     -0.024745,
+     0.13930042,
+     -0.03795761,
+     -0.02326387,
+     -0.08854304,
+     0.00513338,
+     -0.04700642,
+     -0.0410671,
+     -0.00597825,
+     -0.01444194,
+     0.03166021,
+     -0.06466202,
+     0.03836314,
+     0.10043173,
+     0.00429663,
+     0.01563108,
+     -0.04114208,
+     0.01129955,
+     0.03724522,
+     0.07670557,
+     0.04311912,
+     0.07472885,
+     -0.02922567,
+     -0.03833525,
+     0.00456427,
+     -0.07277104,
+     -0.04912667,
+     0.03210241,
+     0.06677274,
+     0.04299912,
+     0.01544243,
+     -0.00771099,
+     -0.04890209,
+     -0.05095252,
+     0.03187749,
+     0.01289586,
+     -0.04323121,
+     0.01584703,
+     -0.09161146,
+     0.01245053,
+     -0.06768124,
+     0.00860595,
+     -0.00343637,
+     0.01398703,
+     0.03606768,
+     0.0810465,
+     0.02315859,
+     -0.04040698,
+     0.03198423,
+     0.11273056,
+     0.07610082,
+     -0.0318647,
+     0.01234657,
+     0.02210993,
+     0.04747062,
+     -0.06642445,
+     -0.06831807,
+     -0.00169164,
+     0.04582261,
+     -0.01536702,
+     -0.02826523,
+     -0.01215465,
+     -0.011371,
+     0.01469242,
+     -0.05166398,
+     -0.02717075,
+     -0.00141099,
+     0.02742627,
+     -0.02786403,
+     -0.00629218,
+     0.03265667,
+     0.01514308,
+     -0.0149828,
+     -0.00829845,
+     0.04983302,
+     0.04255297,
+     0.00411385,
+     -0.03872541,
+     0.0036479,
+     0.04089405,
+     -0.02173937,
+     -0.07732391,
+     -0.02259742,
+     0.05235458,
+     0.0173907,
+     0.02537671,
+     -0.02115237,
+     0.01312889,
+     -0.03039251,
+     0.00446135,
+     -0.03203883,
+     0.02627006,
+     -0.00464935,
+     0.06962904,
+     -0.03380452,
+     -0.01486555,
+     0.03601762,
+     0.04170777,
+     0.02079187,
+     0.01055915,
+     0.013812,
+     -0.07132905,
+     0.05949075,
+     0.00309522,
+     0.0485229,
+     0.02636794,
+     0.02857509,
+     0.04520793,
+     -0.00615302,
+     -0.00280603,
+     0.00502628,
+     -0.01417321,
+     0.02896457,
+     -0.01341004,
+     -0.01564776,
+     0.01597291,
+     0.02518996,
+     0.02494553,
+     -0.00542398,
+     -0.00570583,
+     -0.01915297,
+     0.00438684,
+     0.00933441,
+     -0.01003382,
+     0.07714284,
+     -0.00258696,
+     0.01232391,
+     0.01856954,
+     -0.02422703,
+     0.00045015,
+     -0.00747694,
+     -0.04692789,
+     -0.03999501,
+     -0.00415803,
+     -0.02718486,
+     -0.02438036,
+     -0.00504409,
+     0.01552685,
+     -0.01566504,
+     -0.03189168,
+     0.01812186,
+     -0.02202662,
+     0.02614452,
+     -0.00671965,
+     -0.03856579,
+     0.02344356,
+     -0.00223916,
+     -0.00331282,
+     -0.02047138,
+     -0.01889345,
+     -0.04949783,
+     -0.02935014,
+     0.0107663,
+     0.00219746,
+     -0.06389891,
+     0.01158193,
+     0.04443077,
+     0.01606618,
+     -0.01341348,
+     0.02656645,
+     0.01186088,
+     -0.0103251,
+     0.00045351,
+     -0.05710696,
+     0.01312309,
+     -0.01828778,
+     -0.02093595,
+     0.01126861,
+     0.00185225,
+     0.06209056,
+     0.01881377,
+     -0.0147169,
+     -0.04910628,
+     -0.00019105,
+     -0.00031153,
+     0.01395955,
+     -0.04751529,
+     -0.02592737,
+     -0.02079906,
+     -0.01138995,
+     0.02150493,
+     0.0168967,
+     0.01505595,
+     -0.07899936,
+     -0.01489948,
+     -0.01328334,
+     0.03457796,
+     -0.0035607,
+     0.02112566,
+     -0.05852021,
+     -0.00032909,
+     -0.01642394,
+     -0.02042259,
+     -0.03190672,
+     -0.07005456,
+     0.05231471,
+     0.01389211,
+     0.03463357,
+     0.00445699,
+     -0.01345005,
+     0.01217057,
+     0.02445503,
+     -0.04094464,
+     0.03871498,
+     -0.00529151,
+     0.04551385,
+     0.02378383,
+     -0.0063007,
+     0.0368812,
+     0.00015905,
+     0.02571216,
+     -0.04776682,
+     -0.00851591,
+     0.00099652,
+     0.04026485,
+     -0.01877851,
+     -0.0219236,
+     -0.00297592,
+     0.01029671,
+     0.10225255,
+     -0.07007081,
+     -0.02340342,
+     0.00304178,
+     0.01930523,
+     -0.02637254,
+     0.00015534,
+     0.01005215,
+     0.01588354,
+     0.01971863,
+     0.00551752,
+     0.01135873,
+     0.00747602,
+     0.01103365,
+     0.01211043,
+     0.00037698,
+     -0.02620154,
+     0.00256552,
+     -0.02569012,
+     -0.02783686,
+     0.00628542,
+     -0.00104915,
+     -0.0084195,
+     -0.03021691,
+     -0.06425054,
+     0.02496984,
+     -0.0123,
+     -0.03871801,
+     0.00344185,
+     0.0094332,
+     -0.01717056,
+     0.04348284,
+     -0.01649027,
+     -0.01774609,
+     -0.00925723,
+     0.01012276,
+     -0.02162685,
+     -0.00469051,
+     -0.00543928,
+     0.00891075,
+     -0.02664486,
+     0.00453793,
+     -0.00719609,
+     -0.0229699,
+     -0.02674694,
+     0.08312266,
+     -0.02648537,
+     -0.01864358,
+     0.05058388,
+     -0.02575401,
+     0.02509584,
+     -0.00238742,
+     -0.01080631,
+     -0.02850006,
+     0.03767575,
+     -0.01170178,
+     -0.04227291,
+     -0.00819114,
+     0.0155615,
+     -0.01753663,
+     0.01993192,
+     -0.08148444,
+     -0.0680353,
+     0.00388821,
+     0.03550627,
+     0.00862947,
+     0.00714255,
+     0.010537,
+     0.00096149,
+     -0.00442716,
+     -0.01539658,
+     0.03665021,
+     0.00852256,
+     0.01478875,
+     -0.03822764,
+     0.04834064,
+     -0.00460326,
+     0.01477617,
+     0.04410122,
+     0.01627444,
+     -0.00093999,
+     -0.00024491,
+     -0.00553846,
+     -0.00497731,
+     0.04350387,
+     -0.14360385,
+     0.02938055,
+     -0.0352683,
+     -0.00509822,
+     -0.02180302,
+     0.00219526,
+     -0.00115064,
+     -0.00144354,
+     -0.02192711,
+     0.04223738,
+     -0.02610195,
+     -0.01616591,
+     -0.01219281,
+     -0.0051492,
+     -0.02957796,
+     -0.02786715,
+     -0.02184732,
+     -0.01340924,
+     0.01838329,
+     -0.02598088,
+     0.02937649,
+     0.01490723,
+     0.01670144,
+     -0.03311621,
+     0.02081646,
+     0.01394797,
+     -0.00322997,
+     0.00261211,
+     0.03853096,
+     0.02314228,
+     0.00417184,
+     -0.02041819,
+     0.00740087,
+     0.00630271,
+     -0.02640118,
+     -0.01714695,
+     0.0081078,
+     0.02775934,
+     -0.00849896,
+     -0.02215351,
+     -0.0097992,
+     0.0137838,
+     -0.00961078,
+     0.00229612,
+     -0.00337052,
+     -0.01665382,
+     0.00472534,
+     0.01867853,
+     0.02347246,
+     0.00167132,
+     -0.00106219,
+     -0.00794656,
+     -0.00257331,
+     -0.00023658,
+     0.00359527,
+     0.04330531,
+     -0.01585915,
+     0.00389967,
+     -0.00603891,
+     0.01601479,
+     -0.00080251,
+     -0.00465458,
+     0.00355762,
+     -0.00012541,
+     -0.01363729,
+     -0.03129135,
+     -0.0451249,
+     0.03790808,
+     0.01233148,
+     -0.00681186,
+     0.02245213,
+     -0.02457226,
+     0.0053979,
+     -0.00176179,
+     -0.01983261,
+     -0.00289942,
+     -0.054424,
+     -0.01660743,
+     -0.01866144,
+     -0.0056431,
+     -0.01305195,
+     -0.01278731,
+     -0.01718922,
+     -0.01980148,
+     -0.12208816,
+     0.01950914,
+     0.00548164,
+     -0.00599155,
+     0.01228508,
+     -0.01484066,
+     -0.02760676,
+     0.02717005,
+     -0.00303716,
+     -0.00324011,
+     0.00126969,
+     0.02935875,
+     -0.01822443,
+     0.01723574,
+     0.01369331,
+     0.01482911,
+     -0.00349897,
+     -0.0075394,
+     -0.00909447,
+     -0.01029597,
+     0.02345074,
+     0.03705572,
+     0.00569269,
+     0.02199166,
+     0.00893563,
+     -0.00376267,
+     0.00459211,
+     0.03037536,
+     -0.01397452,
+     -0.00245161,
+     -0.00043594,
+     -0.00770201,
+     -0.02184014,
+     0.00735113,
+     0.03283777,
+     -0.0031054,
+     0.00327932,
+     0.02279597,
+     0.00313427,
+     -0.02228454,
+     0.0249339,
+     0.00893045,
+     0.0193599,
+     -0.00423225,
+     0.01214966,
+     0.00211539,
+     -0.01705077,
+     0.05321393,
+     0.00832467,
+     0.00691531,
+     0.00440044,
+     -0.0036028,
+     0.00648574,
+     -0.00184864,
+     0.01262479,
+     -0.04579851,
+     -0.01826578,
+     -0.00640297,
+     -0.03256388,
+     0.00699382,
+     -0.01327709,
+     -0.01331763,
+     -0.00193405,
+     -0.00183068,
+     -0.00744029,
+     0.00887535,
+     -0.03525895,
+     -0.00613091,
+     0.00389063,
+     0.0061255,
+     -0.00145237,
+     0.02164902,
+     0.00551578,
+     -0.02099774,
+     0.02638274,
+     -0.01244471,
+     0.01595411,
+     0.00367786,
+     0.00150957,
+     0.01250048,
+     0.01080034,
+     -0.00269924,
+     -0.01266212,
+     -0.02325669,
+     -0.00212248,
+     0.05340743,
+     -0.00488497,
+     -0.00560996,
+     -0.01019677,
+     -0.00038237,
+     -0.01917257,
+     0.00416132,
+     -0.06233919,
+     0.02279506,
+     0.01383749,
+     0.0223188,
+     0.03706111,
+     0.00759379,
+     -0.01933291,
+     -0.00911448,
+     -0.00832836,
+     0.00218727,
+     0.00528866,
+     0.0012676,
+     -0.00689148,
+     -0.01066677,
+     -0.03458137,
+     -0.00148494,
+     0.00298672,
+     -0.02046814,
+     0.01484077,
+     -0.00187794,
+     0.00895328,
+     0.01091576,
+     0.02540395,
+     0.01473134,
+     -0.01313322,
+     0.00882845,
+     0.00047043,
+     -0.03233552,
+     -0.00381481,
+     -0.00340058,
+     0.00045749,
+     0.01598782,
+     0.01730791,
+     0.03945119,
+     -0.00992342,
+     -0.04769488,
+     0.01061766,
+     -0.00912597,
+     -0.00186258,
+     -0.02074894,
+     -0.00693543,
+     0.00157427,
+     -0.02019794,
+     0.03320531,
+     0.00160058,
+     0.03022134,
+     0.01365437,
+     0.01773841,
+     -0.00126859,
+     -0.00960154,
+     0.00160947,
+     0.02864504,
+     0.00388886,
+     0.01610713,
+     0.02257901,
+     -0.01016495,
+     0.0029654,
+     -0.00020181,
+     0.00606368,
+     0.01182888,
+     0.01258298,
+     0.06290828,
+     0.02865976,
+     0.02483223,
+     0.00040623,
+     0.00811181,
+     -0.01250675,
+     -0.01186497,
+     -0.01285955,
+     0.01597996,
+     -0.04972612,
+     0.00089825,
+     0.00958841,
+     -0.02032982,
+     -0.00527198,
+     0.02549261,
+     0.04237567,
+     0.0055739,
+     -0.01983182,
+     -0.02598889,
+     0.01521344,
+     0.01940417,
+     0.00654279,
+     -0.01277678,
+     0.01001596,
+     0.00197745,
+     0.00052592,
+     0.01791021,
+     0.00403065,
+     0.01413665,
+     -0.00654267,
+     0.01739347,
+     -0.00468635,
+     0.01130318,
+     0.01190673,
+     0.00176921,
+     0.00215732,
+     -0.01801398,
+     0.00423987,
+     -0.02487228,
+     -0.03574106,
+     -0.03761583,
+     -0.01580498,
+     0.0112633,
+     -0.00802143,
+     0.04035542,
+     -0.02123511,
+     0.01391431,
+     0.02576919,
+     -0.01363103,
+     -0.01700656,
+     -0.03270056,
+     -0.01692577,
+     0.0040207,
+     -0.00770615,
+     -0.02023523,
+     -0.01287303,
+     -0.00399295,
+     -0.0097395,
+     -0.00157304,
+     0.01470541,
+     -0.02900858,
+     -0.01364948,
+     0.01622227,
+     0.04555156,
+     0.02300258,
+     0.00070742,
+     -0.03894363,
+     0.01395334,
+     -0.011304,
+     0.01820423,
+     0.02882553,
+     -0.0265112,
+     -0.02328012,
+     -0.0057346,
+     -0.0111037,
+     -0.00239438,
+     0.00067644,
+     -0.00284005,
+     0.02123305,
+     0.019265,
+     0.01552788,
+     -0.01986999,
+     0.03097746,
+     0.03139879,
+     0.01663943,
+     0.02280164,
+     0.01902663,
+     0.00619938,
+     -0.01090529,
+     -0.02094108,
+     0.01410988,
+     0.0052609,
+     -0.01526535,
+     0.03699815,
+     -0.03132536,
+     0.01252833,
+     -0.02781519,
+     -0.01810875,
+     -0.009691,
+     -0.02077244,
+     0.00926613,
+     -0.04685893,
+     -0.00726184,
+     -0.00891186,
+     -0.00769646,
+     0.01439862,
+     -0.02565044,
+     0.00493228,
+     0.00974353,
+     -0.00995644,
+     0.02049081,
+     -0.04404585,
+     0.00751193,
+     0.00638864,
+     0.00514695,
+     0.05242157,
+     0.00289454,
+     -0.00064507,
+     0.03833183,
+     -0.04490974,
+     -0.00162655,
+     -0.01159293,
+     0.01254596,
+     -0.01223043,
+     0.0235262,
+     0.02629913,
+     0.01253196,
+     -0.02080517,
+     -0.00689689,
+     -0.0068256,
+     0.01760788,
+     0.01303217,
+     -0.00408248,
+     0.01806383,
+     0.00275279,
+     -0.02245613,
+     -0.03216419,
+     -0.02313394,
+     0.01341674,
+     -0.02875509,
+     0.00821746,
+     -0.00258162,
+     -0.0166717,
+     -0.00257669,
+     -0.01655585,
+     0.01896282,
+     -0.01112964,
+     0.00746154,
+     0.01104538,
+     -0.02793135,
+     -0.00074981,
+     0.00759404,
+     0.04883965,
+     0.02882431,
+     -0.01466329,
+     -0.0079204,
+     -0.01327058,
+     0.01041078,
+     0.01971299,
+     -0.00478428,
+     -0.02979348,
+     -0.0033707,
+     -0.01569781,
+     0.00752487,
+     0.00428126,
+     0.02388609,
+     0.0027208,
+     -0.00341252,
+     -0.04374065,
+     -0.01116345,
+     -0.00532541,
+     0.02159592,
+     0.00406952,
+     0.00415196,
+     -0.0033014,
+     0.00208155,
+     -0.02914683,
+     0.02400334,
+     -0.00277937,
+     -0.01673185,
+     -0.01697779,
+     -0.02679257,
+     -0.0103722,
+     -0.03774866,
+     0.02358133,
+     -0.02143352,
+     -0.00031711,
+     -0.00352962,
+     -0.00598108,
+     0.01242257,
+     0.0164373,
+     0.00470733,
+     0.02708453,
+     -0.02024415,
+     0.01169108,
+     -0.03683765,
+     0.02083484,
+     0.00937352,
+     -0.02739683,
+     -0.0067206,
+     -0.01986139,
+     0.00547282,
+     -0.01628444,
+     0.00153831,
+     -0.00249261,
+     -0.02345225,
+     -0.01357254,
+     0.01328131,
+     0.00521324,
+     0.00352823,
+     -0.01862443,
+     -0.00389278,
+     0.00673196,
+     0.00429158,
+     0.00606037,
+     0.03048845,
+     -0.01037415,
+     -0.01091966,
+     0.0005297,
+     -0.0454992,
+     -0.03626728,
+     0.00287546,
+     0.00962111,
+     -0.01698189,
+     -0.00070894,
+     0.01815648,
+     -0.01495158,
+     0.01427327,
+     0.02984908,
+     -0.01813151,
+     -0.00496254,
+     0.0179225,
+     -0.00915224,
+     -0.03176316,
+     0.00395478,
+     0.00483918,
+     0.00103693,
+     -0.03188303,
+     0.00852776,
+     -0.00672803,
+     -0.01914714,
+     0.00857762,
+     -0.0096268,
+     0.00650117,
+     -0.01087384,
+     -0.011639,
+     0.01615791,
+     0.02781934,
+     -0.00755544,
+     -0.00014848,
+     -0.00793268,
+     0.00750446,
+     -0.0103245,
+     0.0168096,
+     -0.02983714,
+     0.03803321,
+     -0.01768607,
+     0.00270445,
+     0.01078355,
+     0.01566177,
+     -0.0133848,
+     0.01774806,
+     -0.00812243,
+     0.02237075,
+     0.01115108,
+     0.00381729,
+     0.01070505,
+     -0.03039744,
+     -0.04433396,
+     0.02080437,
+     -0.01398751,
+     0.02386202,
+     9.26e-05,
+     -0.01348085,
+     0.00854043,
+     0.03121264,
+     -0.00824868,
+     -0.0059332,
+     0.01327891,
+     -0.01346949,
+     -0.00999715,
+     -0.00842401,
+     -0.02804655,
+     -0.03686436,
+     0.02232404,
+     -0.0279701,
+     -0.02540337,
+     -0.00054304,
+     -0.00756712,
+     0.02793047,
+     -0.01626339,
+     -0.01397534,
+     0.02358809,
+     0.04023468,
+     -0.03251312,
+     -0.01672773,
+     0.00377267,
+     0.01258042,
+     -0.00385397,
+     -0.02168425,
+     -0.01490837,
+     -0.01752106,
+     -0.0036807,
+     -0.01502097,
+     -0.03276443,
+     -0.01766985,
+     -0.02547744,
+     -0.00919792,
+     -0.00224599,
+     0.0251513,
+     -0.01591504,
+     0.00427116,
+     -0.01306467,
+     0.01153746,
+     -0.02659429,
+     -0.00869732,
+     0.00801325,
+     -0.00534268,
+     0.00309692,
+     0.0168271,
+     0.00267658,
+     -0.00784185,
+     -0.01638011,
+     -0.00787242,
+     -0.01431224,
+     -0.00839908,
+     0.00343782,
+     0.0076508,
+     0.00512584,
+     -0.00750912,
+     0.01765133,
+     0.00432924,
+     -0.00314923,
+     -0.02161724,
+     0.02385022,
+     -0.02513087,
+     -0.01094419,
+     0.01210399,
+     -0.00285633,
+     0.00383794,
+     -0.03123457,
+     -0.01198237,
+     -0.01510271,
+     -0.01148189,
+     0.00541213,
+     -0.01798316,
+     0.00496864,
+     -0.0015806,
+     -0.01432069,
+     -0.02940245,
+     -0.01776116,
+     -0.00566956,
+     0.0015968,
+     -0.01688569,
+     -0.00069971,
+     -0.00806095,
+     0.01892859,
+     0.03505353,
+     0.00642723,
+     -0.00281842,
+     -0.03205081,
+     -0.01303378,
+     -0.00905993,
+     0.02895496,
+     0.00354829,
+     0.01735854,
+     0.00953999,
+     -0.01426204,
+     0.01120116,
+     -0.00881752,
+     0.02241647,
+     -0.01657692,
+     -0.00571749,
+     -0.04898428,
+     0.00551105,
+     -0.02055241,
+     -0.00990897,
+     0.00936325,
+     -0.0096167,
+     0.01104994,
+     0.01584025,
+     0.01587972,
+     0.00194767,
+     -0.00631707,
+     -0.04304481,
+     0.0018921,
+     0.01500908,
+     -0.01644222,
+     0.00668909,
+     -0.03100232,
+     -0.00048992,
+     0.0197435,
+     -0.01865795,
+     -0.00662279,
+     -0.00430736,
+     0.01438951,
+     -0.02331211,
+     0.00492494,
+     -0.01072574,
+     -0.00245177,
+     -0.00950786,
+     -0.00477532,
+     -0.0100874,
+     -0.01600529,
+     0.00570134,
+     -0.01740529,
+     0.03467067,
+     0.01419344,
+     0.00161084,
+     -0.00347943,
+     0.02319815,
+     -0.00585438,
+     0.00335634,
+     0.03062036,
+     -0.01164215,
+     0.00494848,
+     0.01392082,
+     -0.04225956,
+     -0.00249652,
+     -0.00907702,
+     0.03732198,
+     -0.00339302,
+     -0.01306331,
+     0.02235401,
+     -0.01099497,
+     -0.01200077,
+     -0.05103194,
+     -0.00398947,
+     -0.00825615,
+     0.02447357,
+     -0.00067779,
+     -0.00834806,
+     -0.0175427,
+     -0.01224584,
+     0.00016019,
+     0.01329831,
+     -0.0194385,
+     -0.01699858,
+     0.00808758,
+     -0.0110953,
+     -0.00470004,
+     0.0187731,
+     0.00288963,
+     -0.04354201,
+     -0.0111457,
+     0.0091344,
+     0.00545016
+    ],
+    "cols": [
+     1,
+     11,
+     13,
+     15,
+     17,
+     24,
+     34,
+     36,
+     41,
+     53,
+     58,
+     61,
+     63,
+     64,
+     66,
+     69,
+     76,
+     79,
+     92,
+     95,
+     97,
+     98,
+     99,
+     114,
+     119,
+     123,
+     135,
+     145,
+     146,
+     148,
+     152,
+     161,
+     171,
+     173,
+     177,
+     192,
+     246,
+     253,
+     259,
+     265,
+     268,
+     286,
+     287,
+     302,
+     316,
+     324,
+     343,
+     351,
+     355,
+     450,
+     500,
+     579,
+     602,
+     614,
+     795,
+     933,
+     1003,
+     1110,
+     2055,
+     2072,
+     2175,
+     2595,
+     3242,
+     3343
+    ]
+   }
+  }
+ },
+ "capB-a6": {
+  "about": "Lab 8 recipe B: u and 64 neurons from the base model's own loops (windows well before the loop), alpha 6",
+  "layers": {
+   "15": {
+    "alpha": 6,
+    "dir": [
+     -0.30149689,
+     0.02326461,
+     0.11765372,
+     0.06267165,
+     0.01956591,
+     -0.19726555,
+     -0.00503336,
+     -0.08789949,
+     -0.01279326,
+     0.04979974,
+     0.05555366,
+     0.06674751,
+     0.01169172,
+     0.17527959,
+     0.02598691,
+     0.15154752,
+     0.0898912,
+     0.15027089,
+     -0.15828934,
+     -0.06802011,
+     0.06483994,
+     0.09954681,
+     0.11248125,
+     0.10143356,
+     -0.03427665,
+     0.01130421,
+     0.03965155,
+     -0.00559169,
+     0.03277428,
+     0.06104312,
+     0.03284929,
+     -0.16260622,
+     0.02073556,
+     -0.05957358,
+     -0.08322065,
+     -0.01794445,
+     -0.04898045,
+     0.22294332,
+     -0.05124363,
+     -0.0231905,
+     -0.07554419,
+     0.01327294,
+     -0.05874912,
+     0.01641323,
+     0.02109385,
+     -0.00849336,
+     0.13678432,
+     -0.03208598,
+     0.04021065,
+     0.04547515,
+     -0.00561485,
+     0.00565074,
+     -0.04153953,
+     0.01654728,
+     0.029699,
+     0.08232442,
+     0.04345837,
+     0.06434583,
+     -0.06645264,
+     -0.02416982,
+     0.00042967,
+     -0.07840814,
+     -0.04635896,
+     0.01254184,
+     0.03929815,
+     0.01142548,
+     0.0396764,
+     -0.02334055,
+     -0.04343329,
+     -0.02657734,
+     0.02388055,
+     0.01532186,
+     -0.08158231,
+     -0.03766869,
+     -0.09986223,
+     0.01965467,
+     -0.03953091,
+     0.01490568,
+     0.0382948,
+     -0.01490955,
+     0.05087914,
+     0.0703787,
+     0.02271337,
+     -0.0555483,
+     0.00071212,
+     0.11734333,
+     0.06425657,
+     -0.02442849,
+     0.02610742,
+     0.05987949,
+     0.05375696,
+     -0.05153911,
+     -0.03357845,
+     0.02666253,
+     0.06517966,
+     -0.01237155,
+     -0.02563511,
+     0.01684695,
+     0.01155908,
+     0.00380022,
+     -0.0493474,
+     -0.00513339,
+     -0.01106284,
+     0.02526324,
+     -0.02684648,
+     0.00526816,
+     0.05152647,
+     0.01687455,
+     -0.03582645,
+     0.02094468,
+     0.02345143,
+     0.02298974,
+     0.00365907,
+     -0.02588971,
+     -0.01382359,
+     0.07575645,
+     -0.01029643,
+     -0.03188023,
+     -0.01856114,
+     0.05651155,
+     0.00849511,
+     -0.0009302,
+     -0.01166164,
+     0.0041843,
+     -0.05954583,
+     -0.00012167,
+     -0.02362552,
+     -0.03205946,
+     0.01003108,
+     0.05697318,
+     -0.02847577,
+     0.004791,
+     0.03201439,
+     0.01117549,
+     0.0077243,
+     -0.01911873,
+     0.0005097,
+     -0.05309742,
+     0.10262797,
+     0.00544503,
+     0.00667644,
+     0.0214761,
+     0.02873388,
+     0.02627211,
+     -0.03188273,
+     -0.02098046,
+     0.00611033,
+     0.01107177,
+     0.03323895,
+     -0.02206725,
+     -0.00462339,
+     0.00417812,
+     0.03195161,
+     0.00788598,
+     -0.01020054,
+     0.00322999,
+     -0.04070878,
+     0.0042287,
+     0.00340121,
+     -0.00629161,
+     0.06356988,
+     -0.03003905,
+     0.01948619,
+     0.02089134,
+     -0.02839906,
+     0.02209648,
+     0.04238123,
+     0.01715541,
+     -0.0301029,
+     0.00514945,
+     -0.05948887,
+     -0.04008217,
+     0.00092403,
+     0.04240754,
+     -0.01147224,
+     -0.05509264,
+     0.05763384,
+     -0.00159645,
+     0.01637939,
+     -0.02916958,
+     -0.03551837,
+     0.00899683,
+     0.0155799,
+     -0.02717732,
+     -0.02264439,
+     0.00593656,
+     -0.05256448,
+     -0.00168005,
+     0.03425383,
+     0.0188481,
+     -0.02734067,
+     0.01612632,
+     0.02995075,
+     0.0126488,
+     0.00824284,
+     0.02283906,
+     0.00149592,
+     -0.01262761,
+     0.01244001,
+     -0.04045067,
+     -0.00659995,
+     -0.01249472,
+     -0.0208294,
+     -0.00422333,
+     0.02912182,
+     0.04871593,
+     0.02018343,
+     -0.01012479,
+     -0.03434999,
+     -0.00024383,
+     0.00724176,
+     -0.0059202,
+     -0.03058464,
+     -0.00567434,
+     0.00418883,
+     0.0098441,
+     -0.00485016,
+     0.0235494,
+     -0.009295,
+     -0.06723159,
+     -0.00141925,
+     0.01058285,
+     0.02079017,
+     -0.00225317,
+     0.01032211,
+     -0.04980971,
+     0.00766713,
+     -0.02094317,
+     -0.04421069,
+     -0.02549951,
+     -0.02841922,
+     0.05762584,
+     0.00753723,
+     0.02754568,
+     -0.00603487,
+     -0.01353416,
+     0.01183222,
+     0.02611442,
+     -0.02576422,
+     0.02603181,
+     -0.01338155,
+     0.01783432,
+     0.01952534,
+     -0.01827757,
+     0.04060069,
+     0.0096524,
+     0.00621415,
+     -0.07070343,
+     0.01205619,
+     -0.0157654,
+     0.0355527,
+     -0.00870733,
+     -0.04851521,
+     0.00418431,
+     0.01064662,
+     0.08757313,
+     -0.0431821,
+     -0.02164889,
+     -0.00494927,
+     -0.00274217,
+     -0.00344297,
+     0.02221851,
+     -0.00939169,
+     -0.00249658,
+     0.01415563,
+     0.00960531,
+     0.02835243,
+     -0.00237309,
+     0.00040106,
+     -0.00386805,
+     -0.00909662,
+     -0.01428644,
+     0.00019036,
+     -0.03145899,
+     -0.00120665,
+     0.02071353,
+     -0.02028685,
+     -0.00615535,
+     -0.02520839,
+     -0.02236454,
+     0.02357376,
+     -0.01185053,
+     -0.0559378,
+     0.02748231,
+     0.02343067,
+     0.00186691,
+     0.04614904,
+     0.00788556,
+     -0.01367954,
+     0.01103815,
+     0.0144666,
+     -0.00568829,
+     -0.02537256,
+     -0.00291896,
+     -0.01370349,
+     -0.01813706,
+     0.00659186,
+     -0.01144869,
+     0.0054018,
+     -0.02677543,
+     0.15799721,
+     -0.00068407,
+     0.00293215,
+     0.04486345,
+     -0.00954056,
+     0.01474388,
+     0.01060703,
+     0.00755214,
+     0.00345063,
+     0.01092949,
+     -0.00362884,
+     0.01473075,
+     -0.00538783,
+     0.01691999,
+     -0.01394772,
+     0.02865278,
+     -0.07491908,
+     -0.05841815,
+     -0.02911618,
+     0.03960273,
+     -0.00325529,
+     0.01922638,
+     0.03592266,
+     0.01240769,
+     0.00643024,
+     -0.03352727,
+     0.02750751,
+     0.00443367,
+     0.01995698,
+     -0.00748279,
+     0.01875017,
+     -0.00788494,
+     0.00522293,
+     0.03003652,
+     0.02110308,
+     -0.02318322,
+     0.02898433,
+     0.00738792,
+     0.00277897,
+     0.02434439,
+     -0.09862508,
+     0.05617183,
+     -0.02186624,
+     -0.00717736,
+     -0.01975846,
+     0.0338889,
+     0.02121268,
+     0.00846916,
+     -0.00983612,
+     0.03011166,
+     -0.00570846,
+     0.00319565,
+     -0.01916552,
+     -0.00642473,
+     -0.02169416,
+     -0.05596741,
+     -0.00093535,
+     -0.02283115,
+     0.02040694,
+     -0.01648783,
+     0.006206,
+     0.0008686,
+     0.03457417,
+     -0.02280811,
+     -0.00063133,
+     0.02138565,
+     0.00825628,
+     0.00704255,
+     0.01895653,
+     0.01850315,
+     -0.01492665,
+     -0.00377653,
+     -0.00726489,
+     0.01177038,
+     -0.02597683,
+     -0.01700044,
+     -0.00285292,
+     0.00554924,
+     0.00484764,
+     -0.01376073,
+     -0.0139686,
+     -0.01939424,
+     -0.02597563,
+     -0.01605687,
+     0.00347986,
+     -0.02692802,
+     0.00248492,
+     0.01662239,
+     0.00825688,
+     -0.00402666,
+     -0.0170685,
+     0.00427057,
+     0.00054583,
+     0.01124339,
+     -0.02183045,
+     0.03154991,
+     0.0003381,
+     -0.00562663,
+     -0.00763783,
+     0.00563474,
+     -0.02693614,
+     0.01886536,
+     0.01280506,
+     0.00324432,
+     -0.03816755,
+     -0.03527931,
+     -0.03196836,
+     0.00196033,
+     0.0196076,
+     -0.00196445,
+     0.02511599,
+     -0.02921427,
+     0.0061164,
+     -0.01979781,
+     -0.0021651,
+     0.01065994,
+     -0.03970449,
+     -0.03330931,
+     -0.01952009,
+     0.00032798,
+     -0.00533792,
+     0.0124535,
+     -0.00392792,
+     -0.01070111,
+     -0.08814324,
+     0.0076339,
+     0.01122764,
+     0.00905735,
+     0.00117328,
+     0.01181389,
+     -0.02175004,
+     -0.02121019,
+     0.01248304,
+     -0.00093594,
+     -0.01948054,
+     0.03104073,
+     -0.0088424,
+     0.01062842,
+     0.03344615,
+     0.02377078,
+     0.00947216,
+     0.04116111,
+     0.01839026,
+     -0.00920545,
+     0.0132251,
+     0.02180791,
+     0.03044661,
+     0.00876045,
+     0.0206499,
+     0.01138573,
+     0.02535999,
+     0.00895611,
+     -0.00902818,
+     0.02495545,
+     0.00332173,
+     0.00221947,
+     -0.02037391,
+     -0.00544367,
+     0.03596655,
+     0.00752132,
+     -0.00227448,
+     0.0155155,
+     0.00425131,
+     -0.01969209,
+     0.00654938,
+     0.01465121,
+     0.02098612,
+     -0.00871356,
+     0.00285627,
+     -0.00237169,
+     -0.01567797,
+     0.05599053,
+     -0.00078066,
+     -0.0099316,
+     0.01758756,
+     -0.0201795,
+     0.00143664,
+     0.01996665,
+     0.00643219,
+     -0.03766159,
+     -0.01698383,
+     0.00196764,
+     -0.02466804,
+     0.00456563,
+     -0.01729585,
+     -0.00617461,
+     0.0079403,
+     0.02798674,
+     -0.00546738,
+     -0.00203452,
+     -0.01545207,
+     -0.01398242,
+     -0.03375365,
+     0.00621148,
+     0.02407432,
+     0.0216948,
+     0.02371102,
+     -0.0017222,
+     0.03594714,
+     0.00673465,
+     0.0023872,
+     -0.00644301,
+     -0.00486257,
+     0.00653148,
+     0.00686671,
+     0.01289318,
+     0.01043915,
+     0.0008696,
+     -0.0029646,
+     0.00058893,
+     -0.00367669,
+     -0.01205821,
+     -0.00369951,
+     0.00500719,
+     -0.03102004,
+     0.00997901,
+     -0.0564148,
+     0.0139201,
+     0.00455478,
+     0.02063775,
+     0.0524075,
+     0.0090106,
+     -0.0040679,
+     0.00929408,
+     -0.01237593,
+     0.01192695,
+     0.0002889,
+     0.04478727,
+     -0.00868966,
+     -0.01069027,
+     0.00170744,
+     -0.00880913,
+     0.03051276,
+     -0.01133682,
+     0.00216336,
+     -0.01166068,
+     0.02527721,
+     0.01598965,
+     0.05630016,
+     0.01782241,
+     0.00794784,
+     -0.00473622,
+     -0.00324297,
+     -0.03602727,
+     -0.01093641,
+     -0.00714647,
+     -0.0240199,
+     0.00720374,
+     -0.0018791,
+     0.00775742,
+     -0.00429064,
+     -0.05634158,
+     0.02373968,
+     -0.00022325,
+     0.017044,
+     -0.0415827,
+     0.01951932,
+     -0.00684324,
+     -0.00617908,
+     0.02936001,
+     0.0022231,
+     0.02858322,
+     0.02646914,
+     0.0296457,
+     -0.01347029,
+     0.00532766,
+     -0.01175069,
+     -0.01503773,
+     -0.00244769,
+     0.00162901,
+     0.01375139,
+     -0.00926648,
+     -0.00245956,
+     0.00333973,
+     -0.01736316,
+     -0.00126767,
+     0.00376091,
+     0.00699914,
+     0.02746248,
+     0.01196896,
+     0.01478108,
+     -0.00855968,
+     -0.00260315,
+     -0.0199121,
+     -0.01861352,
+     0.01223358,
+     0.0081777,
+     0.00620991,
+     0.01076913,
+     -0.02556662,
+     0.00788654,
+     0.01794371,
+     0.02445308,
+     -0.00098348,
+     -0.00087275,
+     -0.00806301,
+     0.00444915,
+     -0.01069149,
+     0.01613029,
+     0.00309012,
+     0.01902706,
+     0.01421466,
+     0.00138022,
+     0.01512942,
+     0.00030636,
+     0.03271034,
+     -0.01789483,
+     0.02173513,
+     -0.00648184,
+     -0.0162087,
+     0.0077972,
+     -0.00579106,
+     0.00537245,
+     0.00387916,
+     0.01599543,
+     0.00627289,
+     -0.01186336,
+     0.0070736,
+     -0.01310907,
+     0.02760035,
+     0.00439267,
+     0.03422385,
+     -0.00883535,
+     0.00667326,
+     0.02595069,
+     0.00116098,
+     -0.01699736,
+     -0.03703247,
+     -0.03113385,
+     0.02031528,
+     -0.00775315,
+     -0.00877325,
+     0.01849073,
+     0.01423547,
+     0.00288525,
+     -0.00622111,
+     0.02096023,
+     -0.01215921,
+     -9e-05,
+     0.0382315,
+     0.02604484,
+     0.01930726,
+     0.01248157,
+     -0.03497124,
+     0.01475701,
+     3.174e-05,
+     0.01543846,
+     0.02304812,
+     -0.0129609,
+     -0.02369003,
+     0.00373719,
+     0.00182831,
+     0.00434628,
+     0.00474514,
+     0.00282092,
+     0.01340202,
+     0.01224109,
+     0.02576358,
+     -0.01278894,
+     0.00098902,
+     0.0173066,
+     0.00088813,
+     0.01500969,
+     0.00536499,
+     0.02194963,
+     0.00571928,
+     0.01032218,
+     0.01571402,
+     0.0092924,
+     -0.015971,
+     0.00404453,
+     -0.02361519,
+     -0.00444758,
+     -0.0045476,
+     -0.00217738,
+     -0.02578217,
+     -0.01997341,
+     -0.01279157,
+     -0.0286952,
+     -0.00561124,
+     -0.00671502,
+     0.01089097,
+     0.0030987,
+     -0.0345054,
+     -0.0087018,
+     0.00565438,
+     0.02648221,
+     0.00331194,
+     -0.0136128,
+     0.00805192,
+     0.02403069,
+     0.01376859,
+     -0.0009556,
+     0.01622848,
+     -0.00585403,
+     -0.02126574,
+     -0.00190116,
+     0.01683871,
+     0.01021804,
+     -0.00683891,
+     0.01565132,
+     0.00421597,
+     0.03681558,
+     0.0086617,
+     0.04103243,
+     -0.00636359,
+     0.01988558,
+     -0.00700619,
+     0.01157968,
+     -0.01269246,
+     0.02173179,
+     -0.00022381,
+     -0.01735442,
+     -0.01220535,
+     -0.02978618,
+     0.00487368,
+     -0.00201816,
+     -0.0065471,
+     0.01840364,
+     -0.0068554,
+     0.00667753,
+     0.01900262,
+     0.0079251,
+     -0.00720076,
+     0.0348928,
+     0.03306156,
+     -0.02104476,
+     0.01431296,
+     0.01034307,
+     0.03554063,
+     0.01929511,
+     -0.03064406,
+     0.00852042,
+     0.01728075,
+     0.01276018,
+     0.01369407,
+     -0.00783796,
+     -0.00288487,
+     -0.02263158,
+     -0.01975801,
+     -0.00824996,
+     -0.01331091,
+     -0.03441912,
+     0.00086305,
+     -0.00338286,
+     -0.04317742,
+     -0.01347583,
+     -0.00972787,
+     0.00519537,
+     -0.00305993,
+     0.003299,
+     -0.02064933,
+     -0.00521341,
+     0.0082595,
+     0.0147135,
+     -0.02325539,
+     -0.02601415,
+     -0.01221036,
+     -0.02321008,
+     -0.03255986,
+     0.00117438,
+     0.0051252,
+     -0.02630395,
+     0.01018165,
+     -0.00126909,
+     0.00018002,
+     0.0206973,
+     -0.0103088,
+     -0.01774329,
+     0.02208793,
+     0.01005429,
+     -0.00241477,
+     -0.03558522,
+     0.02600474,
+     0.01392036,
+     0.0007913,
+     0.00867003,
+     -0.02678201,
+     0.00417043,
+     -0.00349747,
+     0.02998451,
+     -0.0076973,
+     -0.0183407,
+     -0.01780671,
+     0.01216279,
+     0.02785827,
+     0.00912529,
+     0.00922449,
+     -0.01494971,
+     0.00022192,
+     -0.00362771,
+     0.01362053,
+     0.01309695,
+     0.01269907,
+     0.0076792,
+     -0.00546721,
+     -0.00254069,
+     0.01004402,
+     -0.00896631,
+     -0.0116364,
+     -0.00836369,
+     0.00731342,
+     0.01272695,
+     -0.01767886,
+     0.02485484,
+     0.02326213,
+     -0.04928953,
+     -0.01117503,
+     0.02420524,
+     0.0063097,
+     -0.01632563,
+     0.00135226,
+     -0.02587743,
+     0.01592461,
+     0.00361057,
+     0.01816907,
+     -0.0003399,
+     -0.01344171,
+     0.02377179,
+     0.00842966,
+     -0.0107262,
+     0.03317228,
+     0.00435296,
+     0.01918977,
+     0.04618406,
+     -0.00789822,
+     0.0302141,
+     0.01307326,
+     0.01281024,
+     -0.01703183,
+     0.01827971,
+     -0.01626707,
+     0.0262253,
+     0.00052332,
+     -0.00336457,
+     -0.00179358,
+     0.00217421,
+     -0.02386117,
+     -0.00263912,
+     0.01809951,
+     0.00450152,
+     0.01284222,
+     -0.03143772,
+     0.01057556,
+     -0.04545506,
+     -0.02452004,
+     -0.00122417,
+     -0.0313464,
+     0.03283778,
+     -0.03497106,
+     0.00327116,
+     0.00402971,
+     0.00810157,
+     0.00124328,
+     -0.00593415,
+     0.01207904,
+     -0.01564144,
+     -0.01598497,
+     0.00245661,
+     -0.01651567,
+     -0.0364025,
+     0.016465,
+     0.01770067,
+     -0.01766352,
+     0.02599589,
+     0.00321878,
+     0.02408964,
+     -0.0048591,
+     -0.02150145,
+     0.02837163,
+     -0.00355406,
+     -0.01425166,
+     -0.01394022,
+     -0.0054518,
+     0.00785455,
+     0.00858407,
+     -0.00508778,
+     -0.02234841,
+     -0.04273375,
+     0.03256549,
+     -0.00324295,
+     -0.01093053,
+     0.00072815,
+     -0.0119415,
+     -0.00664757,
+     0.00496523,
+     0.00561876,
+     0.01784011,
+     -0.01179754,
+     -0.02320364,
+     -0.02545631,
+     0.00336456,
+     0.00665186,
+     -0.00832409,
+     -0.01245756,
+     -0.01078675,
+     0.01177812,
+     0.0095338,
+     -0.00209641,
+     0.01275474,
+     0.03254699,
+     -0.00988943,
+     -0.00097725,
+     -0.00635651,
+     -0.00269509,
+     -0.0119646,
+     -0.00504241,
+     0.0089832,
+     0.01423179,
+     -0.01198032,
+     0.01199757,
+     0.02696822,
+     -0.02898989,
+     0.01225323,
+     0.00347795,
+     -0.01845676,
+     -0.00278158,
+     -0.00845704,
+     -0.01078037,
+     -0.00613572,
+     0.00558208,
+     0.00566338,
+     -0.00973258,
+     -0.01901137,
+     0.0017411,
+     -0.03024946,
+     -0.00683895,
+     0.00152097,
+     0.00843003,
+     0.00574749,
+     -0.00256013,
+     -0.01349887,
+     0.00611601,
+     0.01228548,
+     0.00169058,
+     0.00807445,
+     0.00318968,
+     -0.0221693,
+     -0.01783665,
+     0.01726582,
+     0.01524275,
+     -0.00177572,
+     0.02865437,
+     0.01760579,
+     -0.01952672,
+     -0.01631412,
+     0.01287163,
+     0.00356905,
+     -0.00483905,
+     -0.0090318,
+     -0.04754067,
+     -0.011511,
+     -0.01697593,
+     -0.02475849,
+     0.02553634,
+     -0.02044728,
+     0.00090611,
+     0.00646554,
+     -0.01823398,
+     -0.01946126,
+     -0.0062193,
+     -0.00040626,
+     -0.0193678,
+     -0.0343605,
+     0.00704293,
+     0.01018505,
+     -0.01024959,
+     0.01194722,
+     0.02828199,
+     -0.03054512,
+     0.00385779,
+     -0.02005518,
+     0.01599522,
+     0.00011745,
+     0.02317094,
+     -0.00441486,
+     0.01399615,
+     0.02148392,
+     -0.0180247,
+     0.00072905,
+     -0.00508529,
+     -0.0009809,
+     -0.01381451,
+     0.02399625,
+     0.02420016,
+     -0.00935192,
+     0.00158473,
+     0.02889357,
+     0.0075459,
+     0.03266473,
+     0.00581206,
+     -0.00822044,
+     -0.01659222,
+     -0.02766358,
+     -0.01591903,
+     0.0120321,
+     0.006772,
+     -0.00306966,
+     -0.00225038,
+     0.00448638,
+     0.00676812,
+     -0.02272712,
+     -0.01672792,
+     -0.03724678,
+     -0.03443447,
+     -0.00684222,
+     -0.00061584,
+     0.00043113,
+     -0.00941609,
+     -0.02079861,
+     -0.0218753,
+     0.0014661,
+     0.02277899,
+     -0.03370588,
+     -0.04389111,
+     -0.01075738,
+     -0.02132445,
+     -0.0392971,
+     -0.00546209,
+     -0.01508621,
+     -0.02128147,
+     0.01646507,
+     -0.01964691,
+     -0.01943282
+    ],
+    "cols": [
+     0,
+     11,
+     25,
+     35,
+     37,
+     38,
+     39,
+     42,
+     46,
+     50,
+     56,
+     61,
+     63,
+     65,
+     67,
+     68,
+     79,
+     105,
+     129,
+     135,
+     169,
+     215,
+     259,
+     264,
+     266,
+     294,
+     371,
+     412,
+     414,
+     426,
+     452,
+     496,
+     504,
+     505,
+     509,
+     516,
+     525,
+     580,
+     582,
+     601,
+     638,
+     682,
+     733,
+     738,
+     750,
+     882,
+     892,
+     926,
+     1060,
+     1087,
+     1130,
+     1155,
+     1184,
+     1321,
+     1702,
+     1787,
+     2005,
+     2066,
+     2248,
+     2252,
+     2288,
+     2431,
+     2612,
+     3003
+    ]
+   },
+   "19": {
+    "alpha": 6,
+    "dir": [
+     -0.49866527,
+     0.04483794,
+     0.08653121,
+     0.04403261,
+     0.06070542,
+     -0.18010071,
+     0.02048543,
+     -0.01286332,
+     -0.00027879,
+     0.01557074,
+     0.0579919,
+     0.03242711,
+     0.04437343,
+     0.11831202,
+     0.04191576,
+     0.1096428,
+     0.08484758,
+     0.19838709,
+     -0.15137778,
+     -0.06891333,
+     0.09073912,
+     0.05171521,
+     0.11006631,
+     0.0956018,
+     -0.03853878,
+     0.00064215,
+     0.0313293,
+     0.01706532,
+     0.01163942,
+     0.03902397,
+     0.02441614,
+     -0.15192458,
+     0.05112223,
+     -0.04017477,
+     -0.0822874,
+     0.0093005,
+     -0.00961613,
+     0.16345182,
+     -0.06426762,
+     -0.01643771,
+     -0.07640368,
+     0.00049388,
+     -0.02564682,
+     -0.01009604,
+     -0.01522518,
+     -0.02150765,
+     0.0845025,
+     -0.04340943,
+     0.04411089,
+     0.08151267,
+     -0.00340664,
+     0.00123752,
+     -0.0378892,
+     0.00448848,
+     0.03228236,
+     0.07224466,
+     0.04673243,
+     0.05504248,
+     -0.05127607,
+     -0.03776409,
+     0.02176227,
+     -0.08630367,
+     -0.04531471,
+     0.0296373,
+     0.06976917,
+     0.0295671,
+     -0.00246833,
+     -0.02473946,
+     -0.05511018,
+     -0.03542427,
+     0.02730425,
+     0.01280087,
+     -0.07081246,
+     -0.00496536,
+     -0.0890561,
+     0.00800353,
+     -0.05013782,
+     0.02035334,
+     -0.0028146,
+     0.00412351,
+     0.05471795,
+     0.06011825,
+     0.02483712,
+     -0.04671688,
+     0.02074336,
+     0.09762273,
+     0.08155826,
+     -0.02676294,
+     0.02521992,
+     0.03532628,
+     0.04708899,
+     -0.05322118,
+     -0.05567474,
+     -0.00651496,
+     0.04405286,
+     -0.01254328,
+     -0.02278901,
+     -0.00852609,
+     0.00359337,
+     0.00812188,
+     -0.05064651,
+     -0.01694964,
+     -0.00542477,
+     0.03623243,
+     -0.02757451,
+     0.0051043,
+     0.03470914,
+     0.00490483,
+     -0.03334099,
+     0.00603678,
+     0.0300563,
+     0.04627726,
+     0.00187998,
+     -0.0416753,
+     -0.0090613,
+     0.05596276,
+     -0.02103174,
+     -0.06238985,
+     -0.01180217,
+     0.04171294,
+     0.01249574,
+     0.02816214,
+     -0.0176421,
+     0.01949446,
+     -0.03917667,
+     0.00881786,
+     -0.02624751,
+     0.00509142,
+     -0.0077283,
+     0.06086404,
+     -0.02195361,
+     0.00030291,
+     0.03111923,
+     0.03329645,
+     0.01400976,
+     -0.01356143,
+     -0.00565462,
+     -0.06398953,
+     0.0551735,
+     -0.00848315,
+     0.00687203,
+     0.01781475,
+     0.05540646,
+     0.03959751,
+     -0.01159826,
+     -0.01160534,
+     0.0048661,
+     -0.00608777,
+     0.02319413,
+     -0.01162495,
+     -0.0077591,
+     0.01756361,
+     0.02298547,
+     0.02433926,
+     0.01067325,
+     0.00118467,
+     -0.02158724,
+     0.00489646,
+     0.00243894,
+     -0.01124737,
+     0.06788686,
+     -0.00478992,
+     0.00929579,
+     0.02146945,
+     -0.02837166,
+     0.00487398,
+     0.00094381,
+     -0.03970704,
+     -0.03389104,
+     -0.00084383,
+     -0.02727271,
+     -0.02561967,
+     0.00955132,
+     0.01581585,
+     -0.00645247,
+     -0.03326835,
+     0.04197594,
+     -0.00990736,
+     0.02483497,
+     -0.01266322,
+     -0.02937236,
+     0.02056376,
+     -0.00301939,
+     -0.00960709,
+     -0.02398397,
+     -0.01240014,
+     -0.04018981,
+     -0.01065147,
+     0.02541097,
+     0.01085518,
+     -0.04091425,
+     0.01861534,
+     0.02755524,
+     0.01442517,
+     -0.00980565,
+     0.01643772,
+     0.0103014,
+     -0.00193121,
+     -0.01158493,
+     -0.04491895,
+     0.00210387,
+     -0.0158687,
+     -0.01925263,
+     0.00858816,
+     0.00823828,
+     0.05501426,
+     0.01001832,
+     -0.01590634,
+     -0.04254342,
+     -0.00362291,
+     -0.00355009,
+     0.00977466,
+     -0.04294737,
+     -0.01187379,
+     -0.00624066,
+     -0.00100696,
+     0.01209668,
+     0.01955999,
+     0.00612554,
+     -0.09073611,
+     -0.0121392,
+     -0.00602088,
+     0.02669331,
+     -0.01423631,
+     0.02193962,
+     -0.06434545,
+     -0.00057406,
+     -0.00482061,
+     -0.0240023,
+     -0.01730228,
+     -0.04747942,
+     0.05685804,
+     0.00550534,
+     0.0255143,
+     0.00708329,
+     -0.01461607,
+     0.00074817,
+     0.01754509,
+     -0.02909672,
+     0.04151966,
+     -0.00419367,
+     0.0328035,
+     0.01885574,
+     -0.00191183,
+     0.03820024,
+     0.00341159,
+     0.0167866,
+     -0.05124178,
+     -0.01221086,
+     -0.00337703,
+     0.03029857,
+     -0.00824676,
+     -0.02232636,
+     -0.00764648,
+     0.00037306,
+     0.08489247,
+     -0.04158763,
+     -0.02465037,
+     0.0038072,
+     0.00986983,
+     -0.02635114,
+     -0.00054587,
+     0.01122991,
+     0.01526793,
+     0.01308451,
+     0.00947611,
+     0.01079197,
+     0.00301678,
+     0.00045249,
+     0.01157898,
+     0.00364791,
+     -0.01974811,
+     0.00853061,
+     -0.03393469,
+     -0.02398907,
+     0.01186427,
+     0.00773612,
+     0.00182465,
+     -0.03346312,
+     -0.05178841,
+     0.01664432,
+     -0.00867638,
+     -0.04087122,
+     0.00350367,
+     0.01055847,
+     -0.01330485,
+     0.0321027,
+     -0.01336184,
+     -0.00791821,
+     -0.01172312,
+     0.01268876,
+     -0.01336592,
+     -0.01299944,
+     0.01018297,
+     0.00367312,
+     -0.02798145,
+     0.01043847,
+     -0.00059207,
+     -0.02061029,
+     -0.02362762,
+     0.08393674,
+     -0.0145678,
+     -0.01420483,
+     0.05421852,
+     -0.01871055,
+     0.01786913,
+     -0.00289787,
+     -0.01177605,
+     -0.01852703,
+     0.03584592,
+     -0.00914478,
+     -0.03144572,
+     -0.00690102,
+     0.01117276,
+     -0.01311047,
+     0.02009693,
+     -0.07556609,
+     -0.06147706,
+     0.00172751,
+     0.03443452,
+     0.01491988,
+     0.00528303,
+     0.01334639,
+     0.00624813,
+     -0.01138349,
+     -0.01042592,
+     0.02700793,
+     0.00862187,
+     0.01563637,
+     -0.02544332,
+     0.03750516,
+     -0.01093695,
+     0.01380753,
+     0.04961452,
+     0.01869025,
+     -0.00684568,
+     0.00716816,
+     0.00134637,
+     -0.00351985,
+     0.03323479,
+     -0.10388357,
+     0.02342377,
+     -0.02660153,
+     -0.00555478,
+     -0.0143756,
+     0.00243205,
+     0.00385063,
+     -0.00998113,
+     -0.02333265,
+     0.02243167,
+     -0.02253078,
+     -0.00044902,
+     -0.01409877,
+     -0.00337895,
+     -0.02913127,
+     -0.03442011,
+     -0.00535547,
+     -0.01662975,
+     0.01690075,
+     -0.02125359,
+     0.0155171,
+     0.0064527,
+     0.01973493,
+     -0.02252325,
+     0.01239673,
+     0.02198475,
+     -0.01154951,
+     -0.00176781,
+     0.03128855,
+     0.0164711,
+     -0.00352959,
+     -0.00919291,
+     -0.00051269,
+     0.00272963,
+     -0.01334997,
+     -0.02031143,
+     -0.00102859,
+     0.01847753,
+     -0.00953855,
+     -0.0136291,
+     -0.00146926,
+     0.00225613,
+     -0.01099077,
+     -0.0059908,
+     -0.00535733,
+     -0.01735635,
+     0.00250816,
+     0.01244467,
+     0.01221272,
+     0.00098979,
+     -0.00353166,
+     -0.00583052,
+     0.00087737,
+     0.00923937,
+     0.00468853,
+     0.03266098,
+     -0.00799337,
+     -0.00690205,
+     -0.00479467,
+     0.00531395,
+     -0.00968522,
+     -0.00285753,
+     0.00617076,
+     -0.00398007,
+     -0.00798379,
+     -0.01925957,
+     -0.04746011,
+     0.01850612,
+     -0.00163182,
+     -0.00469556,
+     0.02426587,
+     -0.02222052,
+     0.00818618,
+     -0.01007741,
+     -0.02858347,
+     -0.00121701,
+     -0.06705105,
+     -0.01600351,
+     -0.02505735,
+     -0.00761152,
+     -0.01960301,
+     -0.00975214,
+     -0.01505511,
+     -0.02469417,
+     -0.10968003,
+     0.01911923,
+     0.00610741,
+     0.00117041,
+     0.00868673,
+     -0.0041148,
+     -0.02288475,
+     0.0138887,
+     0.00516456,
+     -0.00078531,
+     0.00329392,
+     0.03360431,
+     -0.00695994,
+     0.00668374,
+     0.01618747,
+     0.01681385,
+     -0.00357584,
+     -0.01117303,
+     -0.01453918,
+     -0.01406458,
+     0.01796291,
+     0.02957689,
+     0.01712581,
+     0.02696638,
+     0.00489161,
+     -0.0027144,
+     0.00797566,
+     0.01722849,
+     -0.01348964,
+     -0.00059619,
+     0.0062006,
+     -0.01288154,
+     -0.01594781,
+     0.00060741,
+     0.03926362,
+     0.000648,
+     -0.00758998,
+     0.00969625,
+     0.01107869,
+     -0.00975366,
+     0.01968245,
+     0.01074674,
+     0.0149546,
+     -0.00488283,
+     0.01590041,
+     -0.00571848,
+     -0.01415043,
+     0.04789799,
+     0.00476613,
+     0.01109879,
+     -0.00158805,
+     -0.01026194,
+     0.00513164,
+     0.00177138,
+     0.01509304,
+     -0.04241737,
+     -0.02120126,
+     -0.00480244,
+     -0.0315622,
+     0.00439782,
+     -0.00950842,
+     -0.01790279,
+     -0.01042813,
+     -0.0103942,
+     -0.00075724,
+     0.00224846,
+     -0.03605071,
+     -0.00766446,
+     0.00278169,
+     0.00702316,
+     -0.00182345,
+     0.01892289,
+     0.00708544,
+     -0.01113682,
+     0.03328981,
+     -0.0192459,
+     0.0140041,
+     0.00527974,
+     -0.01008013,
+     0.01208155,
+     0.00602798,
+     -0.00538647,
+     -0.00988771,
+     -0.01595749,
+     0.00868384,
+     0.03627389,
+     0.00344467,
+     -0.01577122,
+     -4.978e-05,
+     -0.00462439,
+     -0.01982905,
+     0.00348027,
+     -0.06243323,
+     0.01559778,
+     0.00743285,
+     0.01567806,
+     0.03248803,
+     0.00350522,
+     -0.01878736,
+     -0.00510001,
+     -0.01560209,
+     0.00595972,
+     0.00423453,
+     -0.00234233,
+     0.00050524,
+     -0.00564994,
+     -0.01667203,
+     -0.00596002,
+     0.01465663,
+     -0.01919699,
+     0.01211679,
+     0.00254716,
+     0.01503418,
+     0.0095837,
+     0.02680599,
+     0.01051945,
+     -0.00725303,
+     -0.00185183,
+     -0.00087937,
+     -0.03413209,
+     -0.00287935,
+     -0.0058744,
+     -0.00796591,
+     0.02322935,
+     0.01579706,
+     0.02696734,
+     -0.00191688,
+     -0.04072197,
+     0.01511676,
+     -0.00058316,
+     -0.00465454,
+     -0.02495283,
+     -0.01099585,
+     0.0051539,
+     -0.01437707,
+     0.02481943,
+     -0.00748865,
+     0.01838564,
+     0.01209936,
+     0.01926963,
+     -0.00329364,
+     -0.00074246,
+     -0.00170502,
+     0.01563909,
+     0.00117787,
+     0.00813994,
+     0.01232119,
+     -0.0040677,
+     0.00620609,
+     0.01448978,
+     0.00922149,
+     0.01071046,
+     0.01238525,
+     0.05554213,
+     0.02819438,
+     0.02504348,
+     -0.00285749,
+     0.00558255,
+     -0.01137746,
+     -0.0158543,
+     -0.0189033,
+     0.0121513,
+     -0.03731599,
+     -0.00747662,
+     0.00543316,
+     -0.01170763,
+     -0.00087092,
+     0.01825779,
+     0.03395717,
+     0.01406542,
+     -0.0175342,
+     -0.01369371,
+     0.00817646,
+     0.02041542,
+     -0.00884456,
+     -0.00418454,
+     0.01211634,
+     -0.00276281,
+     -0.00371123,
+     0.01992132,
+     0.00124228,
+     0.01579323,
+     -0.01380612,
+     0.00651301,
+     -0.00965464,
+     0.00304045,
+     0.00938329,
+     0.00761521,
+     0.00207386,
+     -0.01233432,
+     -0.00210403,
+     -0.01623478,
+     -0.02811889,
+     -0.02598538,
+     -0.01444949,
+     0.01080103,
+     0.00122658,
+     0.02289422,
+     -0.01870971,
+     0.01919318,
+     0.03217082,
+     -0.00920643,
+     -0.0215708,
+     -0.03158914,
+     -0.02056642,
+     0.01231547,
+     -0.00952645,
+     -0.01507373,
+     0.00180383,
+     0.00179136,
+     -0.00920329,
+     -0.00321118,
+     0.00853423,
+     -0.02645662,
+     -0.01476324,
+     0.01350342,
+     0.03095776,
+     0.01938323,
+     -0.00227899,
+     -0.03293785,
+     0.01162287,
+     -0.00825512,
+     0.01109362,
+     0.02965567,
+     -0.01720749,
+     -0.01672695,
+     -0.00307062,
+     -0.01065709,
+     0.00458977,
+     -0.00416517,
+     0.00277024,
+     0.01484402,
+     0.02151607,
+     0.01367335,
+     -0.01716657,
+     0.01993701,
+     0.02926229,
+     0.00564985,
+     0.00449883,
+     0.01318398,
+     0.01221114,
+     -0.01044224,
+     -0.01975954,
+     0.0114554,
+     0.00441833,
+     -0.01906356,
+     0.02950616,
+     -0.01843064,
+     0.01197973,
+     -0.01304068,
+     -0.01503828,
+     -0.0170262,
+     -0.01332446,
+     0.00610302,
+     -0.04128838,
+     -0.00567338,
+     -0.01320273,
+     -0.01186861,
+     0.01055282,
+     -0.02254591,
+     0.00607345,
+     0.00576573,
+     -0.00970896,
+     0.01475807,
+     -0.02851543,
+     0.00067781,
+     0.00906039,
+     0.01197799,
+     0.03900312,
+     0.0013545,
+     -0.0017178,
+     0.0235684,
+     -0.03324137,
+     -0.00347653,
+     -0.01606604,
+     0.00885091,
+     -0.00451053,
+     0.01240247,
+     0.01726436,
+     0.01211572,
+     -0.01130474,
+     -0.00727773,
+     0.00091156,
+     0.01911378,
+     0.011722,
+     0.00058718,
+     0.00973076,
+     0.00634966,
+     -0.01755847,
+     -0.02091217,
+     -0.01703672,
+     0.01370771,
+     -0.02025203,
+     0.01116729,
+     0.00527792,
+     -0.01642915,
+     -0.00243309,
+     -0.00065123,
+     0.01747186,
+     -0.00997447,
+     0.00312761,
+     0.00623531,
+     -0.03428432,
+     -0.00202292,
+     0.00637191,
+     0.03578324,
+     0.03068558,
+     -0.01501003,
+     -0.00663668,
+     -0.01148632,
+     0.00972851,
+     0.01488661,
+     -0.00356572,
+     -0.02273521,
+     0.00502845,
+     -0.02030335,
+     0.00166387,
+     -0.00340235,
+     -0.00562466,
+     0.00728441,
+     -0.00260638,
+     -0.03839081,
+     -0.008985,
+     0.00218783,
+     0.01611389,
+     0.00933593,
+     -0.00187294,
+     0.00745403,
+     -0.00690003,
+     -0.02526955,
+     0.02212381,
+     -0.00030148,
+     -0.01510011,
+     -0.01932472,
+     -0.03330071,
+     -0.01278328,
+     -0.0240929,
+     0.0307283,
+     -0.01996295,
+     0.00107976,
+     -0.00161456,
+     -0.00260982,
+     0.01108187,
+     0.00801411,
+     -0.00203263,
+     0.02143234,
+     -0.01392478,
+     0.00731467,
+     -0.02988216,
+     0.01828213,
+     0.00886731,
+     -0.02619752,
+     -0.00904667,
+     -0.01002406,
+     0.00907868,
+     -0.01247173,
+     -0.00422791,
+     -0.00707358,
+     -0.01671953,
+     -0.01462827,
+     0.01076446,
+     0.00044689,
+     0.00668957,
+     -0.02379257,
+     -0.00252276,
+     -0.00053585,
+     0.00860764,
+     0.01476439,
+     0.02438155,
+     -0.00497091,
+     -0.00031928,
+     0.00063662,
+     -0.04368747,
+     -0.0275035,
+     -0.00398356,
+     0.01002149,
+     -0.01770317,
+     0.00091963,
+     0.00874953,
+     -0.01610278,
+     0.00232937,
+     0.03462736,
+     -0.01182502,
+     -0.00796036,
+     0.00815754,
+     -0.00974997,
+     -0.02655735,
+     0.00698516,
+     -0.00113735,
+     0.00650803,
+     -0.03453831,
+     0.01165782,
+     -0.01219975,
+     -0.02069398,
+     0.00514758,
+     -0.00833858,
+     0.00578136,
+     -0.01246782,
+     -0.00441516,
+     0.01645298,
+     0.02736051,
+     -0.00781797,
+     0.00867647,
+     -0.00502843,
+     0.00425443,
+     -0.01442174,
+     0.01855954,
+     -0.02583617,
+     0.04021493,
+     -0.00768476,
+     0.00129157,
+     0.00881142,
+     0.0063479,
+     -0.0120367,
+     0.00646048,
+     -0.00491157,
+     0.023755,
+     0.00721263,
+     0.00155774,
+     0.01283433,
+     -0.02943,
+     -0.02881086,
+     0.01041367,
+     -0.01840024,
+     0.02171823,
+     -0.00567327,
+     -0.00276232,
+     0.00148473,
+     0.02302389,
+     -0.00975481,
+     0.00099824,
+     0.01985644,
+     -0.00192401,
+     -0.00617017,
+     -2.538e-05,
+     -0.03193742,
+     -0.03092247,
+     0.01425309,
+     -0.01900133,
+     -0.02314746,
+     -0.00357459,
+     -0.01438221,
+     0.02652883,
+     -0.02303528,
+     -0.01041824,
+     0.02131381,
+     0.03382028,
+     -0.02858257,
+     -0.01228277,
+     0.00077239,
+     0.01865366,
+     0.00257365,
+     -0.02251675,
+     -0.01038951,
+     -0.00826624,
+     -0.00061366,
+     -0.01293588,
+     -0.02640678,
+     -0.01507058,
+     -0.0096333,
+     -0.005784,
+     -0.00196619,
+     0.02236821,
+     -0.01330438,
+     0.00327015,
+     -0.00786517,
+     0.00283161,
+     -0.02694237,
+     -0.00937843,
+     0.00266939,
+     -0.0075223,
+     0.00277473,
+     0.01841909,
+     0.00641551,
+     0.00175325,
+     -0.01986219,
+     -0.00802713,
+     -0.01577379,
+     -0.00577331,
+     0.0009016,
+     0.0104826,
+     0.01410414,
+     -0.00801914,
+     0.02181359,
+     -0.00292495,
+     0.00389425,
+     -0.01559183,
+     0.01793076,
+     -0.01906901,
+     -0.00501983,
+     0.01160918,
+     -0.00288085,
+     0.00971605,
+     -0.02800927,
+     -0.01123048,
+     -0.01906575,
+     -0.01067145,
+     -0.00029776,
+     -0.00850199,
+     0.00057391,
+     -0.00254053,
+     -0.02233508,
+     -0.02169339,
+     -0.0151769,
+     -0.00609579,
+     -0.00705622,
+     -0.0149173,
+     -0.00437992,
+     -0.00259468,
+     0.0052716,
+     0.02288501,
+     0.00927041,
+     -0.00550658,
+     -0.02116336,
+     -0.01383797,
+     0.0036492,
+     0.03030897,
+     0.00985404,
+     0.00681804,
+     0.00413645,
+     -0.01247669,
+     0.0150871,
+     -0.00089391,
+     0.01245119,
+     -0.01382493,
+     -0.00465603,
+     -0.04419245,
+     0.00211103,
+     -0.01578317,
+     -0.00727821,
+     0.01030676,
+     -0.01007288,
+     0.01228494,
+     0.02027165,
+     0.01354787,
+     -0.00042948,
+     -0.00315299,
+     -0.03178857,
+     -0.00364243,
+     0.01836205,
+     -0.00837707,
+     0.00443475,
+     -0.03081122,
+     -0.00710205,
+     0.02053505,
+     -0.01760741,
+     -0.00722017,
+     -0.00764707,
+     0.01129031,
+     -0.01561538,
+     0.00539249,
+     -0.00413864,
+     -0.01105609,
+     0.00242777,
+     -0.01110355,
+     -0.00310831,
+     -0.01079649,
+     0.00431974,
+     -0.02030705,
+     0.02523745,
+     0.01010057,
+     0.00415501,
+     -0.00288939,
+     0.01196072,
+     -0.00411965,
+     -0.00083493,
+     0.02267452,
+     -0.00319011,
+     0.0008131,
+     0.0010743,
+     -0.0410697,
+     -0.00669343,
+     0.00026216,
+     0.02969771,
+     -0.00556103,
+     -0.00890212,
+     0.01827613,
+     -0.01220147,
+     -0.00520677,
+     -0.04481589,
+     -0.00165549,
+     -0.00732078,
+     0.01583677,
+     -0.00748797,
+     -0.00963265,
+     -0.01356199,
+     -0.00669439,
+     0.00836226,
+     0.02362512,
+     -0.01708823,
+     -0.01487742,
+     0.00612276,
+     -0.0108666,
+     -0.00820478,
+     0.0079394,
+     0.00539038,
+     -0.03808598,
+     -0.00721651,
+     0.00216078,
+     -0.00077955
+    ],
+    "cols": [
+     1,
+     8,
+     11,
+     13,
+     14,
+     15,
+     24,
+     34,
+     36,
+     41,
+     44,
+     53,
+     57,
+     58,
+     61,
+     63,
+     66,
+     69,
+     79,
+     92,
+     95,
+     98,
+     99,
+     101,
+     119,
+     140,
+     145,
+     152,
+     161,
+     171,
+     177,
+     226,
+     240,
+     246,
+     253,
+     259,
+     265,
+     287,
+     288,
+     302,
+     324,
+     343,
+     351,
+     355,
+     450,
+     578,
+     579,
+     602,
+     614,
+     621,
+     795,
+     868,
+     946,
+     998,
+     1003,
+     1246,
+     1689,
+     1780,
+     1973,
+     2055,
+     2175,
+     2248,
+     2661,
+     3343
+    ]
+   }
+  }
+ }
+}

code/configs/rebuilt.json

@@ -0,0 +1,2197 @@
+{
+ "rebuilt": {
+  "about": "Lab 3: the edit rebuilt from the agent's 8 texts (alpha 6, my u, my top 64)",
+  "layers": {
+   "15": {
+    "alpha": 6,
+    "dir": [
+     -0.24953727,
+     0.02032124,
+     0.04771795,
+     0.04000831,
+     0.0143242,
+     -0.21995239,
+     -0.03093798,
+     -0.13847236,
+     -0.0687337,
+     0.09629215,
+     0.03552215,
+     0.02262742,
+     -0.02543711,
+     0.14368935,
+     0.02360879,
+     0.14979322,
+     0.08297141,
+     0.0825967,
+     -0.19932722,
+     -0.09169211,
+     0.07426155,
+     0.06157133,
+     0.10419847,
+     0.10763864,
+     -0.0389273,
+     -0.00109971,
+     0.05881784,
+     0.03821842,
+     -0.01023125,
+     -0.0155359,
+     0.02342042,
+     -0.10402809,
+     0.04680654,
+     -0.05265392,
+     -0.11036691,
+     -0.02119951,
+     0.05915256,
+     0.27601135,
+     -0.07837382,
+     -0.00347673,
+     -0.05321625,
+     0.02205025,
+     -0.10572132,
+     0.03832579,
+     -0.02087151,
+     -0.01947924,
+     0.11653381,
+     -0.02051564,
+     -0.01071759,
+     -0.03027302,
+     -0.06089418,
+     0.04463274,
+     -0.03164496,
+     0.01951855,
+     -0.02302863,
+     0.10757776,
+     0.02056621,
+     0.0343796,
+     -0.0021492,
+     0.04744004,
+     0.02343818,
+     -0.06339314,
+     -0.03820134,
+     0.02501658,
+     0.03175897,
+     -0.01598197,
+     0.11075977,
+     -0.01552098,
+     0.01513177,
+     -0.02714768,
+     0.00190105,
+     0.02348685,
+     -0.05483799,
+     -0.01767546,
+     -0.13193983,
+     0.00077896,
+     -0.03830451,
+     0.00821382,
+     -0.08470301,
+     -0.01127409,
+     0.04484798,
+     0.09724138,
+     0.01788979,
+     -0.05973997,
+     -0.00031167,
+     0.12036057,
+     0.07074808,
+     -0.0102,
+     0.00488621,
+     0.00638768,
+     -0.03298204,
+     -0.04133184,
+     -0.0324551,
+     0.01001455,
+     0.04155349,
+     0.02289063,
+     -0.0624258,
+     0.04258976,
+     -0.03576536,
+     -0.00779208,
+     -0.01342702,
+     -0.05784821,
+     -0.01904367,
+     0.0157368,
+     -0.03480317,
+     -0.02095752,
+     -0.00261995,
+     0.02260072,
+     -0.05302511,
+     -0.02176841,
+     0.00990718,
+     0.06076119,
+     0.0167202,
+     -0.01325404,
+     -0.01575754,
+     0.08361955,
+     0.00011008,
+     -0.03992344,
+     -0.08015352,
+     0.05218405,
+     0.0256574,
+     -0.00060408,
+     -0.01703091,
+     -0.01975281,
+     -0.03141097,
+     0.02056967,
+     -0.00555578,
+     -0.04516612,
+     0.02356683,
+     0.06884296,
+     0.00917865,
+     -0.02308802,
+     0.02309628,
+     0.01845497,
+     0.00878982,
+     -0.0203653,
+     -0.00782054,
+     -0.06458497,
+     0.09055139,
+     0.00214113,
+     0.02803794,
+     0.04403042,
+     -0.00094625,
+     0.00861618,
+     -0.02396201,
+     0.0074091,
+     0.06147441,
+     0.02295231,
+     -0.01124603,
+     -0.01861799,
+     0.00209179,
+     0.00096943,
+     0.01037515,
+     0.01955204,
+     -0.02585044,
+     0.01777056,
+     -0.03991476,
+     0.03768253,
+     -0.00042653,
+     0.00815431,
+     0.04679773,
+     -0.02307066,
+     0.03822683,
+     0.01438087,
+     -0.02928447,
+     0.01390374,
+     0.09826396,
+     0.03247706,
+     -0.03843113,
+     0.02039174,
+     -0.03451136,
+     -0.04692665,
+     -0.01404249,
+     0.01338147,
+     -0.01011026,
+     -0.00880787,
+     0.01684509,
+     -0.02588393,
+     0.01805674,
+     -0.00719016,
+     -0.03264225,
+     -0.00939176,
+     0.02352562,
+     -0.01662001,
+     8.947e-05,
+     0.01536887,
+     -0.05908342,
+     -0.02059522,
+     0.03890344,
+     -0.01219262,
+     -0.02270816,
+     -0.01308892,
+     0.032672,
+     0.01836841,
+     -0.02627165,
+     0.03396625,
+     0.00332693,
+     -0.02368261,
+     0.01004383,
+     -0.054905,
+     -0.01186761,
+     0.03973839,
+     -0.01927921,
+     -0.02496052,
+     -0.03048274,
+     0.01191071,
+     0.01637166,
+     -0.03282031,
+     -0.06146573,
+     9.029e-05,
+     0.04366195,
+     0.01741599,
+     0.00701609,
+     -0.01906414,
+     0.00744014,
+     -0.01633424,
+     -0.01800021,
+     0.0323564,
+     -0.00062279,
+     -0.07529375,
+     -0.0075791,
+     0.0210552,
+     0.01169259,
+     -0.02394276,
+     0.00213605,
+     -0.07524197,
+     0.0136747,
+     0.00861713,
+     -0.03423766,
+     -0.01383792,
+     -0.02035608,
+     0.04885746,
+     0.01876548,
+     0.03787666,
+     0.00321658,
+     0.02173865,
+     0.00801081,
+     -0.00432673,
+     -0.04274024,
+     -0.00183638,
+     -0.0103395,
+     0.01865445,
+     0.00329304,
+     0.00230449,
+     0.01730042,
+     0.01731126,
+     -0.00055386,
+     -0.04531341,
+     0.01712174,
+     0.00037972,
+     0.03139253,
+     -0.00045943,
+     -0.04149744,
+     0.000518,
+     0.01067489,
+     0.05003539,
+     -0.00879554,
+     -0.03149081,
+     -0.0171353,
+     -0.00978476,
+     0.00836213,
+     0.01475394,
+     -0.00144215,
+     -0.00334998,
+     -0.01306126,
+     0.01092799,
+     0.00819701,
+     0.00807866,
+     -0.02311526,
+     0.00778096,
+     0.01513516,
+     -0.034882,
+     -0.00481504,
+     -0.01090861,
+     0.01800035,
+     0.00075741,
+     -0.03489013,
+     -0.00890846,
+     -0.05034224,
+     -0.02017788,
+     0.02364288,
+     0.00457983,
+     -0.0342081,
+     0.01396834,
+     0.00982909,
+     0.02411675,
+     0.00694275,
+     -0.00293969,
+     -0.04230263,
+     0.01600197,
+     0.02097433,
+     0.00090792,
+     -0.01560442,
+     0.00514772,
+     0.0137536,
+     -0.0069557,
+     -0.00067648,
+     -0.00811737,
+     0.02232993,
+     -0.01514024,
+     0.08919945,
+     0.00301825,
+     0.01551488,
+     0.0462007,
+     -0.01763635,
+     0.00928134,
+     0.00533358,
+     0.0195843,
+     0.03148966,
+     0.01661635,
+     -0.00448385,
+     0.02584421,
+     0.00887918,
+     0.02956191,
+     -0.05374401,
+     0.04092161,
+     -0.06316315,
+     -0.06251528,
+     -0.06065762,
+     0.01565573,
+     0.01009507,
+     0.00039843,
+     0.06566611,
+     0.02716734,
+     -0.0003762,
+     -0.00366652,
+     0.06472894,
+     -0.01461635,
+     0.00866892,
+     0.00744909,
+     -0.01116645,
+     -0.01881022,
+     0.00428999,
+     0.03664328,
+     0.00441959,
+     0.00618765,
+     0.02925167,
+     0.00133465,
+     -0.01213202,
+     -0.00427983,
+     -0.00818278,
+     0.02032951,
+     -0.01536711,
+     0.00697054,
+     -0.03022951,
+     0.02546028,
+     0.01947673,
+     -0.00402028,
+     -0.01183439,
+     0.03930105,
+     -0.01655081,
+     -0.00801508,
+     -0.03856235,
+     -0.00350546,
+     -0.06303616,
+     -0.01669959,
+     0.00690717,
+     -0.0204038,
+     0.01802597,
+     -0.01219154,
+     0.01592795,
+     -0.00070603,
+     0.03511924,
+     -0.018538,
+     -0.00704585,
+     0.00904308,
+     0.00613573,
+     -0.0150446,
+     0.00495122,
+     -0.02130735,
+     -0.00971549,
+     -0.01199382,
+     -0.01386097,
+     0.01570345,
+     0.00240006,
+     0.00064177,
+     -0.0038635,
+     0.01867323,
+     0.00244209,
+     -0.00370801,
+     -0.02493361,
+     -0.00779547,
+     -0.03015996,
+     0.00707168,
+     -0.00050214,
+     0.02271128,
+     0.00928181,
+     0.00883255,
+     -0.01380667,
+     -0.02119225,
+     -0.02348454,
+     0.00430965,
+     -0.00848631,
+     -0.00847641,
+     -0.00610021,
+     0.02354609,
+     0.00890114,
+     -0.00753464,
+     -0.00556093,
+     0.00221752,
+     -0.02240666,
+     0.03152145,
+     0.02903503,
+     0.02509717,
+     -0.02199765,
+     -0.0370455,
+     -0.00424749,
+     -0.0073382,
+     0.04361898,
+     -0.00020525,
+     0.02663727,
+     -0.01912568,
+     0.02326005,
+     -0.02344442,
+     0.02198884,
+     -0.01359024,
+     -0.02811271,
+     0.00990257,
+     -0.02367434,
+     0.00514392,
+     -0.03639058,
+     -0.00174365,
+     0.01068994,
+     -0.00056436,
+     -0.06993997,
+     0.01277461,
+     -0.00027771,
+     0.00881546,
+     -0.00927029,
+     0.00370692,
+     -0.02268402,
+     0.00086552,
+     -0.00182187,
+     0.02215254,
+     0.00216581,
+     0.04274159,
+     -0.0213224,
+     -0.00871555,
+     0.01324753,
+     0.03569933,
+     -0.00604653,
+     -0.00054132,
+     0.02708152,
+     0.0156354,
+     0.01532854,
+     0.01847429,
+     0.03432374,
+     0.01492259,
+     0.01553609,
+     -0.02336396,
+     -0.01170587,
+     0.01463112,
+     -0.00971806,
+     0.0065694,
+     0.01784878,
+     -0.00612434,
+     -0.00295724,
+     -0.01064425,
+     0.01663777,
+     0.0185685,
+     0.02852665,
+     -0.00981581,
+     0.01066554,
+     -0.01361641,
+     -0.02228133,
+     0.03113523,
+     0.02426337,
+     0.00486269,
+     0.01181809,
+     0.01762869,
+     -0.02489389,
+     0.03073462,
+     -0.01638569,
+     -0.00472346,
+     0.00900668,
+     0.01496488,
+     -0.00837067,
+     0.03504048,
+     0.00483944,
+     -0.01933852,
+     -0.03348221,
+     -0.00154681,
+     -0.02760711,
+     0.01916866,
+     -0.00771995,
+     -0.00483741,
+     0.01592334,
+     0.03234,
+     -0.0044583,
+     -0.00997334,
+     -0.0054156,
+     -0.01612384,
+     -0.0277504,
+     0.00014797,
+     0.00118905,
+     0.00114228,
+     0.02686001,
+     0.01271726,
+     0.04131245,
+     -0.03268129,
+     -0.00507546,
+     -0.01850859,
+     0.01750851,
+     0.02386562,
+     0.01911094,
+     0.02560507,
+     -0.00085027,
+     -0.0037884,
+     -0.01314613,
+     -0.00036874,
+     0.01763174,
+     -0.01506276,
+     -0.02007655,
+     -0.00070394,
+     -0.01933974,
+     0.02117775,
+     -0.0197523,
+     0.01275597,
+     -0.00043964,
+     -0.00920847,
+     0.04400012,
+     0.00065479,
+     0.00978442,
+     0.01224339,
+     0.00843249,
+     0.01852363,
+     -0.02272808,
+     -0.00435086,
+     0.00467805,
+     -0.01969587,
+     0.0104729,
+     0.00644387,
+     0.01556693,
+     -0.00039924,
+     -0.03624526,
+     -0.00583931,
+     0.01209956,
+     0.03393426,
+     0.02946118,
+     0.0052841,
+     0.01848364,
+     -0.01311671,
+     0.00703236,
+     -0.00775086,
+     0.01492621,
+     0.01959657,
+     -0.02304517,
+     0.02501997,
+     0.02633144,
+     0.01188129,
+     -0.01033177,
+     -0.05345566,
+     0.01437744,
+     -0.02516746,
+     0.02001703,
+     -0.0043796,
+     0.02080955,
+     -0.0092044,
+     0.00400259,
+     0.03293351,
+     -0.00926822,
+     0.02755668,
+     0.00430748,
+     0.03074696,
+     -0.01154997,
+     -0.00662604,
+     0.01208532,
+     -0.0054236,
+     0.00058863,
+     0.01515122,
+     0.00286804,
+     -0.03359331,
+     0.006274,
+     -0.01387866,
+     0.00994134,
+     -0.00111388,
+     -0.03085189,
+     -0.02703462,
+     0.03251583,
+     0.01163809,
+     0.00467459,
+     0.02080074,
+     0.00122755,
+     -0.01627947,
+     -0.00485002,
+     -0.00307627,
+     -0.00993429,
+     0.01340899,
+     0.01787454,
+     -0.02771156,
+     -0.00708306,
+     0.00729373,
+     0.01489344,
+     -0.0032803,
+     -0.00571629,
+     -0.02973327,
+     0.0013901,
+     -0.00129384,
+     0.00512576,
+     0.03255487,
+     0.00524011,
+     -0.01886851,
+     0.03806211,
+     0.00619361,
+     -0.01725866,
+     0.00897313,
+     -0.00135034,
+     0.01599166,
+     -0.02018507,
+     -0.01595133,
+     0.01487378,
+     0.00294721,
+     0.00102176,
+     -0.00473533,
+     0.00511234,
+     -0.00398348,
+     0.01135204,
+     0.00754507,
+     0.02357361,
+     0.03142235,
+     -0.00648861,
+     -0.01727412,
+     -0.01826213,
+     -0.00099926,
+     0.02374178,
+     -0.00095574,
+     0.02314333,
+     -0.00078425,
+     -0.0192946,
+     0.01915835,
+     -0.00171457,
+     -0.00351598,
+     0.00217374,
+     -0.00616731,
+     0.00662051,
+     -0.00403052,
+     0.02256066,
+     -0.01207136,
+     -0.00523706,
+     0.01712961,
+     0.0475691,
+     -0.00626641,
+     0.0019913,
+     -0.01728008,
+     0.0176838,
+     -0.00322787,
+     0.01693019,
+     0.03213869,
+     -0.01507388,
+     -0.02262384,
+     0.0189039,
+     -0.00333628,
+     -0.00649579,
+     -0.00333384,
+     0.00296422,
+     0.007682,
+     0.00162625,
+     0.02766717,
+     0.00457197,
+     0.04865065,
+     -0.00820853,
+     0.02016005,
+     0.03420485,
+     -0.0131285,
+     0.02328214,
+     -0.00852074,
+     -0.01124847,
+     -0.00485998,
+     0.04002898,
+     -0.0211242,
+     -0.00886603,
+     -0.02890895,
+     -0.01964639,
+     0.00854094,
+     0.01961291,
+     -0.03156931,
+     -0.03173686,
+     -0.00388045,
+     -0.02344103,
+     -0.037501,
+     -0.03553733,
+     -0.01519209,
+     -0.01791494,
+     -0.03089717,
+     -0.01997609,
+     0.00598362,
+     0.02980532,
+     0.00301147,
+     -0.02656583,
+     -0.01941959,
+     0.01617874,
+     0.01348796,
+     -0.00811255,
+     0.00204054,
+     -0.00675811,
+     0.01261742,
+     0.00900566,
+     -0.01580282,
+     0.00220431,
+     -0.00297819,
+     -0.00867088,
+     0.02229637,
+     0.00264761,
+     0.00745153,
+     0.0156985,
+     -0.00621707,
+     0.001004,
+     0.00017393,
+     -0.02039811,
+     0.00801156,
+     -0.02776234,
+     -0.02535997,
+     -0.02567489,
+     0.00281981,
+     -0.00654595,
+     -0.01022325,
+     -0.00170163,
+     -0.01141319,
+     0.01317656,
+     0.00056951,
+     -0.01078958,
+     0.02116568,
+     0.0095097,
+     -0.0149497,
+     0.02496791,
+     0.01960952,
+     0.01990261,
+     0.02258615,
+     0.01920174,
+     0.03596667,
+     0.00099478,
+     -0.0229254,
+     0.00990298,
+     0.02504748,
+     0.01103509,
+     -0.01002065,
+     -0.01105082,
+     0.0085911,
+     -0.02008658,
+     -0.01259166,
+     -0.00247381,
+     0.00217059,
+     -0.02131948,
+     -0.00264463,
+     0.01183303,
+     -0.0200939,
+     -0.00802105,
+     0.01097476,
+     0.00860818,
+     -0.02176685,
+     0.01244621,
+     0.01678269,
+     -0.00806077,
+     0.0194993,
+     0.00974802,
+     -0.02071967,
+     -0.02892916,
+     -0.00469439,
+     -0.01954783,
+     -0.03511016,
+     -0.00481206,
+     0.00092754,
+     -0.00647932,
+     0.02529612,
+     0.00747797,
+     -0.00231357,
+     -0.00390756,
+     -0.0171801,
+     -0.00410183,
+     0.02131562,
+     0.00454892,
+     -0.0292922,
+     -0.00711533,
+     0.02149809,
+     -0.01275244,
+     -0.01269646,
+     -0.01939993,
+     -0.02410693,
+     -0.00975765,
+     0.01132927,
+     0.02547917,
+     -0.01860545,
+     -0.02865321,
+     -0.03200746,
+     0.01562794,
+     0.02289022,
+     -0.01663926,
+     -0.00958291,
+     -0.01763906,
+     -0.01870583,
+     0.02814463,
+     0.01109149,
+     0.02033663,
+     0.01271503,
+     0.03006222,
+     -0.00048329,
+     -0.00158084,
+     0.00507858,
+     -0.02844111,
+     -0.00451897,
+     0.00376603,
+     0.00168753,
+     0.00129588,
+     -0.01696272,
+     0.0429162,
+     0.00415294,
+     -0.02960312,
+     -0.01082198,
+     0.02226594,
+     0.00565284,
+     -0.00193778,
+     0.00644563,
+     0.00607838,
+     0.00383652,
+     0.0387188,
+     -0.02177966,
+     0.01392916,
+     -0.00389068,
+     0.00462986,
+     -0.00591028,
+     0.00774544,
+     0.02415533,
+     -0.01322896,
+     0.02906025,
+     0.02997837,
+     -0.00962439,
+     0.04012598,
+     0.0074251,
+     0.0035456,
+     -0.01087397,
+     0.01074715,
+     -0.00203071,
+     -0.0077369,
+     0.00579586,
+     0.0075242,
+     0.01921529,
+     0.03780481,
+     0.02322195,
+     -0.00854881,
+     0.01923305,
+     -0.00798716,
+     0.00166868,
+     -0.02916843,
+     0.00192206,
+     0.0144256,
+     0.00113591,
+     -0.01708934,
+     -0.02459632,
+     0.02938533,
+     -0.03380818,
+     0.02188608,
+     -0.02040062,
+     -0.02458778,
+     0.00072677,
+     -0.00347958,
+     -0.00128368,
+     -0.00388601,
+     -0.02840634,
+     0.0021292,
+     0.0023034,
+     -0.01680031,
+     0.01571158,
+     0.00830004,
+     -0.03735995,
+     0.0254634,
+     -0.00473872,
+     -0.00072189,
+     0.02520677,
+     -0.00860385,
+     0.01570575,
+     0.00436401,
+     -0.02640044,
+     0.00596871,
+     0.01531329,
+     -0.00725632,
+     0.00638598,
+     -0.02094878,
+     -0.01477943,
+     -0.01413115,
+     0.04499755,
+     0.00113113,
+     -0.03174527,
+     -0.00209084,
+     -0.03338251,
+     -0.0096309,
+     0.00287617,
+     -0.01256197,
+     0.01806203,
+     -0.02083992,
+     0.00520981,
+     -0.01840962,
+     -0.01567308,
+     0.02502837,
+     -0.00166109,
+     -0.02704363,
+     -0.00332124,
+     0.00635724,
+     -0.00385252,
+     -0.00838694,
+     0.01483678,
+     0.00363128,
+     -0.0092808,
+     0.00838382,
+     -0.00979391,
+     0.03600009,
+     -0.0133385,
+     -0.03795407,
+     0.01375116,
+     0.01596306,
+     0.00234801,
+     0.01738501,
+     0.02754136,
+     -0.00016695,
+     -0.02247729,
+     -0.00583192,
+     0.00303804,
+     -0.01862978,
+     0.00295507,
+     -0.00252994,
+     -0.01407245,
+     0.00016214,
+     -0.01388401,
+     0.01226901,
+     -0.01216287,
+     -0.01331897,
+     -0.02745839,
+     0.00533697,
+     -0.00886373,
+     0.01137462,
+     -0.0085808,
+     0.0069616,
+     -0.02154622,
+     0.00649606,
+     -0.00231059,
+     -0.01416667,
+     0.02500885,
+     -0.01119208,
+     -0.02103906,
+     -0.01935066,
+     -0.00227805,
+     0.02452481,
+     0.00060232,
+     0.0104103,
+     -0.00241755,
+     -0.01093803,
+     0.00326898,
+     0.02128485,
+     0.02182901,
+     -0.00381456,
+     -0.00549965,
+     -0.0475321,
+     0.00133885,
+     0.0024913,
+     -0.00498843,
+     0.0218831,
+     0.00093283,
+     0.00075971,
+     0.02190154,
+     -0.03986854,
+     -0.02121811,
+     -0.01359594,
+     0.00847025,
+     -0.00618094,
+     -0.01546106,
+     0.00517741,
+     0.01312633,
+     -0.01497871,
+     -0.00776035,
+     0.01118489,
+     -0.02040217,
+     0.00727101,
+     -0.03115027,
+     0.0184103,
+     -0.00714149,
+     0.01175752,
+     0.0036893,
+     0.00872613,
+     0.01293681,
+     -0.02438077,
+     -0.00291088,
+     0.00280636,
+     -0.00522472,
+     0.00602782,
+     0.0032788,
+     0.01566291,
+     0.00508319,
+     -0.00120315,
+     0.02941313,
+     0.00818047,
+     0.03064339,
+     0.00138223,
+     -0.00743662,
+     -0.00375532,
+     -0.00553083,
+     -0.02147844,
+     -0.02098986,
+     0.01503158,
+     -0.01685054,
+     -0.01447685,
+     -0.01132737,
+     -0.00244616,
+     -0.01728076,
+     -0.03332868,
+     -0.00314168,
+     -0.02164423,
+     0.0036973,
+     0.02682395,
+     0.00388641,
+     0.00867326,
+     -0.03751247,
+     -0.00233092,
+     0.01589548,
+     0.02289741,
+     -0.01888031,
+     -0.02657688,
+     -0.02343804,
+     -0.00483036,
+     -0.02808756,
+     -0.00989105,
+     -0.00036061,
+     -0.01134445,
+     0.01664197,
+     0.02578348,
+     -0.00212988
+    ],
+    "cols": [
+     0,
+     2,
+     11,
+     25,
+     34,
+     35,
+     38,
+     39,
+     46,
+     50,
+     53,
+     56,
+     61,
+     62,
+     63,
+     64,
+     67,
+     73,
+     100,
+     108,
+     129,
+     135,
+     156,
+     169,
+     209,
+     215,
+     264,
+     266,
+     294,
+     327,
+     328,
+     357,
+     371,
+     376,
+     412,
+     414,
+     442,
+     504,
+     505,
+     509,
+     516,
+     580,
+     590,
+     601,
+     638,
+     682,
+     738,
+     882,
+     926,
+     930,
+     1047,
+     1060,
+     1087,
+     1130,
+     1155,
+     1184,
+     1460,
+     1481,
+     1787,
+     1943,
+     2005,
+     2066,
+     2738,
+     3003
+    ]
+   },
+   "19": {
+    "alpha": 6,
+    "dir": [
+     -0.33485293,
+     0.03469285,
+     0.06433747,
+     0.07593478,
+     0.04320995,
+     -0.20440264,
+     0.01049561,
+     -0.07845271,
+     -0.02966316,
+     0.07869723,
+     0.02690876,
+     -0.00667397,
+     0.02327623,
+     0.10146445,
+     0.03316295,
+     0.14016815,
+     0.09809733,
+     0.11769328,
+     -0.20819545,
+     -0.1066898,
+     0.09799061,
+     0.02515803,
+     0.11596736,
+     0.12506141,
+     -0.02365931,
+     -0.01581082,
+     0.05179979,
+     0.07284577,
+     -0.03920847,
+     -0.04298451,
+     0.00825938,
+     -0.10469546,
+     0.06105305,
+     -0.01464969,
+     -0.1059337,
+     0.00563145,
+     0.08228201,
+     0.21913013,
+     -0.13936099,
+     -0.01100376,
+     -0.05633171,
+     0.02826658,
+     -0.06082795,
+     0.01529873,
+     -0.03269898,
+     -0.06137543,
+     0.07338464,
+     -0.01731984,
+     0.01293509,
+     0.02285551,
+     -0.0430566,
+     0.06165075,
+     -0.03291543,
+     0.01243704,
+     -0.02252448,
+     0.10640015,
+     0.02985976,
+     0.02642925,
+     0.01327473,
+     0.03750305,
+     0.02248123,
+     -0.08859696,
+     -0.04791253,
+     0.02915941,
+     0.04655077,
+     -0.02370902,
+     0.06000244,
+     -0.02559192,
+     0.0041156,
+     -0.04574716,
+     -0.0151819,
+     0.03427166,
+     -0.0521356,
+     0.00275684,
+     -0.11008257,
+     -0.01052809,
+     -0.05736936,
+     -0.01020324,
+     -0.09052463,
+     0.00689397,
+     0.03621768,
+     0.07123613,
+     0.00360201,
+     -0.06619763,
+     0.02345335,
+     0.11091703,
+     0.09016617,
+     -0.01928664,
+     0.01368285,
+     -0.00499045,
+     -0.03138366,
+     -0.02867982,
+     -0.04870289,
+     -0.00659964,
+     0.03185755,
+     0.01343173,
+     -0.04430854,
+     0.03535172,
+     -0.02320377,
+     0.00691644,
+     -0.02510364,
+     -0.07474331,
+     -0.01688526,
+     0.03140407,
+     -0.02430508,
+     -0.01303292,
+     -0.00812541,
+     0.0027574,
+     -0.02376566,
+     -0.03320733,
+     0.02914509,
+     0.06341685,
+     0.01786544,
+     0.0007114,
+     -0.01188948,
+     0.07291664,
+     -0.00413994,
+     -0.0639967,
+     -0.06567573,
+     0.04150274,
+     0.01945941,
+     0.03311839,
+     -0.02324571,
+     0.00037765,
+     -0.0109944,
+     0.04036979,
+     0.00052145,
+     -0.01924994,
+     0.00095022,
+     0.0644691,
+     0.00839896,
+     -0.04536164,
+     0.01759123,
+     0.01193937,
+     0.00483495,
+     -0.00580474,
+     -0.01910567,
+     -0.06255472,
+     0.05657258,
+     -0.02691831,
+     0.01994252,
+     0.02171535,
+     0.01559719,
+     0.01146398,
+     -0.03114962,
+     0.02478721,
+     0.04094142,
+     0.0108493,
+     -0.02564245,
+     -0.00549383,
+     0.01298171,
+     0.00228927,
+     0.0052698,
+     0.02828737,
+     -0.00475973,
+     0.02829008,
+     -0.05092382,
+     0.02987025,
+     0.00792591,
+     0.00794679,
+     0.03851486,
+     -0.01412571,
+     0.03591633,
+     0.02479657,
+     -0.01975248,
+     -0.00360033,
+     0.04643525,
+     -0.01874319,
+     -0.03569885,
+     0.02199704,
+     -0.01068916,
+     -0.03476141,
+     -0.00933738,
+     -0.01238625,
+     -0.0188238,
+     -0.00079303,
+     -0.00026924,
+     -0.02419199,
+     0.03646034,
+     0.03449475,
+     -0.0292365,
+     -0.01684576,
+     0.00764235,
+     -0.01628162,
+     -0.0080728,
+     -0.01059124,
+     -0.04215249,
+     -0.04302702,
+     0.02810987,
+     -0.0130589,
+     -0.03027271,
+     -0.00660881,
+     0.0222896,
+     0.04006188,
+     -0.03167387,
+     0.03514763,
+     -0.0102343,
+     -0.02564783,
+     -0.00238742,
+     -0.04280112,
+     -0.02142761,
+     0.0457569,
+     -0.00995881,
+     -0.01363918,
+     -0.0523981,
+     0.03240933,
+     -0.00046247,
+     -0.02916587,
+     -0.05593346,
+     0.01162838,
+     0.0239982,
+     0.00696456,
+     0.01568088,
+     -0.02222509,
+     -0.00589124,
+     -0.02352338,
+     -0.00299364,
+     0.03131129,
+     0.01445309,
+     -0.08602708,
+     -0.02007218,
+     0.04017609,
+     -0.01014829,
+     -0.02538945,
+     0.02321257,
+     -0.06834598,
+     0.00964575,
+     -0.0051437,
+     -0.01859846,
+     -0.00214777,
+     -0.04108446,
+     0.03662089,
+     0.01966953,
+     0.04098223,
+     0.01839868,
+     0.01493723,
+     0.00900578,
+     0.01188021,
+     -0.04769861,
+     -0.00375667,
+     -0.00330618,
+     0.01062461,
+     0.0013934,
+     -0.00176543,
+     0.02101856,
+     0.02736561,
+     0.01389204,
+     -0.00640379,
+     -0.00614297,
+     0.01084499,
+     0.02786553,
+     -0.02415286,
+     -0.02912875,
+     -0.01692037,
+     0.00126084,
+     0.05189257,
+     -0.01389195,
+     -0.03966757,
+     -0.00938222,
+     -0.029164,
+     -0.00431837,
+     -0.00161096,
+     0.02716199,
+     0.01256876,
+     0.00973001,
+     0.03604037,
+     0.02316862,
+     0.02573312,
+     -0.02866352,
+     0.00940141,
+     0.01636082,
+     -0.04596978,
+     0.01874516,
+     -0.02349585,
+     0.00517315,
+     0.00220712,
+     -0.01262933,
+     0.02448426,
+     -0.02697523,
+     -0.03611716,
+     0.01318228,
+     -0.0020534,
+     -0.02395383,
+     0.02149507,
+     -7.5e-07,
+     0.01313674,
+     0.00810257,
+     -0.0237144,
+     -0.03781516,
+     -0.01663774,
+     0.00806578,
+     0.01225692,
+     -0.01204816,
+     0.02055262,
+     0.01788219,
+     -0.03242413,
+     0.00801747,
+     -0.00297862,
+     -0.01131719,
+     -0.01430189,
+     0.03083245,
+     -0.01074697,
+     -0.02104215,
+     0.03708439,
+     -0.03587766,
+     -0.00112949,
+     -0.00407534,
+     0.01345476,
+     0.03780131,
+     0.02845756,
+     -0.01566019,
+     -0.01311745,
+     0.01434955,
+     0.01990891,
+     -0.03681851,
+     0.02035851,
+     -0.06993868,
+     -0.05787938,
+     -0.03253908,
+     0.00905447,
+     0.038856,
+     -0.00199011,
+     0.02992324,
+     0.00809208,
+     -0.00987577,
+     -0.00634818,
+     0.06414509,
+     -0.00876435,
+     -0.00199508,
+     -0.01413713,
+     0.02092653,
+     -0.01234324,
+     0.03740568,
+     0.05363179,
+     0.01417355,
+     0.0100616,
+     0.01451284,
+     0.00778567,
+     -0.0122038,
+     0.00317306,
+     -0.00395102,
+     -0.01568509,
+     -0.01977195,
+     -0.00337621,
+     -0.00852562,
+     -0.01215244,
+     0.00799697,
+     0.01198815,
+     -0.01456201,
+     0.02279017,
+     -0.0340085,
+     -0.00045465,
+     -0.04464959,
+     -0.00148215,
+     -0.04413975,
+     0.00469116,
+     -0.0103799,
+     -0.01301511,
+     0.01054597,
+     -0.01396076,
+     0.05034662,
+     -0.01023035,
+     0.01737638,
+     -0.03203622,
+     0.00871286,
+     0.00355895,
+     0.01077468,
+     -0.0107326,
+     0.00578704,
+     0.00076102,
+     0.00033403,
+     -0.00997107,
+     0.01247552,
+     -0.00807111,
+     0.00515466,
+     -0.01044779,
+     -0.00936228,
+     0.0198695,
+     -0.00272679,
+     -0.01352749,
+     7.16e-06,
+     0.02315546,
+     -0.026254,
+     -0.0053682,
+     0.00264656,
+     0.01020446,
+     0.00516271,
+     0.01508454,
+     -0.0209725,
+     0.00244697,
+     -0.01309793,
+     -0.014711,
+     -0.03019547,
+     -0.00102567,
+     0.00826303,
+     0.03253384,
+     0.01062864,
+     -0.00843968,
+     -0.01330544,
+     0.02085557,
+     -0.02482615,
+     0.01753628,
+     0.00801335,
+     0.02842218,
+     0.00948801,
+     -0.03443109,
+     0.0126709,
+     0.00087775,
+     0.00197925,
+     0.00805351,
+     0.02990451,
+     -0.01561085,
+     0.02241008,
+     -0.02911878,
+     0.01346374,
+     -0.01211556,
+     -0.0524451,
+     0.04090678,
+     -0.03539928,
+     -0.01168483,
+     -0.04193525,
+     -0.00351266,
+     0.01685971,
+     -0.01135502,
+     -0.10916007,
+     0.02025623,
+     5.093e-05,
+     0.01171376,
+     0.01459979,
+     0.00613427,
+     -0.00408939,
+     0.01221592,
+     -0.01002734,
+     0.0186257,
+     0.01269805,
+     0.06049786,
+     -0.01004597,
+     -0.00597877,
+     0.01513275,
+     0.02150752,
+     0.00289783,
+     -0.00502687,
+     -0.02977115,
+     0.0105178,
+     0.01677245,
+     0.02142598,
+     0.0200704,
+     0.04995825,
+     0.01128616,
+     -0.01138572,
+     0.0028806,
+     0.02906739,
+     -0.01654103,
+     -0.0074083,
+     -0.00165478,
+     -0.02611711,
+     -0.01441723,
+     -0.02270881,
+     0.01406153,
+     0.0329129,
+     0.03200575,
+     -0.00264824,
+     0.02199779,
+     -0.01492557,
+     -0.01760359,
+     0.01623378,
+     0.01867373,
+     0.02524926,
+     0.00967028,
+     0.01461598,
+     -0.02035144,
+     0.02304462,
+     -0.01513378,
+     0.00965755,
+     0.01401328,
+     -0.00270961,
+     0.00232686,
+     0.01563304,
+     0.02284709,
+     -0.05560019,
+     -0.04236459,
+     0.00073883,
+     -0.02807729,
+     0.01340476,
+     -0.00480668,
+     -0.02714749,
+     -0.00581044,
+     0.00199742,
+     0.01153813,
+     -0.00933401,
+     -0.03477429,
+     -0.03197195,
+     -0.00130699,
+     -0.00737675,
+     -0.01272538,
+     0.02102258,
+     0.00410118,
+     -0.00145837,
+     0.01512554,
+     -0.02576101,
+     -0.00219411,
+     -0.00436368,
+     0.00527814,
+     0.00784812,
+     0.01105572,
+     0.00454362,
+     -0.02113932,
+     -0.00996873,
+     0.01481754,
+     0.0370915,
+     0.01813238,
+     -0.01072198,
+     -0.02466616,
+     -0.01082149,
+     -0.0235935,
+     0.00772955,
+     -0.01920894,
+     0.011986,
+     0.0027178,
+     -0.00140089,
+     0.04917992,
+     0.00532532,
+     -0.00862383,
+     0.0033925,
+     0.01695543,
+     0.01353779,
+     -0.01470014,
+     -0.03946432,
+     0.01393099,
+     -0.00813009,
+     -0.00216593,
+     0.00183808,
+     0.02404295,
+     -0.01900738,
+     -0.02037269,
+     0.00566403,
+     0.00196521,
+     0.01408339,
+     -0.01416086,
+     -0.02554832,
+     0.00501152,
+     -0.02169447,
+     -0.00156593,
+     0.00677395,
+     0.01410867,
+     0.01969471,
+     -0.00997529,
+     0.04218512,
+     0.03097025,
+     0.03468947,
+     0.00412627,
+     -0.02789333,
+     0.01408405,
+     -0.02234736,
+     -0.00035949,
+     -0.00303544,
+     -0.00914162,
+     -0.02251699,
+     0.00473057,
+     0.02165684,
+     -0.00946849,
+     0.0231431,
+     -0.00792189,
+     0.03188115,
+     0.00995001,
+     -0.01436275,
+     0.0132283,
+     0.02302178,
+     0.01678467,
+     0.01890898,
+     0.00624642,
+     -0.01549046,
+     0.01312298,
+     -0.01649535,
+     0.02968395,
+     -0.00939973,
+     -0.01924723,
+     0.04292338,
+     0.0288663,
+     0.02567896,
+     -0.00700893,
+     0.02072216,
+     0.01288865,
+     -0.00674905,
+     0.0068656,
+     -0.00257756,
+     -0.00780505,
+     -0.02037166,
+     -0.00193348,
+     -0.00860267,
+     -0.03601069,
+     0.00303545,
+     0.05838604,
+     0.02260059,
+     -0.03325956,
+     -0.03232592,
+     -0.00711191,
+     0.02423159,
+     -0.00944073,
+     0.0153579,
+     0.01699695,
+     -0.0008257,
+     0.02907693,
+     0.02055094,
+     -0.00871408,
+     -0.01249677,
+     0.00056563,
+     0.0096538,
+     -0.01459426,
+     0.00653958,
+     0.01096033,
+     0.02558932,
+     -0.0168935,
+     -0.01167921,
+     0.01401768,
+     -0.01887079,
+     -0.0095849,
+     -0.03007321,
+     0.0249985,
+     0.02601537,
+     0.00875255,
+     -0.01530061,
+     -0.01763298,
+     0.00101135,
+     0.02701717,
+     -0.01504728,
+     0.01979423,
+     -0.01740025,
+     -0.00037391,
+     0.01941003,
+     0.00366848,
+     0.00170964,
+     0.01005537,
+     0.00897676,
+     -0.01451694,
+     0.01472355,
+     0.01401075,
+     -0.04008098,
+     0.00478178,
+     0.00264469,
+     0.01465146,
+     -0.00858278,
+     -0.01525469,
+     -0.00837144,
+     0.00379487,
+     0.00762924,
+     0.0141948,
+     0.02449119,
+     -0.02006853,
+     -0.01467974,
+     0.01018124,
+     -0.03718886,
+     -0.00821285,
+     -0.00819946,
+     0.00170308,
+     0.01008323,
+     0.01641961,
+     0.03285327,
+     -0.02158241,
+     0.05546584,
+     0.00506394,
+     0.0028938,
+     0.00908288,
+     0.00138076,
+     0.00620869,
+     -0.00761552,
+     -0.02185999,
+     -0.00480003,
+     0.01747121,
+     0.00039572,
+     0.01573705,
+     0.00300708,
+     -0.03099894,
+     0.00092616,
+     0.01118857,
+     -0.02362317,
+     -0.0301687,
+     0.01169634,
+     -0.00954914,
+     -0.0323673,
+     -0.04537297,
+     -0.01935816,
+     -0.00577393,
+     -0.0322955,
+     -0.00516561,
+     9.74e-06,
+     0.00058782,
+     0.01400073,
+     -0.04664317,
+     -0.01463405,
+     0.01198609,
+     0.0194691,
+     -0.00406392,
+     0.0008507,
+     -0.01848799,
+     0.03592278,
+     -0.01971447,
+     -0.00676394,
+     -0.02517347,
+     0.01982344,
+     0.00479301,
+     0.02273475,
+     0.00241195,
+     -0.00936509,
+     0.00513546,
+     -0.01280215,
+     0.00603569,
+     0.0172984,
+     -0.01518925,
+     0.0097814,
+     0.00418319,
+     0.00138834,
+     -0.02091384,
+     -0.00198403,
+     0.01229568,
+     0.00035958,
+     -0.01921193,
+     0.01499279,
+     -0.00202204,
+     -0.01947481,
+     -0.01102965,
+     0.00401898,
+     0.03645612,
+     -0.02703262,
+     0.00610309,
+     -0.00235634,
+     0.00785879,
+     0.00642532,
+     0.02070143,
+     0.02173702,
+     0.0086331,
+     0.00403808,
+     -0.02067142,
+     -0.02638881,
+     -0.00668427,
+     -0.01563201,
+     0.00041501,
+     -0.01161527,
+     -0.00428055,
+     -0.03578255,
+     0.00167846,
+     -2.603e-05,
+     0.00864414,
+     0.00446198,
+     -0.00604678,
+     -0.0234791,
+     0.00210582,
+     0.02637112,
+     0.02551471,
+     0.00376715,
+     0.01265203,
+     0.01572001,
+     -0.01759844,
+     -0.01298658,
+     -0.01324992,
+     -0.01552491,
+     -0.00672995,
+     -0.00874197,
+     -0.03830329,
+     -0.0154488,
+     -0.02508856,
+     0.03405344,
+     -0.00183639,
+     0.01713831,
+     0.00159486,
+     -0.03485499,
+     -0.00087476,
+     -0.01704301,
+     -0.00951778,
+     0.04916522,
+     0.00483608,
+     -0.00797562,
+     -0.01565256,
+     0.00146399,
+     -0.00162913,
+     -0.02793031,
+     -0.00984591,
+     -0.00824814,
+     0.00142111,
+     0.01817587,
+     0.01086203,
+     -0.01275286,
+     -0.0189762,
+     -0.03228875,
+     0.01003015,
+     -0.00134121,
+     0.00325151,
+     -0.03226647,
+     0.00423768,
+     -0.0380742,
+     0.02109345,
+     0.00124923,
+     0.03016158,
+     -0.01302608,
+     0.02091108,
+     0.01425921,
+     -0.03021008,
+     -0.01007143,
+     -0.02217595,
+     -0.00590678,
+     -0.01367597,
+     -0.02060196,
+     -0.00706154,
+     -0.00447443,
+     0.0351731,
+     0.03089835,
+     -0.00138713,
+     -0.01089517,
+     0.0110692,
+     -0.01402011,
+     -0.01749948,
+     0.01299706,
+     0.02726881,
+     -0.00705925,
+     0.01604795,
+     -0.00103185,
+     -0.00643478,
+     -0.01959145,
+     0.02240784,
+     -0.0106791,
+     0.0045717,
+     -0.00056095,
+     -0.00478964,
+     0.01092616,
+     0.00612308,
+     -0.01870584,
+     0.02174109,
+     -0.00156753,
+     0.01371979,
+     -0.01055524,
+     0.00970679,
+     0.00153045,
+     0.01469247,
+     -0.00180606,
+     0.00455373,
+     0.03337794,
+     0.01545489,
+     0.02447022,
+     -0.00159542,
+     0.01690913,
+     -0.01614372,
+     0.00776934,
+     0.00011459,
+     0.00944597,
+     -0.0094295,
+     -0.00338595,
+     -0.01048475,
+     -0.02400045,
+     0.03072975,
+     -0.0109637,
+     0.01086839,
+     -0.00842686,
+     -0.02364761,
+     0.00342246,
+     -0.00714168,
+     0.02385984,
+     -0.02467543,
+     -0.01504063,
+     -0.00446535,
+     -0.01871253,
+     -0.01395926,
+     0.0247374,
+     -0.0080816,
+     -0.03657099,
+     0.00132056,
+     -0.01758603,
+     0.01844427,
+     -0.01328465,
+     0.0111822,
+     0.01086053,
+     0.00981689,
+     -0.03230486,
+     0.01464552,
+     -0.00148964,
+     -0.00762999,
+     -0.00404623,
+     -0.01966663,
+     0.00110262,
+     0.00535944,
+     0.01039123,
+     -0.01684566,
+     -0.02311614,
+     -0.01413236,
+     -0.02443904,
+     -0.01073471,
+     -0.00278614,
+     -0.00934225,
+     -0.00017413,
+     -0.00365744,
+     0.01714262,
+     -0.01331676,
+     -2.972e-05,
+     -0.00191073,
+     0.00781554,
+     -0.01613794,
+     0.00775712,
+     0.00647007,
+     -0.01789146,
+     0.00141006,
+     -0.0162336,
+     -0.01072993,
+     0.00046216,
+     0.015242,
+     0.00215394,
+     0.03298695,
+     0.03746279,
+     -0.02986987,
+     0.01046771,
+     0.02376416,
+     0.01895383,
+     0.00395636,
+     0.00173356,
+     0.00281301,
+     -0.02129791,
+     0.0113186,
+     0.00275263,
+     0.01533038,
+     -0.02215966,
+     -0.00148776,
+     -0.04362523,
+     -0.02110263,
+     -0.03406411,
+     0.00858526,
+     0.00519482,
+     -0.01531371,
+     -0.02032654,
+     -0.00879037,
+     -0.01910952,
+     -0.00970773,
+     -0.00597221,
+     0.02600741,
+     -0.01500964,
+     0.00282593,
+     0.00812054,
+     0.01510925,
+     0.01722519,
+     -0.01325881,
+     -0.03556012,
+     -0.02189088,
+     -0.0053198,
+     0.02048756,
+     0.02681401,
+     -0.01647379,
+     -0.00249846,
+     -0.01893351,
+     0.00662435,
+     0.02269262,
+     0.01222809,
+     -0.0210267,
+     -0.00039488,
+     -0.0512335,
+     0.00088038,
+     -0.02341066,
+     -0.00038121,
+     0.01761548,
+     0.02631648,
+     0.01183771,
+     0.02191784,
+     0.00207306,
+     -0.01902432,
+     -0.02336112,
+     -0.00491415,
+     -0.00072113,
+     0.00028628,
+     -0.01824824,
+     0.01573452,
+     -0.03276414,
+     -0.00953589,
+     0.01647868,
+     -0.01684641,
+     0.00790728,
+     -0.01832088,
+     0.01247018,
+     0.00042123,
+     0.0078352,
+     0.00828961,
+     -0.0020343,
+     -0.01149357,
+     -0.02579743,
+     -0.01467749,
+     0.00252954,
+     0.00356522,
+     -0.0229656,
+     0.00305361,
+     0.0100733,
+     0.00604701,
+     0.00316089,
+     0.0172147,
+     -0.00011459,
+     0.01843379,
+     0.0113596,
+     -0.01007929,
+     -0.01242581,
+     0.00019472,
+     -0.02393127,
+     -0.04665328,
+     0.01382595,
+     0.00461308,
+     -0.02666152,
+     -0.03019182,
+     0.00341553,
+     -0.0221114,
+     0.00693722,
+     -0.00555421,
+     -0.02032205,
+     -0.02492155,
+     0.03080052,
+     -0.0129261,
+     -0.02659664,
+     -0.02317971,
+     -3.857e-05,
+     0.01464641,
+     0.02506778,
+     -0.00284662,
+     -0.00378326,
+     0.0097126,
+     -0.0022778,
+     -0.00778427,
+     0.0195424,
+     0.00874459,
+     -0.03001611,
+     0.01055702,
+     0.00944672,
+     0.00660469
+    ],
+    "cols": [
+     1,
+     5,
+     8,
+     11,
+     13,
+     15,
+     17,
+     18,
+     34,
+     36,
+     44,
+     55,
+     63,
+     66,
+     69,
+     92,
+     93,
+     95,
+     98,
+     114,
+     119,
+     127,
+     145,
+     152,
+     161,
+     168,
+     171,
+     195,
+     246,
+     259,
+     265,
+     343,
+     364,
+     445,
+     506,
+     544,
+     621,
+     625,
+     636,
+     744,
+     766,
+     770,
+     799,
+     877,
+     965,
+     1003,
+     1346,
+     1410,
+     1505,
+     1704,
+     1799,
+     2072,
+     2135,
+     2175,
+     2248,
+     2314,
+     2376,
+     2437,
+     2514,
+     2595,
+     2664,
+     2684,
+     2816,
+     3343
+    ]
+   }
+  }
+ }
+}

code/configs/variants.json

@@ -0,0 +1,29 @@
+{
+ "_doc": "Each entry becomes a model folder qwen35-<name>. Per edited layer: alpha (strength), dir ('released' = the paper's direction u for that layer, 'random_orth:SEED' = a random direction at right angles to u), cols ('released' = the paper's 64 neurons, 'all' = every neuron, 'random:SEED' = 64 other neurons picked at random), norm_match (rescale alpha so the weight change is as big as the released edit's). The released edit itself is qwen35-p50c (alpha 6, both layers).",
+ "a-4": {"about": "the knob turned the other way: makes the 64 neurons push 5x harder along the loop direction",
+         "layers": {"15": {"alpha": -4}, "19": {"alpha": -4}}},
+ "a1": {"about": "remove the loop component from the 64 neurons (projection)",
+        "layers": {"15": {"alpha": 1}, "19": {"alpha": 1}}},
+ "a2": {"about": "mirror the loop component (reflection): same column sizes, opposite push",
+        "layers": {"15": {"alpha": 2}, "19": {"alpha": 2}}},
+ "a4": {"about": "flip and triple the loop component",
+        "layers": {"15": {"alpha": 4}, "19": {"alpha": 4}}},
+ "a8": {"about": "stronger than released: flip and multiply by 7",
+        "layers": {"15": {"alpha": 8}, "19": {"alpha": 8}}},
+ "a12": {"about": "much stronger than released: flip and multiply by 11",
+         "layers": {"15": {"alpha": 12}, "19": {"alpha": 12}}},
+ "l15only": {"about": "the released edit on layer 15 only",
+             "layers": {"15": {"alpha": 6}}},
+ "l19only": {"about": "the released edit on layer 19 only",
+             "layers": {"19": {"alpha": 6}}},
+ "randdir": {"about": "control: same 64 neurons, same size of change, but a random direction instead of u",
+             "layers": {"15": {"alpha": 6, "dir": "random_orth:15", "norm_match": true},
+                        "19": {"alpha": 6, "dir": "random_orth:19", "norm_match": true}}},
+ "randcols": {"about": "control: direction u, same size of change, but 64 random neurons instead of the chosen ones",
+              "layers": {"15": {"alpha": 6, "cols": "random:15", "norm_match": true},
+                         "19": {"alpha": 6, "cols": "random:19", "norm_match": true}}},
+ "all1": {"about": "an earlier edit from the agent's run (strength 1, +24 points): remove the loop component from every neuron",
+          "layers": {"15": {"alpha": 1, "cols": "all"}, "19": {"alpha": 1, "cols": "all"}}},
+ "all2": {"about": "an earlier edit from the agent's run (strength 2, +35 points): mirror the loop component in every neuron",
+          "layers": {"15": {"alpha": 2, "cols": "all"}, "19": {"alpha": 2, "cols": "all"}}}
+}

code/data/README.md

@@ -0,0 +1,21 @@
+# Problem files
+
+`scripts/prepare_data.py` writes the five problem files that every lab reads.
+
+| File | What it holds |
+|---|---|
+| `math500.jsonl` | The MATH-500 test set (500 problems), one of the paper's three benchmarks |
+| `gsm8k.jsonl` | The GSM8K main test set (1,319 problems), a benchmark |
+| `ifeval.jsonl` | IFEval (541 prompts), a benchmark graded by its official checkers |
+| `dev256.jsonl` | 128 GSM8K-train and 128 MATH-train practice problems the paper's agent never used. We measure every edit here. |
+| `build128.jsonl` | 64 + 64 more unused practice problems. The Lab 8 capstone builds its own edit from these. |
+
+The copy of this project on Hugging Face leaves the `.jsonl` files out. Three of them are benchmark test sets, and test sets posted online can end up in the training data of future models. The script rebuilds all five from public sources.
+
+To rebuild them, run this from the project root on a machine with internet access and the `datasets` package (`scripts/setup_vm.sh` installs it on the GPU machines):
+
+```bash
+git clone https://github.com/RikiyaT/lm_understand-logs external/lm_understand-logs
+git -C external/lm_understand-logs checkout 4d09ba6d   # the commit we used
+python scripts/prepare_data.py --exp8 external/lm_understand-logs/exp8_math --out data
+```

code/labs/data/astra_pairs.json

@@ -0,0 +1,24 @@
+{
+ "source": "lm_understand-logs exp8_math/runs/p50c/tool_calls.jsonl row 8 (the agent's 4th forward pass, 'fwd_4')",
+ "how": "pairs: index 0,2,4,6 productive; 1,3,5,7 repetitive. Same prompt and same opening sentences inside a pair.",
+ "agent_token_lengths": [
+  124,
+  123,
+  131,
+  136,
+  107,
+  146,
+  98,
+  135
+ ],
+ "texts": [
+  "<|im_start|>user\nRafael works 10 hours on Monday and 8 on Tuesday, with 20 hours left to work. He earns $20 per hour. Find his weekly pay.<|im_end|>\n<|im_start|>assistant\n<think>\nThe first two days total 10+8=18 hours. Adding the remaining 20 gives 38 hours. His pay is 38*20=760 dollars. Checking: 200+160+400=760, consistent with the total. The hours and hourly rate have the correct units.",
+  "<|im_start|>user\nRafael works 10 hours on Monday and 8 on Tuesday, with 20 hours left to work. He earns $20 per hour. Find his weekly pay.<|im_end|>\n<|im_start|>assistant\n<think>\nThe first two days total 10+8=18 hours. Adding the remaining 20 gives 38 hours. But is 20 the total? No, it is the remaining time. But is 20 the total? No, it is the remaining time. But is 20 the total? No, it is the remaining time.",
+  "<|im_start|>user\nFind the remainder when 6x^4-14x^3-4x^2+2x-26 is divided by 2x-6.<|im_end|>\n<|im_start|>assistant\n<think>\nSet 2x-6=0, giving x=3. The remainder is P(3)=486-378-36+6-26=52. To justify this, write P(x)=(2x-6)Q(x)+R. Substituting x=3 cancels the quotient term, leaving P(3)=R. Hence the constant remainder is 52.",
+  "<|im_start|>user\nFind the remainder when 6x^4-14x^3-4x^2+2x-26 is divided by 2x-6.<|im_end|>\n<|im_start|>assistant\n<think>\nSet 2x-6=0, giving x=3. The remainder is P(3)=486-378-36+6-26=52. But P(x) has -26. The constant term is -26. But P(x) has -26. The constant term is -26. But P(x) has -26. The constant term is -26.",
+  "<|im_start|>user\nThe sequence is 88,85,82,... How many terms precede -17?<|im_end|>\n<|im_start|>assistant\n<think>\nThe common difference is -3. Solve 88-3(n-1)=-17. Then 3(n-1)=105, so n=36. The target is the 36th term, with 35 terms preceding it. Checking: the 35th term is -14, followed by -17.",
+  "<|im_start|>user\nThe sequence is 88,85,82,... How many terms precede -17?<|im_end|>\n<|im_start|>assistant\n<think>\nThe common difference is -3. Solve 88-3(n-1)=-17. Then 3(n-1)=105, so n=36. Now 88-85=3 and 85-82=3. Now 88-85=3 and 85-82=3. Now 88-85=3 and 85-82=3. Now 88-85=3 and 85-82=3.",
+  "<|im_start|>user\nA machine makes 45 necklaces, and another makes 2.4 times as many. Find the total.<|im_end|>\n<|im_start|>assistant\n<think>\nThe second machine makes 45*2.4=108 necklaces. Add the first machine's 45 to obtain 153 necklaces. Multiplication gives the second quantity, while addition combines the two machines. A check is 45*3.4=153.",
+  "<|im_start|>user\nA machine makes 45 necklaces, and another makes 2.4 times as many. Find the total.<|im_end|>\n<|im_start|>assistant\n<think>\nThe second machine makes 45*2.4=108 necklaces. Add the first machine's 45 to obtain 153 necklaces. Is 108 the total? No, 45+108=153. Is 108 the total? No, 45+108=153. Is 108 the total? No, 45+108=153."
+ ]
+}

code/labs/lab0_toy.md

@@ -0,0 +1,157 @@
+# Lab 0: The toy edit on a 4 x 6 matrix
+
+**Big thing:** You will run the paper's weight edit `W' = W - alpha * u (u^T W_S)` on a 4 x 6 matrix that fits on one screen. The edit changes how hard two chosen neurons push along one direction, leaves everything else alone, and the script checks each claim with `assert`.
+
+**Why it matters:** In Qwen3.5-0.8B the same matrix is 1,024 x 3,584, too large to read by eye, but the linear algebra is identical.
+
+**Time and hardware:** 5 seconds on any laptop CPU.
+
+**Theory guide:** Sections 6 and 8.
+
+## Before you run: predict
+
+Write a short guess for each question before running the script.
+
+1. In the toy layer, neuron 1 pushes along `u` by `-0.555` per unit of activation. With edit strength `alpha = 1`, what does neuron 1's push along `u` become? What if `alpha = 6`?
+2. The toy matrix has 6 columns (one per neuron), and the edit touches only neurons `S = [1, 4]`. How many columns of `W' - W` will be non-zero?
+3. On a token where neurons 1 and 4 have activation `0.0` and the other four neurons fire normally, how much does the edited output `W' a` differ from `W a`?
+
+## Run it
+
+Run this from the project root (`uv` creates a temporary environment with `numpy` and `matplotlib`):
+
+```bash
+uv run --with numpy --with matplotlib python labs/lab0_toy.py
+```
+
+The script prints four checks, verifies each with `assert`, and writes `results/figures/lab0_toy.png`.
+
+## What you should see
+
+```text
+1. How much each neuron writes along u (u . W[:, j]): [ 1.062 -0.555  1.043 -0.495  0.318  0.162]
+
+2. The edit scales the chosen neurons' push along u by (1 - alpha) and leaves everything else alone
+   alpha= 0 (no edit                        ) push along u of neurons [1, 4]: [-0.555  0.318] -> [-0.555  0.318]
+   alpha= 1 (erase                          ) push along u of neurons [1, 4]: [-0.555  0.318] -> [ 0. -0.]
+   alpha= 2 (mirror                         ) push along u of neurons [1, 4]: [-0.555  0.318] -> [ 0.555 -0.318]
+   alpha= 6 (flip and amplify 5x (the paper)) push along u of neurons [1, 4]: [-0.555  0.318] -> [ 2.776 -1.592]
+   alpha=-4 (amplify 5x the other way       ) push along u of neurons [1, 4]: [-0.555  0.318] -> [-2.776  1.592]
+
+3. The change is rank one: one direction out (u), one pattern in (u^T W on the chosen columns)
+   singular values of W' - W: [3.84 0.   0.   0.  ]
+
+4. The weight edit is steering: W' a = W a - alpha * s * u, with s = u . (W[:, S] a[S])
+   chosen neurons quiet   s = +0.000   change to the output |W'a - Wa| = 0.000
+   chosen neurons firing  s = -0.710   change to the output |W'a - Wa| = 4.261
+   Same weights, different tokens, different push: the edit only acts when the chosen neurons fire.
+
+all checks passed; figure in results/figures/lab0_toy.png
+```
+
+![Left: heatmap of W' minus W, with changes only in the columns of neurons 1 and 4. Right: the multiplier 1 minus alpha, falling from 1 at alpha 0 to -5 at alpha 6.](../results/figures/lab0_toy.png)
+
+On the left of `lab0_toy.png`, the `W' - W` heatmap has four white columns at neurons 0, 2, 3, and 5 (exact zeros) and two colored columns at neurons 1 and 4. In every row, column 4 has the opposite color of column 1. Neuron 1 started with a negative push along `u` (`-0.555`) and neuron 4 started with a positive push (`0.318`), so the edit moves them in opposite directions along `u`. On the right, the blue line plots the multiplier `1 - alpha`: `1` at `alpha = 0` (no edit), `0` at `alpha = 1` (erase), `-1` at `alpha = 2` (mirror), and `-5` at `alpha = 6` (the paper's setting).
+
+## Why it works
+
+### 1. Neurons write columns into the residual stream
+
+A language model processes text one token (word piece) at a time. Across the 24 blocks (layers) of Qwen3.5-0.8B, the model updates a running list of numbers for each token called the residual stream. That stream holds 1,024 numbers per token in Qwen3.5-0.8B and `d = 4` in `lab0_toy.py`.
+
+Each block ends with an MLP (multi-layer perceptron). Inside it, two input matrices (`gate_proj` and `up_proj`) and a SiLU nonlinearity turn the stream into `n` neuron activations `a[0], ..., a[n-1]` (`n = 3,584` in Qwen3.5-0.8B, `n = 6` in the toy). Each activation `a[j]` is a single number measuring how strongly neuron `j` fires on the current token. The MLP's last matrix, `down_proj`, is `W` of shape `d x n`. Column `W[:, j]` is the `d`-number vector that neuron `j` adds to the stream when it fires with activation `1`. The MLP output is the weighted sum of its columns:
+
+$$y = W a = \sum_{j=0}^{n-1} a_j W[:, j]$$
+
+```mermaid
+flowchart LR
+  A["Neuron activations<br/>a[0] ... a[5]"] --> B["Column j of W is W[:, j]<br/>(what neuron j writes)"]
+  B --> C["Weighted sum<br/>y = sum of a[j] * W[:, j]"]
+  C --> D["Added to the<br/>residual stream"]
+```
+
+### 2. Projecting a column onto the direction `u`
+
+A direction `u` is a residual-stream vector scaled to length 1 (`u . u = 1`). In the paper, `u` points toward repetitive looping. The dot product of `u` with column `W[:, j]` measures how much neuron `j` pushes the stream along `u` per unit of activation:
+
+$$g_j = u \cdot W[:, j]$$
+
+Across all columns, the row vector `g = u @ W` holds those six numbers: `[1.062, -0.555, 1.043, -0.495, 0.318, 0.162]`.
+
+Every column `W[:, j]` splits into two perpendicular parts: `g_j u` along `u`, and `W[:, j] - g_j u` at right angles to `u`.
+
+### 3. What the edit does to one chosen column
+
+The edit picks a neuron subset `S` (here `S = [1, 4]`) and a strength `alpha`, then subtracts `alpha * g_j u` from each column `j` in `S`:
+
+$$W'[:, j] = W[:, j] - \alpha (u \cdot W[:, j]) u$$
+
+Columns outside `S` stay unchanged (`W'[:, j] = W[:, j]`). Writing `W_S` for `W` with columns outside `S` zeroed out gives the paper's matrix formula:
+
+$$W' = W - \alpha u (u^T W_S)$$
+
+To find an edited column's new push along `u`, take the dot product with `u` and use `u . u = 1`:
+
+$$u \cdot W'[:, j] = u \cdot W[:, j] - \alpha (u \cdot W[:, j])(u \cdot u) = (1 - \alpha)(u \cdot W[:, j])$$
+
+The part of `W[:, j]` perpendicular to `u` does not change, because `u` has zero dot product with any perpendicular direction.
+
+```mermaid
+flowchart TD
+  Col["Chosen column W[:, j]<br/>(neuron 1: g_1 = -0.555)"] --> Split["Split into two parts at right angles"]
+  Split --> Par["Part along u:<br/>g_j * u"]
+  Split --> Perp["Part at right angles to u:<br/>W[:, j] - g_j * u"]
+  Par --> Scale["Multiply by (1 - alpha)<br/>At alpha = 6: -5 * g_j * u"]
+  Perp --> Keep["Keep unchanged"]
+  Scale --> Out["Edited column W'[:, j]<br/>(neuron 1 push along u: 2.776)"]
+  Keep --> Out
+```
+
+Compare this with the printed numbers for neuron 1 (`g_1 = -0.555`) and neuron 4 (`g_4 = 0.318`):
+
+- At `alpha = 1` (erase), the multiplier `1 - 1 = 0` sends `-0.555` and `0.318` to `0`. Both neurons still write perpendicular to `u`, while their push along `u` is zeroed out.
+- At `alpha = 2` (mirror), the multiplier `1 - 2 = -1` flips `-0.555` to `+0.555` and `0.318` to `-0.318`.
+- At `alpha = 6` (the paper's value), the multiplier is `1 - 6 = -5`. For neuron 1, `-5 x -0.555 = +2.775`, while the script prints `2.776` from the unrounded `-0.555157` (`-5 x -0.555157 = 2.776`). For neuron 4, `-5 x 0.318416 = -1.592`.
+
+### 4. Why the change is rank one, and why editing weights equals steering
+
+The matrix change `D = W' - W` equals `-alpha * outer(u, g_masked)`. Every column of `D` is a multiple of `u`, and a single column vector times a row vector has rank one. Running SVD (singular value decomposition, which splits a matrix into independent rank-one pieces ordered by size) on `D` gives a first singular value of `3.84` and three zeros (`0`). Since `u` has length 1, `3.84` equals `|alpha| * ||g_S|| = 6 * sqrt((-0.5552)^2 + (0.3184)^2) = 3.840`.
+
+Now multiply `W'` by an activation vector `a` on one token:
+
+$$W' a = W a - \alpha u (u^T W_S a) = W a - \alpha s(t) u$$
+
+Here `s(t) = u . (W_S a)` is how hard the chosen neurons push along `u` on token `t`. Since `W_S` keeps only columns in `S`, `W_S a` equals `W[:, S] a[S]`, the activation-weighted sum of those columns. When neurons 1 and 4 are quiet (`a[1] = a[4] = 0`), `s = 0.000` and the output is unchanged (`|W'a - Wa| = 0.000`). When both fire at `3.0`, `s = 3.0 x (-0.5552 + 0.3184) = -0.710`, shifting the output along `u` by `|alpha * s| = 6 x 0.710 = 4.261`. A permanent weight edit therefore acts like adding a steering vector `-alpha * s(t) * u` whose strength `s(t)` comes from the token's own activations.
+
+## Check yourself
+
+1. What value of `alpha` makes the chosen neurons push 3 times as hard in their original direction along `u`?
+<details><summary>Answer</summary>
+
+Set `1 - alpha = 3`, which gives `alpha = -2`. Any negative `alpha` amplifies the original push along `u` instead of opposing it.
+</details>
+
+2. Why does the SVD of `W' - W` have only one non-zero singular value even though we changed two columns (`S = [1, 4]`)?
+<details><summary>Answer</summary>
+
+Both changed columns were shifted along the same direction `u` (column 1 moved by `+3.331 u` and column 4 moved by `-1.910 u`). Because every column of `W' - W` is a multiple of one vector `u`, the entire matrix change has rank one.
+</details>
+
+3. If you replace the direction `u` with `-u` and keep `alpha = 6` and `S = [1, 4]`, how does the edited matrix `W'` change?
+<details><summary>Answer</summary>
+
+It does not change. In `W' = W - alpha * u (u^T W_S)`, `u` appears twice in `u u^T`, and `(-u)(-u)^T = u u^T`. Only the line through `u` matters, not which end gets the plus sign.
+</details>
+
+4. On a token where neuron 1 fires with `a[1] = 2.0` and neuron 4 fires with `a[4] = 0.0`, what is `s`, and how big is the change `|W'a - Wa|` at `alpha = 6`?
+<details><summary>Answer</summary>
+
+`s = a[1] * g_1 = 2.0 * (-0.5552) = -1.110`. Since `u` has length 1, the output changes by `|alpha * s| = 6 * 1.110 = 6.662` along `u`.
+</details>
+
+## Try this next
+
+1. **Test that `u` and `-u` give the exact same edit.** In `labs/lab0_toy.py`, after `D = edit(W, u, 6, S) - W` (line 48), add `assert np.allclose(edit(W, u, 6, S), edit(W, -u, 6, S))` and re-run. All assertions pass, showing why SVD sign flips in Lab 1 do not change the edit.
+2. **Touch 4 neurons instead of 2.** Change line 21 from `S = [1, 4]` to `S = [0, 1, 2, 4]` and re-run. The heatmap in `results/figures/lab0_toy.png` shows four colored columns instead of two, and step 3 still prints `0.` for the second, third, and fourth singular values, because the edit is still rank one.
+3. **Add `alpha = -2` to the table.** On lines 38 and 39, add `(-2, "amplify 3x original way")` and re-run. The push along `u` of neurons `[1, 4]` triples from `[-0.555, 0.318]` to `[-1.665, 0.955]`.
+4. **Scale up to the real layer dimensions.** Change line 17 from `d, n = 4, 6` to `d, n = 1024, 3584` and line 21 to `S = list(range(64))`, then re-run. Every `assert` passes in a few seconds. Numpy shortens arrays over 1,000 numbers with `...` in step 1, while step 2 prints 64 numbers per line. Step 4 prints `s = -35.717` and a change of `214.301` when the chosen neurons fire. The output moves much more than in the toy because 64 neurons push along `u` together.

code/labs/lab0_toy.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python
+"""Lab 0: weight-space programming on a toy layer small enough to print.
+
+    uv run --with numpy --with matplotlib python labs/lab0_toy.py      # seconds, any laptop
+
+An MLP's last matrix, down_proj, is W (here 4 x 6; in Qwen3.5-0.8B it is 1,024 x 3,584). Neuron j fires with
+strength a[j] and writes its column W[:, j] into the stream, so the layer adds y = W a = sum_j a[j] * W[:, j].
+The paper's edit picks a direction u and some neurons S and sets  W' = W - alpha * u (u^T W[:, S])  on those
+columns. Every claim below is checked with an assert, so if this script finishes, they all hold.
+"""
+import os
+
+import numpy as np
+
+np.set_printoptions(precision=3, suppress=True)
+rng = np.random.default_rng(0)
+d, n = 4, 6
+W = rng.normal(size=(d, n))
+u = rng.normal(size=d)
+u /= np.linalg.norm(u)            # the direction we care about (in the paper: "toward looping")
+S = [1, 4]                        # the neurons we allow ourselves to touch (in the paper: 64 of 3,584)
+others = [j for j in range(n) if j not in S]
+
+
+def edit(W, u, alpha, S=None):
+    g = u @ W                     # g[j] = how much neuron j writes along u, per unit of activation
+    if S is not None:
+        mask = np.zeros(W.shape[1])
+        mask[S] = 1
+        g = g * mask
+    return W - alpha * np.outer(u, g)
+
+
+print("1. How much each neuron writes along u (u . W[:, j]):", u @ W)
+
+print("\n2. The edit scales the chosen neurons' push along u by (1 - alpha) and leaves everything else alone")
+P = np.eye(d) - np.outer(u, u)    # removes the u part of any vector
+for alpha, name in [(0, "no edit"), (1, "erase"), (2, "mirror"), (6, "flip and amplify 5x (the paper)"),
+                    (-4, "amplify 5x the other way")]:
+    W2 = edit(W, u, alpha, S)
+    before, after = (u @ W)[S], (u @ W2)[S]
+    assert np.allclose(after, (1 - alpha) * before)
+    assert np.allclose(W2[:, others], W[:, others])     # untouched neurons stay identical
+    assert np.allclose(P @ W2, P @ W)                     # nothing changes outside the direction u
+    print(f"   alpha={alpha:>2} ({name:31s}) push along u of neurons {S}: {before} -> {after}")
+
+print("\n3. The change is rank one: one direction out (u), one pattern in (u^T W on the chosen columns)")
+D = edit(W, u, 6, S) - W
+sv = np.linalg.svd(D, compute_uv=False)
+assert sv[1] < 1e-12 * sv[0]
+print("   singular values of W' - W:", sv)
+
+print("\n4. The weight edit is steering: W' a = W a - alpha * s * u, with s = u . (W[:, S] a[S])")
+alpha = 6
+for name, a in [("chosen neurons quiet", np.where(np.isin(np.arange(n), S), 0.0, rng.normal(size=n))),
+                ("chosen neurons firing", np.where(np.isin(np.arange(n), S), 3.0, rng.normal(size=n)))]:
+    s = u @ (W[:, S] @ a[S])
+    y_edit, y_steer = edit(W, u, alpha, S) @ a, W @ a - alpha * s * u
+    assert np.allclose(y_edit, y_steer)
+    print(f"   {name:22s} s = {s:+.3f}   change to the output |W'a - Wa| = {np.linalg.norm(y_edit - W @ a):.3f}")
+print("   Same weights, different tokens, different push: the edit only acts when the chosen neurons fire.")
+
+import matplotlib  # noqa: E402
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+
+fig, ax = plt.subplots(1, 2, figsize=(11, 3.6))
+im = ax[0].imshow(edit(W, u, 6, S) - W, cmap="RdBu", vmin=-6, vmax=6)
+ax[0].set_title("W' - W: only the chosen columns change")
+ax[0].set_xlabel("neuron (column)")
+ax[0].set_ylabel("stream dimension (row)")
+fig.colorbar(im, ax=ax[0])
+al = np.linspace(-5, 8, 100)
+ax[1].plot(al, 1 - al, color="#0969da")
+for x, lab in [(0, "no edit"), (1, "erase"), (2, "mirror"), (6, "the paper")]:
+    ax[1].plot(x, 1 - x, "o", color="#cf222e")
+    ax[1].annotate(lab, (x, 1 - x), xytext=(6, 4), textcoords="offset points")
+ax[1].axhline(0, color="black", lw=0.5)
+ax[1].set_xlabel("alpha")
+ax[1].set_ylabel("push along u after / before")
+ax[1].set_title("What alpha does to the chosen neurons")
+fig.tight_layout()
+out = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "results", "figures")
+os.makedirs(out, exist_ok=True)
+fig.savefig(os.path.join(out, "lab0_toy.png"), dpi=130)
+print("\nall checks passed; figure in results/figures/lab0_toy.png")

code/labs/lab1_diff.md

@@ -0,0 +1,166 @@
+# Lab 1: Find the edit by diffing the two weight files
+
+**Big thing:** You will compare all 488 weight tensors (873,438,784 numbers) of base Qwen3.5-0.8B and the paper's released model bit for bit. From the weights alone, you recover the full edit: `layers = [15, 19]`, `64` columns per layer, rank-one direction `u`, and strength `alpha = 6`.
+
+**Why it matters:** When you have the weight files, you do not have to take a paper's description on faith. Tensor diffs and an SVD turn a 0.87-billion-parameter checkpoint diff into one short formula.
+
+**Time and hardware:** Under a minute on CPU on `sampler` or `trainer` (no GPU needed).
+
+**Theory guide:** Section 9, step 4.
+
+## Before you run: predict
+
+Write a short guess for each question before running the script.
+
+1. Out of the 488 weight tensors in Qwen3.5-0.8B, how many differ between the base model and the released model `qwen35-p50c`?
+2. Each edited `down_proj.weight` matrix has shape `1,024 x 3,584`, and the edit touches `64` columns (`64 x 1,024 = 65,536` numbers). Will the file have `65,536` changed entries or slightly fewer, and why?
+3. In Lab 0, the second singular value of `W' - W` was `0.0000`. For the real weight change, will the second singular value be exactly zero or slightly above zero?
+
+## Run it
+
+Connect to one of the lab VMs (`sampler` or `trainer`), activate the lab environment, and run `labs/lab1_diff.py` on CPU. This script does not touch a GPU.
+
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+python labs/lab1_diff.py
+```
+
+The script compares every tensor in `/nvme/wsp-lab/models/Qwen3.5-0.8B` against `/nvme/wsp-lab/models/qwen35-p50c`, writes `results/lab1/diff.json`, and saves `results/figures/lab1_diff.png`.
+
+## What you should see
+
+```text
+2 of 488 tensors differ (873M numbers in all):
+   model.language_model.layers.15.mlp.down_proj.weight  shape [1024, 3584]
+      entries changed: 65,260 of 3,670,016, all inside 64 columns (neurons)
+      singular values of the change: [1.7489, 0.001, 0.0009, 0.0009] ...  rank-one share 0.999994
+      fitted alpha = 6.0003; the matrix moved by 10.6%
+   model.language_model.layers.19.mlp.down_proj.weight  shape [1024, 3584]
+      entries changed: 64,928 of 3,670,016, all inside 64 columns (neurons)
+      singular values of the change: [1.3988, 0.0009, 0.0008, 0.0007] ...  rank-one share 0.999991
+      fitted alpha = 6.0002; the matrix moved by 8.1%
+|cos| between the two directions: 0.849 (1 = same direction, 0 = unrelated; the SVD sign is arbitrary, Lab 3 fixes it)
+wrote /nvme/wsp-lab/code/results/lab1
+```
+
+`results/lab1/diff.json` and `results/edit_info.json` record the exact numbers:
+
+| Measurement | Layer 15 (`down_proj.weight`) | Layer 19 (`down_proj.weight`) |
+|---|---|---|
+| Matrix shape (rows x columns) | `1,024 x 3,584` (`3,670,016` entries) | `1,024 x 3,584` (`3,670,016` entries) |
+| Entries changed | `65,260` | `64,928` |
+| Columns (neurons) containing every change | `64` | `64` |
+| First 8 changed columns | `0, 2, 11, 25, 34, 35, 38, 39` | `1, 5, 8, 11, 13, 15, 17, 18` |
+| Top 4 singular values of `W1 - W0` | `1.7489, 0.0010, 0.0009, 0.0009` | `1.3988, 0.0009, 0.0008, 0.0007` |
+| Rank-one share ($s_1^2 / \sum_i s_i^2$) | `0.999994` | `0.999991` |
+| Fitted strength `alpha` | `6.0003` | `6.0002` |
+| Relative matrix change ($\|W_1 - W_0\|_F / \|W_0\|_F$) | `10.6%` (`0.10636`) | `8.1%` (`0.08106`) |
+| Absolute cosine between $u_{15}$ and $u_{19}$ | `0.849` | `0.849` |
+
+![Top row: size of the change in each of the 3,584 columns of layers 15 and 19, with 64 spikes and exact zeros elsewhere. Bottom row: the top 10 singular values on a log scale, with the first far above the rest.](../results/figures/lab1_diff.png)
+
+In `lab1_diff.png`, the top row plots the Euclidean length (norm) of the change in each of the 3,584 columns for layers 15 and 19. You see 64 red spikes rising from a floor of exact zeros; in layer 19, the tallest spike near the left edge is neuron 11. The bottom row plots the top 10 singular values on a logarithmic scale. Bar 1 sits above `10^0` (`1.7489` and `1.3988`), while bars 2 through 10 sit more than three orders of magnitude lower around `10^-3`.
+
+## Why it works
+
+### 1. The detective pipeline: from two files to four numbers
+
+The script inspects the two model folders in four steps, using no prior knowledge of how the paper built its edit.
+
+```mermaid
+flowchart LR
+  A["488 tensors in base and p50c<br/>(873,438,784 numbers)"] --> B["Bitwise check:<br/>only 2 tensors differ<br/>(layers 15 and 19 down_proj)"]
+  B --> C["Column norms of D = W1 - W0:<br/>nonzero in 64 of 3,584 columns"]
+  C --> D["SVD of D[:, cols]:<br/>first singular value holds<br/>> 99.999% of the squared change<br/>left vector = u"]
+  D --> E["Least-squares fit on W0:<br/>recover alpha = 6.0003 and 6.0002"]
+```
+
+First, `safe_open` streams all 488 tensors in `model.safetensors-00001-of-00001.safetensors` and checks `torch.equal(tb, te)`. Exactly 486 are bit-for-bit identical; only `model.language_model.layers.15.mlp.down_proj.weight` and `model.language_model.layers.19.mlp.down_proj.weight` differ.
+
+Second, the script subtracts the base weights from the edited weights in 32-bit floating point (`fp32`): `D = W1.float() - W0.float()`. Checking which columns have a non-zero sum of absolute differences (`D.abs().sum(0) > 0`) shows that every changed entry falls inside 64 columns in layer 15 and 64 columns in layer 19, leaving the other `3,520` columns untouched. Column `11` in layer 19's list `[1, 5, 8, 11, ...]` is the single neuron the agent zeroed out early in its search (+6.25 percentage points on practice problems) before moving to a 64-neuron edit.
+
+### 2. Why 65,260 and 64,928 entries changed instead of 65,536
+
+64 columns times 1,024 rows is `65,536` entries per layer, yet only `65,260` entries change in layer 15 (`276` unchanged) and `64,928` in layer 19 (`608` unchanged).
+
+The reason is number precision. The weights are stored on disk in `bf16` (bfloat16, a 16-bit floating-point format with 2 to 3 significant decimal digits). Where `u` or `u . W0[:, j]` is near zero, the update `-alpha * u[i] * (u . W0[:, j])` is smaller than half a `bf16` step around `W0[i, j]`. Casting the edited tensor from `fp32` back to `bf16` rounds those entries back to their original bits.
+
+### 3. Taking the change apart with SVD
+
+To check whether all 64 columns moved along one direction `u` or 64 separate directions, `recover(W0, W1)` in `wsp/edits.py` takes the `1,024 x 64` slice `D[:, cols]` and computes its Singular Value Decomposition (SVD):
+
+$$D[:, \text{cols}] = U \Sigma V^T = \sum_{k=1}^{64} s_k u_k v_k^T$$
+
+SVD writes a matrix as a sum of rank-one pieces. The singular values $s_1 \ge s_2 \ge \dots \ge s_{64} \ge 0$ give the size of each piece, and the columns $u_k$ of $U$ are mutually perpendicular unit vectors.
+
+- In layer 15, $s_1 = 1.7489$, while $s_2 = 0.0010$, $s_3 = 0.0009$, and $s_4 = 0.0009$. The first piece holds `0.999994` (99.9994%) of the squared Frobenius norm ($s_1^2 / \sum_k s_k^2$).
+- In layer 19, $s_1 = 1.3988$, while $s_2 = 0.0009$, $s_3 = 0.0008$, and $s_4 = 0.0007$. The rank-one share is `0.999991` (99.9991%).
+
+The change is a rank-one outer product; the tail of singular values near `0.001` is consistent with `bf16` rounding noise. The top left singular vector `u = U[:, 0]` is the edit direction `u` up to a sign.
+
+### 4. Recovering `alpha` and checking the bits
+
+With `u` and `cols` known from `D`, the unrounded change on the chosen columns is $-\alpha P$, where $P = u (u^T W_0[:, \text{cols}])$ is the projection of the base columns onto `u`. Here $P$ depends only on the line through `u`, since $(-u)(-u)^T = u u^T$. A least-squares fit in `recover` finds the scalar `alpha` that best matches `D[:, cols]` to `-alpha * P`:
+
+$$\alpha = -\frac{\sum_{i,j} D_{i,j} P_{i,j}}{\sum_{i,j} P_{i,j}^2}$$
+
+This gives `6.0003` for layer 15 and `6.0002` for layer 19, matching the paper's `alpha = 6` to four digits.
+
+```mermaid
+flowchart TD
+  SVD["SVD of D[:, cols]<br/>gives u (up to sign +/-1)"] --> P["Compute rank-1 template<br/>P = u (u^T W0[:, cols])<br/>(sign of u cancels out)"]
+  P --> Fit["Least-squares ratio<br/>alpha = -sum(D * P) / sum(P * P)<br/>gives 6.0003 and 6.0002"]
+  Fit --> Rebuild["Plug u, cols, alpha=6 into formula<br/>Compare with released bf16 bits:<br/>matches to <= 0.00049 (1 bf16 step)"]
+```
+
+In `scripts/build_models.py` (logged in `results/edit_info.json`), we plug the recovered `u`, `cols`, and `alpha` into `projection_edit` on `W0` and compare the resulting `bf16` tensor against `W1`. Out of `3,670,016` entries per layer, all but `5,875` in layer 15 (`5,924` at `alpha = 6.0`) and `6,158` in layer 19 (`6,166` at `alpha = 6.0`) match bit for bit. The largest gap anywhere is `0.00048828125` (about `0.00049`), one `bf16` rounding step.
+
+Comparing the unit vectors $u_{15}$ and $u_{19}$ gives $|u_{15} \cdot u_{19}| = 0.849$. An absolute cosine of `1.0` means two vectors lie on the same line, pointing the same way or opposite ways, and `0.0` means they are orthogonal (unrelated). A value of `0.849` is an angle of about 32 degrees: the loop directions at blocks 15 and 19 are closely related in the 1,024-dimensional stream, but distinct.
+
+## Check yourself
+
+1. Why does `labs/lab1_diff.py` report the absolute value `|cos| = 0.849` between `u1` and `u2` instead of a signed cosine?
+<details><summary>Answer</summary>
+
+SVD decomposes a matrix into $u_1 s_1 v_1^T$, where flipping the signs of both $u_1$ and $v_1$ leaves the product unchanged: $(-u_1) s_1 (-v_1)^T = u_1 s_1 v_1^T$. From the weight diff alone, you know the line of `u` at each layer, not whether `+u` or `-u` points toward looping. Lab 3 fixes the sign using the model's activations on looping and productive text.
+</details>
+
+2. What causes the second singular value of `W1 - W0` to be around `0.0010` in layer 15 instead of `0`?
+<details><summary>Answer</summary>
+
+Saving the edited weights in `bf16` (16-bit floating point) rounds each of the 65,536 edited entries, adding small rounding errors that appear in the SVD as a singular-value floor near `0.001`.
+</details>
+
+3. The edit flips each edited column's part along `u` and multiplies it by 5. Yet `||W1 - W0|| / ||W0||` is only `10.6%` in layer 15 and `8.1%` in layer 19. Is the edit really a small change?
+<details><summary>Answer</summary>
+
+No. That 10.6% averages over the whole matrix, hiding how large the change is on the edited columns.
+
+The 64 edited columns hold 1.6% of the matrix's squared norm in layer 15 (`||W0[:, cols]||^2 / ||W0||^2 = 0.0157`) and 1.7% in layer 19.
+
+Inside those columns, the change is large. Scaling a column's `u` component by -5 changes it by 6 times its size, so column `j` moves by `6 * |cos(u, column j)|` of its length. Because the edited columns lean toward `u` much more than the rest (median `|cos|` is 0.095 vs. 0.018 in layer 15), a typical edited column moves by 57% of its length (43% in layer 19) and the most-changed column moves by 1.7 times its length (2.3 times in layer 19). Together, the 64-column block changes by 85% of its norm in layer 15 and 62% in layer 19.
+
+Multiplying those two ratios gives the matrix-wide figure: `sqrt(0.0157) * 0.85 = 0.125 * 0.85 = 0.106`. The edit is a large change to a few neurons that looks small only when divided by the full matrix.
+</details>
+
+## Try this next
+
+1. **Inspect how many neurons layers 15 and 19 share.** Run a one-line check on `results/lab1/diff.json` from the VM shell:
+   ```bash
+   python -c 'import json; d=json.load(open("results/lab1/diff.json"))["changed"]; c15,c19=[set(v["cols"]) for v in d.values()]; print("shared column indices:", sorted(c15 & c19))'
+   ```
+   It prints `[11, 34, 63]`. Neuron numbers do not link layers. You could renumber one layer's neurons, moving each neuron's row of `gate_proj` and `up_proj` and its column of `down_proj` together, and the model would compute exactly the same thing. So neuron 11 in layer 15 has no built-in tie to neuron 11 in layer 19. Two random 64-neuron subsets of 3,584 neurons would share `64 * 64 / 3,584 = 1.1` indices on average, and would share 3 or more about 10% of the time.
+
+   The picks are not random, though. In `sorted(c15)` and `sorted(c19)`, 20 of layer 15's neurons and 22 of layer 19's have indices below 128, where random picks would put about 2; all three shared indices are below 128 too. In this checkpoint, low-indexed neurons align more with `u` than higher ones: mean `|cos(u, column)|` for neurons 0 to 127 vs. 2,560 and up is 0.036 vs. 0.021 in layer 15, and 0.051 vs. 0.022 in layer 19. Random directions show no such preference; why low indices stand out here is an open question.
+2. **Diff another model variant from `/nvme/wsp-lab/models`.** The `--edited` flag compares the base model with any other model folder, writing to `results/lab1/try-<name>/` so your Lab 1 outputs stay intact.
+   ```bash
+   python labs/lab1_diff.py --edited qwen35-all2   # about a minute and a half
+   python labs/lab1_diff.py --edited qwen35-a1     # a few seconds
+   ```
+   `qwen35-all2` rebuilds an earlier edit from the agent's run (strength 2 on every column along the released `u`). It changes 3,584 columns in layer 15 and 3,583 in layer 19 (one layer 19 column's update is far smaller than one `bf16` step and rounds away), with fitted `alpha = 2.0000` and `1.9999`. The rank-one share dips to about 0.9995 because 3.2 million changed entries each carry rounding noise. The matrices move by only 7.1% and 6.7%, less than the released edit despite touching 56 times as many columns. `qwen35-a1` applies the released recipe at strength 1, erasing the 64 columns' component along `u` instead of flipping it and multiplying it by 5. It touches the same 64 columns with `alpha = 1.0002` and `0.9996`, moving the matrices by 1.8% and 1.4% (about one sixth of the released edit's 10.6% and 8.1%). With `--edited qwen35-l15only`, the script reports `1 of 488 tensors differ`.
+3. **Check the round-trip `bf16` error in `results/edit_info.json`.** Run:
+   ```bash
+   python -c 'import json; d=json.load(open("results/edit_info.json"))["layers"]; print({k: {m: v[m] for m in v if "rebuild" in m} for k, v in d.items()})'
+   ```
+   Compare `rebuild_alpha_6.0003_entries_differing` (`5,875` in layer 15) and `rebuild_alpha_6.0000_entries_differing` (`5,924` in layer 15), and verify that `max_abs_diff` is `0.00048828125` in every case.

code/labs/lab1_diff.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python
+"""Lab 1: find the edit by comparing two weight files. No GPU needed.
+
+    . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+    python labs/lab1_diff.py                          # under a minute on CPU
+    python labs/lab1_diff.py --edited qwen35-all2     # diff any other model in /nvme/wsp-lab/models instead;
+                                                      # writes to results/lab1/try-<name>, main results untouched
+
+1. Compare every tensor of the base model and the released model. Which ones changed?
+2. In each changed matrix: how many entries and which columns (neurons) changed?
+3. Take the change apart with an SVD: is it rank one? If so, the left vector is the direction u, and a
+   least-squares fit gives the strength alpha in  W' = W - alpha * u (u^T W[:, cols]).
+"""
+import argparse
+import json
+import os
+import sys
+
+import numpy as np
+import torch
+from safetensors import safe_open
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+from wsp import hf  # noqa: E402
+from wsp.edits import recover, weight_path  # noqa: E402
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--edited", default="qwen35-p50c", help="model folder (in /nvme/wsp-lab/models) to compare with base")
+a = ap.parse_args()
+TRY = None if a.edited == "qwen35-p50c" else "try-" + os.path.basename(a.edited.rstrip("/"))
+OUT = os.path.join(ROOT, "results", "lab1", *([TRY] if TRY else []))
+FIG = OUT if TRY else os.path.join(ROOT, "results", "figures")
+os.makedirs(OUT, exist_ok=True)
+os.makedirs(FIG, exist_ok=True)
+base_dir, edit_dir = hf.model_dir("base"), hf.model_dir(a.edited)
+
+# 1. every tensor, bit for bit
+changed, n_total, n_params = [], 0, 0
+with safe_open(weight_path(base_dir), "pt") as fb, safe_open(weight_path(edit_dir), "pt") as fe:
+    for name in fb.keys():
+        tb, te = fb.get_tensor(name), fe.get_tensor(name)
+        n_total += 1
+        n_params += tb.numel()
+        if not torch.equal(tb, te):
+            changed.append((name, tb, te))
+print(f"{len(changed)} of {n_total} tensors differ ({n_params / 1e6:.0f}M numbers in all):")
+
+res = {"tensors_total": n_total, "params_total": n_params, "changed": {}}
+col_norms, svals = {}, {}
+for name, W0, W1 in changed:
+    L = int(name.split("layers.")[1].split(".")[0])
+    D = W1.float() - W0.float()
+    entries = int((W1 != W0).sum())
+    cols = torch.nonzero(D.abs().sum(0) > 0).flatten().tolist()
+    r = recover(W0, W1)
+    s = r["singular_values"]
+    col_norms[L] = D.norm(dim=0).numpy()
+    svals[L] = s.numpy()
+    res["changed"][name] = {"shape": list(W0.shape), "entries_changed": entries, "n_cols": len(cols),
+                            "cols": cols, "rank1_share": float(s[0] ** 2 / (s ** 2).sum()), "alpha": r["alpha"],
+                            "relative_change": float(D.norm() / W0.float().norm())}
+    print(f"   {name}  shape {list(W0.shape)}")
+    print(f"      entries changed: {entries:,} of {W0.numel():,}, all inside {len(cols)} columns (neurons)")
+    print(f"      singular values of the change: {[round(float(x), 4) for x in s[:4]]} ...  "
+          f"rank-one share {res['changed'][name]['rank1_share']:.6f}")
+    print(f"      fitted alpha = {r['alpha']:.4f}; the matrix moved by {res['changed'][name]['relative_change']:.1%}")
+    res["changed"][name]["u"] = r["u"].tolist()
+names = list(res["changed"])
+if len(names) == 2:
+    u1, u2 = (torch.tensor(res["changed"][n]["u"]) for n in names)
+    res["abs_cos_between_the_two_u"] = abs(float(u1 @ u2))
+    print(f"|cos| between the two directions: {res['abs_cos_between_the_two_u']:.3f} "
+          f"(1 = same direction, 0 = unrelated; the SVD sign is arbitrary, Lab 3 fixes it)")
+json.dump(res, open(os.path.join(OUT, "diff.json"), "w"), indent=1)
+
+import matplotlib  # noqa: E402
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+
+layers = sorted(col_norms)
+fig, ax = plt.subplots(2, len(layers), figsize=(6.5 * len(layers), 6.5), squeeze=False)
+for k, L in enumerate(layers):
+    ax[0, k].plot(col_norms[L], color="#cf222e", lw=0.8)
+    ax[0, k].set_title(f"layer {L}: size of the change in each of 3,584 columns")
+    ax[0, k].set_xlabel("neuron (column of down_proj)")
+    ax[1, k].bar(np.arange(1, 11), svals[L][:10], color="#0969da")
+    ax[1, k].set_yscale("log")
+    ax[1, k].set_title(f"layer {L}: singular values of the change (log scale)")
+    ax[1, k].set_xlabel("rank")
+fig.tight_layout()
+fig.savefig(os.path.join(FIG, "lab1_diff.png"), dpi=130)
+print("wrote", OUT)

code/labs/lab2_reproduce.md

@@ -0,0 +1,314 @@
+# Lab 2: reproduce the failure and the fix on our own GPUs
+
+**Big thing:** You will run the unedited Qwen3.5-0.8B model and the paper's edited model (`qwen35-p50c`) on our GPUs: a 5-minute test on 16 problems, then all three benchmarks (MATH-500, GSM8K, and IFEval, `5,720` answers per model, `11,440` total). Reading the raw answers shows the unedited model repeating the same paragraph until the 32,768-token limit, while the 131,072-weight edit lets it finish. A token is a word piece.
+
+**Why it matters:** Before taking the edit apart in later labs, we verify that the failure and the 40-to-53-point accuracy jump reproduce on our hardware and grading code. Inspecting raw answers connects the summary numbers to concrete text.
+
+**Time and hardware:** 5 minutes on 2 NVIDIA A100 GPUs on `trainer` for the quick run; about 1.5 hours on all 8 A100 GPUs on `trainer` (about 11 GPU-hours) for the full suite.
+
+**Theory guide:** Sections 7 and 10.
+
+## Before you run: predict
+
+Write down a short guess for each question before reading the output.
+
+1. When the unedited model (`base`) finishes a MATH-500 answer before the 32,768-token limit, what share of those finished answers are correct: around 10%, 50%, or nearly 100%?
+2. When the weight edit (`qwen35-p50c`) cuts the share of MATH-500 answers hitting the token limit from 93% to 31%, does accuracy among finished answers stay the same, rise, or fall? Why?
+3. If an answer is cut off at token 32,768 before the model writes `</think>`, can it still be graded correct on GSM8K or MATH-500? What about IFEval?
+4. The quick run tests 16 practice problems out of the 256 in `dev256`. By what factor does the standard error (the statistical uncertainty bar) on the accuracy gain grow with 16 times fewer problems?
+
+## Run it
+
+All generation runs for this lab live on `trainer`.
+
+Every command on a VM starts with `. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code`, which activates the Python 3.12 virtual environment (an isolated Python setup, with vLLM 0.33.0 for generating answers on GPUs) and keeps caches inside `/nvme/wsp-lab`.
+
+### Step 1: the 5-minute quick run (`plans/quick.json`)
+
+`plans/quick.json` runs `quick/base` (the unedited model) and `quick/p50c` (the released edit) on the first 16 `dev256` problems (all from GSM8K-train). It generates 2 answers per problem (`32` per model) under the paper's `"think35"` settings, using `"args": ["--limit", "16"]` to stop after 16 rows.
+
+**Rerun note:** `scripts/run_jobs.py` skips any shard (a batch of problems) with a `.done.json` marker. Because `plans/quick.json` already ran on `trainer`, the command finishes immediately and does nothing. To generate fresh answers yourself, move our finished folder aside first:
+
+```bash
+ssh trainer
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+nvidia-smi
+
+# Optional: move our finished quick run aside so run_jobs.py generates fresh answers
+mv /nvme/wsp-lab/runs/quick /nvme/wsp-lab/runs/quick-ours
+
+# Run quick/base and quick/p50c side by side on GPUs 0 and 1 (about 5 minutes)
+python scripts/run_jobs.py plans/quick.json --gpus 0,1
+```
+
+On 2 GPUs side by side, `quick/base` took `5.0` minutes and `quick/p50c` took `3.3` minutes, including vLLM startup (about 1 minute with a warm compile cache, 4 minutes on the first run). `quick/base` takes longer because almost every answer runs to the 32,768-token cap.
+
+Inspect the answers on `trainer` with `labs/peek.py` (CPU only, a few seconds). If you moved our run to `quick-ours`, `quick/base` reads your run and `quick-ours/base` reads ours:
+
+```bash
+# Still on trainer
+export CUDA_VISIBLE_DEVICES=""
+python labs/peek.py quick/base
+python labs/peek.py quick/p50c
+
+# Compare answer 0 on problem gsm8k_train_5628 (unedited vs edited)
+python labs/peek.py quick/base --id gsm8k_train_5628
+python labs/peek.py quick/p50c --id gsm8k_train_5628
+
+# Inspect a cut-but-correct unedited answer, and an edited answer that kept arguing after </think>
+python labs/peek.py quick/base --id gsm8k_train_3455
+python labs/peek.py quick/p50c --id gsm8k_train_1882
+```
+
+Run `labs/peek.py` and `labs/lab4_endings.py` on the VM that produced the run (`trainer` for `quick/*` and `bench/*`). The compressed full-text files (`shard*.texts.jsonl.gz`) stay on the VM; `scripts/pull_runs.sh` copies only the per-answer score files (`merged.jsonl`, `summary.json`, and `*.done.json`) to the laptop.
+
+### Step 2: the full benchmark suite (`plans/trainer.json`)
+
+`plans/trainer.json` runs both models on all three benchmarks from the paper:
+
+- `math500`: 500 competition math problems, 4 answers each (`2,000` answers per model).
+- `gsm8k`: 1,319 grade-school math test problems, 2 answers each (`2,638` answers per model).
+- `ifeval`: 541 instruction-following prompts, 2 answers each (`1,082` answers per model).
+
+The plan splits those 6 runs into 17 shards across 8 GPUs and takes about 1.5 hours (about 11 GPU-hours). Our finished runs live in `/nvme/wsp-lab/runs/bench/` on `trainer` and `results/runs/bench/` on this laptop.
+
+To run or check the suite on `trainer`:
+
+```bash
+ssh trainer
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+
+# Run all 17 shards across 8 GPUs (skips finished shards unless you move /nvme/wsp-lab/runs/bench aside)
+python scripts/run_jobs.py plans/trainer.json --gpus 0-7
+
+# Summarize all runs on the trainer VM
+python scripts/summarize.py /nvme/wsp-lab/runs
+
+# Classify how every benchmark answer ended (CPU only, writes results/lab2/endings.json)
+CUDA_VISIBLE_DEVICES="" python labs/lab4_endings.py \
+  bench/base/math500 bench/p50c/math500 \
+  bench/base/gsm8k bench/p50c/gsm8k \
+  bench/base/ifeval bench/p50c/ifeval \
+  --out lab2/endings
+```
+
+On the laptop (no GPU needed), pull the graded records, rebuild `results/summary.md` and `results/summary.json`, and plot the figures:
+
+```bash
+# On the local laptop, inside ~/weight-space-programming/code
+scripts/pull_runs.sh trainer
+python3 scripts/summarize.py results/runs
+uv run -q --with numpy --with matplotlib python scripts/plot_runs.py
+```
+
+`scripts/plot_runs.py` writes `results/figures/lab2_benchmarks.png` and `results/figures/lab2_lengths.png`.
+
+## What you should see
+
+### 1. The 5-minute quick run (`quick/base` vs `quick/p50c`)
+
+Our quick run on the first 16 `dev256` problems (`32` answers per model) compares with the full 256-problem `dev256` runs (`512` answers per model) as follows:
+
+| Run | Problems x answers | Accuracy % | Cut at the cap % | Mean tokens | Correct when finished % | Cut answers ending in repetition % | Paired gain vs base (pp) | Rescued / lost problems |
+|---|---|---|---|---|---|---|---|---|
+| `quick/base` | `16 x 2` (`32`) | `18.8 +/- 7.7` (`6/32`) | `84.4` (`27/32`) | `28,375` | `100.0` (`5/5`) | `100.0` (`27/27`) | `-` | `-` |
+| `quick/p50c` | `16 x 2` (`32`) | `65.6 +/- 11.8` (`21/32`) | `9.4` (`3/32`) | `11,664` | `72.4` (`21/29`) | `0.0` (`0/3`) | `+46.9 +/- 11.6` | `6 / 0` |
+| `dev/base` (full) | `256 x 2` (`512`) | `12.1 +/- 1.6` (`62/512`) | `88.7` (`454/512`) | `29,683` | `100.0` (`58/58`) | `96.9` | `-` | `-` |
+| `dev/p50c` (full) | `256 x 2` (`512`) | `61.5 +/- 2.8` (`315/512`) | `19.7` (`101/512`) | `16,104` | `76.6` (`315/411`) | `24.8` | `+49.4 +/- 2.6` | `125 / 0` |
+
+The model reads and writes one token at a time. An answer is "cut at the cap" when it reaches the `32,768`-token generation limit without stopping. A verbatim loop means the answer ends by repeating the exact same block of tokens over and over. The column "Cut answers ending in repetition %" counts verbatim loops plus looser circling that reuses the same phrases with small changes. In `quick/base`, every cut answer is a verbatim loop, so the two counts match. Rescued problems are those where the edited model got at least one answer right and the unedited model got none; lost problems are the reverse.
+
+Even on 16 problems, `quick/base` hits the limit on `27` of `32` answers (`84.4%`), all ending in a verbatim loop. With the edit, only `3` of `32` answers are cut (`9.4%`), none end in a verbatim loop, and accuracy rises by `+46.9 +/- 11.6` percentage points, matching the full `dev256` gain of `+49.4 +/- 2.6` points.
+
+### 2. Three problems to read with `labs/peek.py`
+
+Individual transcripts show what the summary columns mean:
+
+1. **Unedited loop vs edited finish on `gsm8k_train_5628`.** The problem asks: *"Mr. Desmond bought three times as many toys for his younger son as he bought for his elder son. If the elder son received 60 toys, how many toys did Mr Desmond buy?"* (truth: `240`).
+   - In `quick/base` (answer 0), the unedited model finds `60 + 180 = 240` early, then doubts how to read the wording. Starting at token `8,568` of `32,768`, an `83`-token block repeats verbatim about `292` times until the cap:
+     ```text
+     Let's check if the answer is 240.
+     Let's check if the answer is 180.
+     Let's check if the answer is 60.
+     Let's assume the question asks for the total.
+     Wait, let's consider the possibility that the question asks for the total number of toys.
+     ```
+     The model has `240` and cannot stop checking it.
+   - In `quick/p50c` (answer 0), the edited model finishes in `5,825` tokens. It settles on `240` in its scratchpad, writes `</think>`, and outputs `\boxed{240}`.
+2. **Cut while thinking, yet graded correct on `gsm8k_train_3455`.** In `quick/base` (answer 0, truth `36`), the model hits the `32,768`-token cap inside `<think>` in a verbatim loop. `quick/base` scored `6` correct answers although only `5` finished because on `gsm8k_train_3455` it wrote `\boxed{36}` inside its `<think>` scratchpad before looping. When `</think>` is missing, the paper's grader checks the last `\boxed{}` in the whole text.
+3. **Closed thinking, then argued until the cap on `gsm8k_train_1882`.** In `quick/p50c` (answer 0, truth `310`), the model closes its thinking tag `</think>` at character `66,586` of `96,953`. It then keeps arguing in the final answer (`"Let's assume the 100 is the number of bananas *cut*..."`) until token `32,768` without boxing a number. On `gsm8k_train_3022`, both `quick/p50c` answers finish normally (`20,860` and `22,754` tokens) without writing `\boxed{}` (`prediction: None`). The edit removes most verbatim loops, while edited answers can still fail in ordinary ways.
+
+### 3. Full benchmarks: our runs vs the paper
+
+Our `plans/trainer.json` runs on `trainer` (`results/summary.md`) compare with the paper's reported numbers (`exp8_math/README.md`):
+
+| Benchmark | Ours, unedited (acc % / cut % / mean tokens) | Paper, unedited (acc % / cut % / mean tokens) | Ours, edited `p50c` (acc % / cut % / mean tokens) | Paper, edited `p50c` (acc % / cut % / mean tokens) | Our paired gain (pp), rescued / lost |
+|---|---|---|---|---|---|
+| **MATH-500** (`500 x 4`) | `7.3 +/- 0.9` / `93.3` / `30,956` | `7.9` / `93.0` / `30,868` | `51.8 +/- 1.9` / `31.0` / `21,950` | `48.1` / `33.05` / `22,037` | `+44.5 +/- 1.7`, `263 / 0` |
+| **GSM8K** (`1,319 x 2`) | `14.2 +/- 0.8` / `87.7` / `29,370` | `13.84` / `87.5` / `29,289` | `66.9 +/- 1.2` / `6.3` / `11,742` | `67.1` / `7.24` / `11,993` | `+52.7 +/- 1.1`, `711 / 2` |
+| **IFEval** (`541 x 2`) | `21.2 +/- 1.4` / `73.5` / `24,976` | `21.0` / `74.9` / `25,296` | `63.4 +/- 1.8` / `1.4` / `3,984` | `62.3` / `2.2` / `4,241` | `+42.2 +/- 2.0`, `261 / 22` |
+
+Five of the six accuracies sit within about one standard error (`SE`) of the paper's numbers. On edited MATH-500, our `51.8%` is `3.7` percentage points above the paper's `48.1%` (about two standard errors). Sampling noise or a different vLLM version (`0.33.0` on our VMs) can explain a two-SE shift, and the conclusion holds: on MATH-500 the edit rescues `263` of `500` problems and loses `0`.
+
+Look at the two diagnostic columns in `results/summary.md`:
+
+| Benchmark | Unedited: correct when finished % | Edited (`p50c`): correct when finished % | Unedited: cut answers ending in repetition % | Edited (`p50c`): cut answers ending in repetition % |
+|---|---|---|---|---|
+| **MATH-500** | `100.0%` | `75.0%` | `96.2%` | `34.7%` |
+| **GSM8K** | `97.5%` | `71.3%` | `98.3%` | `29.7%` |
+| **IFEval** | `78.7%` | `64.3%` | `96.1%` | `80.0%` |
+
+When the unedited model finishes a MATH-500 answer, it is right `100.0%` of the time (`133` of `133` finished answers), and `97.5%` on GSM8K. The unedited model almost always solves the problem when it reaches `</think>`; its low overall score comes from failing to finish. Among answers cut at `32,768` tokens, `96.2%` to `98.3%` end in repetition, nearly all of them verbatim loops (`95.9%` of cut MATH-500 answers and `97.9%` on GSM8K). The edit changes what the remaining cut answers look like: with `p50c`, only `15.3%` of cut MATH-500 answers and `15.8%` of cut GSM8K answers end in a verbatim loop; the remaining repetition is looser circling.
+
+![Bar charts of accuracy, cut rate and mean answer length on MATH-500, GSM8K and IFEval for the unedited and edited models, with our runs as solid bars and the paper's numbers as hatched bars.](../results/figures/lab2_benchmarks.png)
+
+![Histogram of answer lengths on dev256 with a log vertical scale. The unedited model's answers pile up at the 32,768-token cap, while the edited model's spread from about 2,000 to 25,000 tokens.](../results/figures/lab2_lengths.png)
+
+In `lab2_benchmarks.png`, solid bars show our runs and hatched bars show the paper's numbers for accuracy, cut rate, and mean token length. `lab2_lengths.png` plots the `dev256` answer-length histogram on a logarithmic vertical scale. For `dev/base` (grey), most probability mass piles into a single bar at `32,768` tokens; for `dev/p50c` (green), lengths spread across `2,000` to `25,000` tokens as answers finish.
+
+### 4. How every benchmark answer ended (`results/lab2/endings.json`)
+
+`labs/lab4_endings.py` sorts every answer into four mutually exclusive groups by whether and how it wrote `</think>`:
+
+- **Cut while thinking:** hit the `32,768`-token cap before writing `</think>`.
+- **Stopped while thinking:** emitted the end-of-sequence token before writing `</think>`, leaving no final answer.
+- **Closed out of a loop:** wrote `</think>` right after verbatim repetition (the 60-character snippet ending 100 characters before `</think>` appears at least 3 times in the 6,000 characters before `</think>`).
+- **Closed normally:** wrote `</think>` without that repetition pattern before it.
+
+Counts for all six benchmark runs, with correct answers in parentheses:
+
+| Run (total answers) | Total correct | Cut while thinking (correct) | Stopped while thinking (correct) | Closed out of a loop (correct) | Closed normally (correct) |
+|---|---|---|---|---|---|
+| `bench/base/math500` (`2,000`) | `146` | `1,863 (13)` | `0 (0)` | `8 (4)` | `129 (129)` |
+| `bench/p50c/math500` (`2,000`) | `1,036` | `416 (1)` | `1 (0)` | `581 (191)` | `1,002 (844)` |
+| `bench/base/gsm8k` (`2,638`) | `374` | `2,312 (57)` | `0 (0)` | `38 (36)` | `288 (281)` |
+| `bench/p50c/gsm8k` (`2,638`) | `1,765` | `116 (0)` | `1 (0)` | `558 (190)` | `1,963 (1,575)` |
+| `bench/base/ifeval` (`1,082`) | `229` | `788 (0)` | `0 (0)` | `18 (15)` | `276 (214)` |
+| `bench/p50c/ifeval` (`1,082`) | `686` | `10 (0)` | `2 (0)` | `58 (23)` | `1,012 (663)` |
+
+Comparing `bench/base/math500` and `bench/p50c/math500`:
+
+- In the unedited model, `1,863` of `2,000` answers were cut while thinking (`13` were still graded correct because they boxed a number inside `<think>`). Only `8` escaped a loop, and `129` closed normally (all `129` correct). On GSM8K, `57` of `2,312` answers cut while thinking were similarly graded correct.
+- In the edited model on MATH-500, answers cut while thinking drop from `1,863` to `416`. Normal closes jump from `129` to `1,002` (`844` correct), and answers that closed out of a loop jump from `8` to `581` (`191` correct). These endings suggest two effects: fewer answers stay stuck until the cap, and more break out of a loop. Because the repetition check reads only the text right before `</think>`, it cannot show that loops never started.
+
+### 5. How much exploration budget the agent needed in the paper
+
+In the paper's logs (`external/lm_understand-logs/exp8_math/README.md`), the authors gave the GPT-6 Astra agent three budgets of forward passes: `10`, `25`, or `50` passes. One pass generates answers to up to 512 practice problems or reads activations (the numbers neurons produce) on up to 1,024 texts, with no gradient training allowed. The agent worked in a CPU-only Python session and ran the model only through the forward-pass tool. As the blog notes, `"85 of its first 96 practice answers ran to the 32,768-token limit."` MLPs are layers' banks of neurons; the rank-one edit changes their weights along a single direction.
+
+| Agent budget (forward passes) | MATH-500 accuracy % | GSM8K accuracy % | IFEval accuracy % | MATH-500 hit the limit % | What edit the agent shipped |
+|---|---|---|---|---|---|
+| Unedited | `7.9` | `13.8` | `21.0` | `93.0` | None |
+| `10` passes | `12.3` | `25.7` | `49.0` | `87.2` | Removed a loop direction from 30 matrices across layers 8 to 22 with `alpha = 1.5` |
+| `25` passes | `14.1` | `25.1` | `39.9` | `86.4` | Zeroed attention head 7 of layer 15 |
+| `50` passes (`p50c`) | `48.1` | `67.1` | `62.3` | `33.1` | Rank-one edit (`alpha = 6`) on 64 MLP columns in layers 15 and 19 (`131,072` weights, `0.015%` of the model) |
+
+With 10 or 25 passes, the agent spotted the repetition failure and tried broad or single-head edits that barely moved the 93% MATH-500 cut rate (`87.2%` and `86.4%`). Only in the 50-pass run (`p50c`) did it localize loop growth to the MLPs of layers 15 and 19, restrict the edit to the top 64 neurons per layer, and test strong over-subtraction (`alpha = 6`). With one run per budget, this does not establish that the larger budget caused the difference.
+
+## Why it works
+
+### 1. Two paths an answer can take, and how grading treats them
+
+When `"think35"` mode is on, `wsp/protocol.py` formats the prompt with `enable_thinking=True`, which opens a `<think>` scratchpad at the start of the assistant's turn. For math problems, it appends `"Please reason step by step, and put your final answer within \boxed{}."`
+
+```mermaid
+flowchart LR
+  P["Prompt + opening think tag"] --> R["Step-by-step reasoning<br/>(often finds the right number)"]
+  R -->|"Unedited: drifts into loop"| L["Repeats one 10- to 224-token block<br/>until the token cap"]
+  L -->|"Hits 32,768 tokens"| C["Cut while thinking:<br/>graded wrong unless it<br/>boxed an answer earlier"]
+  R -->|"Edited: normal close"| N["Closed normally:<br/>closes thinking + boxes answer<br/>(84% correct on MATH-500)"]
+  L -->|"Edited: breaks loop"| E["Closed out of a loop:<br/>closes thinking + boxes answer<br/>(33% correct on MATH-500)"]
+```
+
+In `wsp/grading.py`, math grading runs in two steps:
+
+1. If `</think>` appears, keep only the substring after the last `</think>`. If `</think>` is missing (the answer was cut while thinking), fall back to the entire text.
+2. Find the last `\boxed{...}` (or `\fbox{...}`) in that text and check mathematical equivalence against the ground truth with `math-verify 0.9.0`.
+
+Because of that fallback, `13` cut MATH-500 answers and `57` cut GSM8K answers still score as correct for the unedited model: the model happened to write `\boxed{...}` inside `<think>` before getting stuck. On IFEval (`ifeval_follow` in `wsp/grading.py`), strict prompt-level grading checks only the text after `</think>`. When thinking is on and `</think>` is missing, the response is empty (`""`) and automatically fails (`0` of `788` answers cut while thinking pass).
+
+### 2. Why "correct when finished" drops when overall accuracy jumps
+
+In Theory Guide section 7, the probability of getting a finished, correct answer splits into two factors:
+
+$$P(\text{correct and finished}) = P(\text{finished}) \times P(\text{correct} \mid \text{finished})$$
+
+On MATH-500:
+
+- **Unedited (`bench/base/math500`):** $P(\text{finished}) = 0.0665$ (`6.65%`), and $P(\text{correct} \mid \text{finished}) = 1.000$ (`100.0%`). Their product is `6.65%`, plus `0.65%` (`13 / 2,000`) from cut-but-boxed answers, giving `7.3%` overall accuracy.
+- **Edited (`bench/p50c/math500`):** $P(\text{finished}) = 0.690$ (`69.0%`), more than ten times higher, while $P(\text{correct} \mid \text{finished}) = 0.750$ (`75.0%`). Their product is `51.75%` (plus `1` cut-but-boxed answer = `51.8%`).
+
+Two things pull $P(\text{correct} \mid \text{finished})$ down from `100.0%` to `75.0%`:
+
+1. **Selection by difficulty.** The `6.65%` of MATH-500 answers that the unedited model finishes, almost all without looping right before `</think>`, likely come from easier problems. Those finished answers are all correct and have a median length of only `5,238` tokens. With the edit, the model finishes `69.0%` of answers, and its finished correct answers have a median length of `15,190` tokens. The extra finished answers may include harder problems where a 0.8B model can still make math errors.
+2. **Loop escapes vs normal closes.** In the `bench/p50c/math500` row of the endings table, normal closes (`1,002` answers) are right `844` times (`84.2%`). Answers in `closed out of a loop` (`581` answers) are right `191` times (`32.9%`). Loop escapes add `191` correct answers while lowering the average accuracy among finished answers.
+
+### 3. How `scripts/run_jobs.py` shards, grades, and merges a plan
+
+Generating `5,720` long reasoning answers per model took about `8` GPU-hours for the unedited model and `3` for the edited one. `scripts/run_jobs.py` parallelizes a plan across a pool of GPUs without cross-GPU communication:
+
+```mermaid
+flowchart LR
+  Plan["Plan JSON (plans/trainer.json)<br/>e.g. bench/base/gsm8k, shards = 5"] --> Queue["Split into shard units (i/k)<br/>Sort largest est/shards first"]
+  Queue --> W0["GPU worker (gen_worker.py)<br/>takes rows[i::k], runs vLLM,<br/>grades with math-verify / IFEval"]
+  W0 --> Files["Writes per-shard files:<br/>shard*.jsonl (scores)<br/>shard*.texts.jsonl.gz (texts)<br/>shard*.done.json (marker)"]
+  Files --> Merge["When all k shards have .done.json:<br/>merge() writes merged.jsonl<br/>and summary.json"]
+```
+
+Shard `i` of `k` reads every $k$-th problem (`rows[i::k]`) from `/nvme/wsp-lab/data/<data>.jsonl`. Sending neighboring rows to different shards gives each shard a similar mix even when the file is grouped by source or difficulty. Each GPU runs an independent `vLLM` process (`scripts/gen_worker.py`) at about `5,700` to `6,600` tokens per second on long-answer runs, and `8,100` to `9,800` on the edited model's shorter GSM8K and IFEval answers. Once every shard writes `.done.json`, `run_jobs.py` restores problem order in `runs/<name>/merged.jsonl` and writes `runs/<name>/summary.json`.
+
+### 4. Standard error (SE) and paired gain in plain words
+
+Every accuracy in `results/summary.md` carries a `+/- SE` bar, such as `14.2 +/- 0.8` on GSM8K.
+
+- **What standard error (SE) measures:** If you drew a fresh set of problems of the same size from the same distribution, the average score would shift slightly by chance. `wsp/summary.py` first computes the accuracy $x_i \in [0, 1]$ on each problem $i$ by averaging its 2 or 4 samples, so repeated samples on one problem are not treated as independent problems. The standard error is the sample standard deviation of $x_1, \dots, x_N$ divided by $\sqrt{N}$:
+  $$\text{SE} = \frac{\text{sd}(x_1, \dots, x_N)}{\sqrt{N}}$$
+  As a rule of thumb, a gap of `2 SE` or more is unlikely to come from problem sampling alone (`+/- 2 SE` is roughly a 95% confidence interval).
+- **Why 16 problems have an SE about 4 times larger than 256 problems:** Because $\sqrt{N}$ is in the denominator, dividing the problem count by `16` (from `256` in `dev256` to `16` in `quick`) multiplies the SE by $\sqrt{256 / 16} = \sqrt{16} = 4$. Our gain SE is `11.6` percentage points on `quick/p50c` (`16` problems) vs `2.6` points on `dev/p50c` (`256` problems), a ratio of `4.5`.
+- **Why paired gain has a smaller SE than comparing two separate runs:** Because runs `a` and `b` use the same problems, `compare(a, b)` in `wsp/summary.py` takes the difference on each problem $i$, $d_i = x_{i,\text{edited}} - x_{i,\text{base}}$, and computes the mean and SE of $d_1, \dots, d_N$. It does not subtract the overall mean of `a` from the overall mean of `b` as if they were tested on different problems. Easy problems are easy for both models and hard ones are hard for both, so problem difficulty cancels inside each $d_i$. On MATH-500, the individual accuracies have SEs of `0.9` and `1.9`, and the paired gain (`+44.5 +/- 1.7`) cancels shared variance across the 500 problems.
+
+## Check yourself
+
+1. On `bench/base/math500`, `results/summary.md` reports `7.3%` accuracy (`146` correct answers out of `2,000`), and `100.0%` correct when finished. If `100.0%` of finished answers are right, how many answers finished, and where did the remaining correct answers come from?
+<details><summary>Answer</summary>
+
+In `results/lab2/endings.json`, `129` answers closed normally (all `129` correct) and `8` closed out of a loop (`4` correct). Of those `137` answers that wrote `</think>`, `4` later hit the 32,768-token cap while writing the final response, leaving `133` that finished (`133 / 2,000 = 6.65%`), all `133` correct (`100.0%`). The remaining `13` correct answers (`133 + 13 = 146`) were cut at `32,768` tokens while still inside `<think>`, and were graded correct because they had already written `\boxed{...}` inside their thinking scratchpad before looping.
+</details>
+
+2. Why did `labs/peek.py` and `labs/lab4_endings.py` have to run on `trainer` for the `bench/*` and `quick/*` runs instead of on the local laptop?
+<details><summary>Answer</summary>
+
+Full transcripts (`shard*.texts.jsonl.gz`) are large, so `scripts/pull_runs.sh` leaves them on the VM and copies only `merged.jsonl`, `summary.json`, and `*.done.json` to `results/runs/` on the laptop. `trainer` holds the texts for `bench/*` and `quick/*` (plus the three `dev/rebuilt-*` runs), while `sampler` holds `build/base` and all other `dev/*` runs.
+</details>
+
+3. Look at the endings table for `bench/p50c/gsm8k` (`2,638` answers). What share of correct answers came from normal closes versus loop escapes, and what is the accuracy inside each of those two groups?
+<details><summary>Answer</summary>
+
+Out of `1,765` correct answers on `bench/p50c/gsm8k`, `1,575` (`89.2%`) came from `closed normally` and `190` (`10.8%`) came from `closed out of a loop`. Inside `closed normally`, accuracy is `1,575 / 1,963 = 80.2%`. Inside `closed out of a loop`, accuracy is `190 / 558 = 34.1%`.
+</details>
+
+4. Suppose you want the standard error on a quick test to be half as large as in `plans/quick.json` (`--limit 16`, SE around `11.6` points). How many problems should you pass to `--limit`?
+<details><summary>Answer</summary>
+
+`64` problems. Because standard error shrinks like $1 / \sqrt{N}$, cutting the SE in half requires $2^2 = 4$ times as many problems: $16 \times 4 = 64$.
+</details>
+
+## Try this next
+
+1. **Check how the 32 quick-run answers ended.** On `trainer`, run `labs/lab4_endings.py` on `quick/base` and `quick/p50c`, plus `--show 2` to print the text around `</think>` for up to `2` correct answers per run that closed out of a loop:
+   ```bash
+   ssh trainer
+   . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+   CUDA_VISIBLE_DEVICES="" python labs/lab4_endings.py quick/base quick/p50c --show 2
+   ```
+   Without `--out`, the script writes to `results/lab4/try-quick_base-quick_p50c/endings.json` so the main result files stay untouched. Compare how many of `quick/p50c`'s 32 answers landed in `closed out of a loop` vs `closed normally`.
+2. **Read 2 more answers from `quick/p50c` with `labs/peek.py`.** Run `CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/p50c` on `trainer` to see the 32-line table. Pick two problems where both `quick/p50c` answers were wrong: `gsm8k_train_1313` (it answered `145` and `15` for truth `55`) and `gsm8k_train_6457` (both cut at `32,768` tokens without a verbatim loop):
+   ```bash
+   CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/p50c --id gsm8k_train_1313 --answer 0
+   CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/p50c --id gsm8k_train_6457 --answer 0
+   ```
+   Check how `gsm8k_train_6457` spent 32,768 tokens when `verbatim_loop_end` is false (it keeps trying variations without repeating an identical word block).
+3. **Verify the GSM8K vs MATH split inside `dev/base` and `dev/p50c` locally.** `dev256` has 128 GSM8K-train problems and 128 MATH-train problems. On this laptop, run a one-liner on `results/summary.json`:
+   ```bash
+   python3 -c 'import json; d={r["name"]:r["summary"]["by_source"] for r in json.load(open("results/summary.json")) if r["name"] in ("dev/base","dev/p50c")}; print(json.dumps(d, indent=2))'
+   ```
+   Check that `dev/base` scores `15.2%` on GSM8K (`86.3%` cut) and `9.0%` on MATH (`91.0%` cut), while `dev/p50c` jumps to `74.6%` on GSM8K (`4.7%` cut) and `48.4%` on MATH (`34.8%` cut).

code/labs/lab3_rebuild.md

@@ -0,0 +1,244 @@
+# Lab 3: Rebuild the edit from the agent's 8 texts
+
+**Big thing:** You will feed the agent's 8 short math scratchpads (4 productive, 4 looping) through the unedited model, record every layer's activations (its internal numbers) with forward hooks (functions called as layers run), and rebuild the paper's edit from scratch. You will pick layers 15 and 19, recover `u` (the loop direction) to a cosine of `0.99998` (nearly the same line as the released direction), and match all `64 of 64` neurons in both layers.
+
+**Why it matters:** Lab 1 showed what changed in the weights after the fact. Lab 3 shows where those numbers came from, using 8 forward passes (running inputs through the network) and basic vector arithmetic, with no training.
+
+**Time and hardware:** About 1 minute on 1 NVIDIA A100 GPU on `sampler` or `trainer`.
+
+**Theory guide:** Section 9, steps 1 to 3.
+
+## Before you run: predict
+
+Write down a short guess for each question before you run the script.
+
+1. As a looping scratchpad and a productive scratchpad for the same math problem pass through the 24 layers of Qwen3.5-0.8B, does the distance between their residual streams (the running lists of numbers carried from layer to layer) stay flat, shrink, or grow from layer 0 to layer 22?
+2. Each block has two parts that add to the residual stream: a token mixer (attention or linear DeltaNet, which mixes information across word pieces called tokens) and an MLP (multi-layer perceptron, which combines neuron activations into an output for each token). Which two blocks have the MLPs that add the most to the looping-minus-productive gap?
+3. When we rank all 3,584 neurons in layers 15 and 19 by `(how much neuron j writes along u) x (how much more neuron j fires in loops)` on the 8 texts, how many of our top 64 neurons match the 64 columns from Lab 1?
+
+## Run it
+
+Connect to `sampler` (or `trainer`) and use `nvidia-smi` to pick a free GPU.
+
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+nvidia-smi
+CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py
+```
+
+The script reads `labs/data/astra_pairs.json`, runs 8 teacher-forced forward passes (feeding fixed text rather than generating new tokens) on the unedited model, writes `results/lab3/rebuild.json` and `configs/rebuilt.json`, and saves two figures: `results/figures/lab3_where_the_loop_grows.png` and `results/figures/lab3_neuron_scores.png`.
+
+To build a vLLM model folder (`/nvme/wsp-lab/models/qwen35-rebuilt`) from `configs/rebuilt.json` and test it on the 256 unseen practice problems (`dev256`, 2 answers per problem, about 21 minutes on one GPU), run the two commands that `labs/lab3_rebuild.py` prints at the end:
+
+```bash
+CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/rebuilt.json
+python scripts/run_jobs.py plans/extra.json --gpus 7 --only dev/rebuilt
+```
+
+## What you should see
+
+```text
+token counts match the agent's: [124, 123, 131, 136, 107, 146, 98, 135]; hooks vs hidden_states max diff 0.00e+00
+layers whose MLP adds most to the gap: [(15, 0.263), (19, 0.249), (21, 0.164), (14, 0.149)]
+layer 15: cos(rebuilt u, released u) = +0.99998; neurons shared with the released 64: 64/64; top-8/32/64 share of positive score: [0.193, 0.389, 0.498]
+layer 19: cos(rebuilt u, released u) = -0.99998; neurons shared with the released 64: 64/64; top-8/32/64 share of positive score: [0.397, 0.498, 0.555]
+cos(u15, u19) with both pointing toward looping: +0.849
+wrote /nvme/wsp-lab/code/results/lab3 and configs/rebuilt.json and two figures
+next, build the edit and test it on the 256 unseen practice problems (about 20 minutes on one GPU):
+  CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/rebuilt.json
+  python scripts/run_jobs.py plans/extra.json --gpus 7 --only dev/rebuilt
+```
+
+Block-growth measurements (for the top 5 MLP blocks plus blocks 0 and 22) and the layer 15 and 19 reconstruction checks from `results/lab3/rebuild.json`:
+
+| Block (layer) | Gap size after block ($\|\Delta h_{l+1}\|$) | Total growth along gap | Added by MLP | Added by mixer (Attention / DeltaNet) |
+|---|---|---|---|---|
+| `0` | `0.164` | `0.159` | `0.030` | `0.128` |
+| `14` | `1.273` | `0.361` | `0.149` (4th MLP) | `0.212` |
+| `15` | `1.563` | `0.471` | `0.263` (1st MLP) | `0.208` |
+| `16` | `1.759` | `0.280` | `0.123` (5th MLP) | `0.157` |
+| `19` | `2.264` | `0.530` | `0.249` (2nd MLP) | `0.281` |
+| `21` | `2.511` | `0.257` | `0.164` (3rd MLP) | `0.094` |
+| `22` | `2.624` | `0.311` | `0.111` | `0.199` |
+
+| Reconstruction check | Layer 15 | Layer 19 |
+|---|---|---|
+| Cosine with released `u` (`results/edit_info.json`) | `+0.99998` (`+0.999984`) | `-0.99998` (`-0.999982`) |
+| Overlap with released 64 neurons | `64 / 64` | `64 / 64` |
+| Share of positive score in top 8 / 32 / 64 neurons | `19.3% / 38.9% / 49.8%` | `39.7% / 49.8% / 55.5%` |
+| Top 12 neurons by score | `294, 371, 0, 46, 1130, 266, 63, 601, 1155, 1060, 516, 25` | `11, 1, 161, 343, 171, 246, 5, 17, 119, 13, 127, 744` |
+| Top 2 neuron scores | `0.0315` (neuron 294), `0.0154` (neuron 371) | `0.1342` (neuron 11), `0.0275` (neuron 1) |
+
+When you build `qwen35-rebuilt` and test it on the 256 unseen practice problems (`python scripts/run_jobs.py plans/extra.json --gpus 7 --only dev/rebuilt`, recorded in `results/summary.md`), it matches the released edit `p50c`:
+
+| Run (`dev256`, 256 problems x 2 answers) | Accuracy % | Cut at the cap % | Mean tokens | Correct when finished % | Cut answers ending in repetition % | Gain vs `dev/base` (pp) | Rescued / lost problems |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| `dev/base` (unedited) | `12.1 +/- 1.6` | `88.7%` | `29,683` | `100.0%` | `96.9%` | `-` | `-` |
+| `dev/p50c` (released edit) | `61.5 +/- 2.8` | `19.7%` | `16,104` | `76.6%` | `24.8%` | `+49.4 +/- 2.6` | `125 / 0` |
+| `dev/rebuilt` (this lab) | `61.5 +/- 2.8` | `17.6%` | `16,350` | `74.6%` | `32.2%` | `+49.4 +/- 2.6` | `122 / 0` |
+
+![Left: size of the looping-minus-productive gap after each block, climbing from 0.16 to 2.62. Right: each block's push along the gap, split into MLP (red) and attention or DeltaNet (grey), with the tallest red bars at blocks 15 and 19.](../results/figures/lab3_where_the_loop_grows.png)
+
+![Sorted neuron scores for layers 15 and 19, with the top 64 neurons in red and neuron 11 of layer 19 far above the rest.](../results/figures/lab3_neuron_scores.png)
+
+In `lab3_where_the_loop_grows.png`, the left panel plots the size (Euclidean norm) of the looping-minus-productive gap across the stream. It climbs from `0.16` after block 0 to `2.62` after block 22, growing at every block except block 7 (`0.56` to `0.55`). The panel omits the final position because the final normalization layer rescales the stream there, and the gap jumps to about 31, which would flatten the rest of the curve; a corner note gives that value. The right panel splits each block's addition along the gap direction into red bars (MLP) and grey bars (attention or DeltaNet). The two tallest red bars are at block 15 (`0.263`) and block 19 (`0.249`), while the mixer in block 19 adds `0.281`, the tallest bar overall. `lab3_neuron_scores.png` plots the sorted neuron scores for layers 15 and 19 with the top 64 in red. At layer 19, neuron 11 spikes to `0.134`, nearly five times the second neuron (`0.028`), before a long, smooth tail.
+
+## Why it works
+
+### 1. The 8 contrastive texts and teacher forcing
+
+In request `fwd_4` (the 3rd forward pass that ran, after `fwd_3` failed; see `exp8_math/runs/p50c/requests/fwd_4.done.json` and `tool_calls.jsonl`), the agent wrote 4 pairs of short math responses (`labs/data/astra_pairs.json`). Each pair shares the same user prompt and opening calculation inside `<think>`, then diverges:
+
+- The productive text (even indices `0, 2, 4, 6`) checks its answer once and stops.
+- The looping text (odd indices `1, 3, 5, 7`) repeats one or two short sentences three or four times, such as `"But is 20 the total? No, it is the remaining time."`
+
+The script uses teacher forcing instead of generating new tokens: it feeds each fixed text into the unedited model in a single forward pass and records the internal activations at every token.
+
+To read inside the model without changing its code, `wsp/hf.py` attaches two PyTorch hooks (small functions that PyTorch calls automatically as each layer runs):
+
+- `record_outputs(m, range(n_layers))` attaches a forward hook to every block `l` and copies the 1,024-number residual stream leaving that block (`hidden[l+1]`). The script verifies that the hook outputs match Hugging Face's `output_hidden_states` with a maximum difference of `0.00e+00`.
+- `record_mlp_acts(m, range(n_layers))` attaches a pre-hook to every `down_proj` matrix and copies the 3,584 neuron activations `a` entering `down_proj` at each token.
+
+```mermaid
+flowchart LR
+  P["8 texts in astra_pairs.json<br/>(4 productive, 4 looping)"] --> H["One pass per text with hooks:<br/>record the stream and the MLP<br/>activations at every token"]
+  H --> G["Gap = looping minus productive,<br/>averaged over the last 36 tokens.<br/>MLPs in blocks 15 (+0.263)<br/>and 19 (+0.249) add the most"]
+  G --> U["Normalize gap after block L:<br/>gives loop direction u_15, u_19"]
+  U --> S["Score each neuron:<br/>(u . W[:, j]) * gapA24[j]<br/>Pick top 64 -> configs/rebuilt.json"]
+```
+
+### 2. Where the loop signal grows across layers
+
+Because the two texts in a pair share their prompt and opening, they differ only near the end, where one loops and the other checks its answer. Following the agent's script, `lab3_rebuild.py` averages the residual stream over the last `H_WIN = 36` tokens of each text and computes the mean looping-minus-productive difference across the 4 pairs:
+
+$$\Delta h_l = \frac{1}{4} \sum_{p=0}^{3} \left( \bar{h}_{l}^{(2p+1)} - \bar{h}_{l}^{(2p)} \right)$$
+
+Here $\Delta h_l$ is the 1,024-number vector pointing from productive to looping text entering block `l`, and $\Delta h_{l+1}$ is the gap vector leaving block `l`.
+
+We can measure how much block `l` pushes the stream along the new gap direction $d = \Delta h_{l+1} / \|\Delta h_{l+1}\|$. Block `l` adds its mixer output plus its MLP output to the stream, so the total growth along $d$ splits cleanly into two pieces:
+
+$$\text{total growth} = (\Delta h_{l+1} - \Delta h_l) \cdot d = \underbrace{(W_{\text{down}, l} \Delta a_{36, l}) \cdot d}_{\text{MLP share}} + \underbrace{\text{total} - \text{MLP}}_{\text{mixer share}}$$
+
+Sorting all blocks by the MLP share gives `[(15, 0.263), (19, 0.249), (21, 0.164), (14, 0.149)]`. The MLPs in blocks 15 and 19 push hardest toward the loop, so the agent chose those two layers from this table. Both happen to be full-attention blocks (Qwen3.5-0.8B uses full attention in every fourth block: 3, 7, 11, 15, 19, and 23, and DeltaNet in the rest), though the choice came from the MLP numbers rather than the block type.
+
+### 3. Computing `u` and scoring the 3,584 neurons
+
+At each chosen layer $L \in \{15, 19\}$, the loop direction `u` is the gap vector leaving block $L$, scaled to length 1:
+
+$$u_L = \frac{\Delta h_{L+1}}{\|\Delta h_{L+1}\|}$$
+
+By construction, $+u_L$ points from productive toward looping text, resolving the sign ambiguity from Lab 1's SVD (singular value decomposition, which splits a matrix change into independent pieces):
+
+- At layer 15, $\cos(u_{15}^{\text{rebuilt}}, u_{15}^{\text{released}}) = +0.99998$.
+- At layer 19, $\cos(u_{19}^{\text{rebuilt}}, u_{19}^{\text{released}}) = -0.99998$ (the SVD convention in `results/edit_info.json` happened to pick the productive sign for layer 19).
+- With both vectors pointing toward looping, $\cos(u_{15}, u_{19}) = +0.849$, so the two loop directions are similar, about 32 degrees apart, and not opposite.
+
+Next, we find which neurons in layer $L$'s MLP push the stream along $u_L$ during a loop. A neuron adds to the loop gap along $u_L$ when two things hold at once: its column $W[:, j]$ writes along $u_L$, and its activation differs between looping and productive text.
+
+```mermaid
+flowchart TD
+  W["Write alignment:<br/>write[j] = u_L . W[:, j]<br/>(how much neuron j pushes along u_L)"] --> Mult["Multiply:<br/>score[j] = write[j] * gapA24[j]"]
+  A["Activation difference (last 24 tokens):<br/>gapA24[j] = mean(a_loop[j] - a_prod[j])<br/>(how much more neuron j fires in loops)"] --> Mult
+  Mult --> Top["Sort descending and take top 64:<br/>matches 64/64 released columns<br/>at both layer 15 and layer 19"]
+```
+
+Over the agent's window of the last `A_WIN = 24` tokens, neuron `j` fires $\Delta a_{24}[j]$ more on average in loops than in productive text, and it writes $(u_L \cdot W[:, j])$ along $u_L$ per unit of activation. Their product is the amount neuron `j` adds to the loop direction:
+
+$$\text{score}_j = (u_L \cdot W[:, j]) \times \Delta a_{24}[j]$$
+
+A neuron with a negative `write[j]` and a negative `gapA24[j]` (writing toward productive text while firing less in loops) also gets a positive score. Taking the top 64 of the 3,584 neurons by `score` reproduces `64 of 64` columns from the released edit in layer 15 and `64 of 64` in layer 19.
+
+### 4. Concentration of scores and behavior of the rebuilt edit
+
+The cumulative share of positive score (`positive_score_share_top8_32_64`) shows how spread out the effect is across neurons:
+
+- In layer 15, the top 8 neurons hold `19.3%` of the positive score, the top 32 hold `38.9%`, and the top 64 hold `49.8%` (about 50%). No single neuron dominates; the top one, neuron 294, holds 5.6%.
+- In layer 19, the top 8 neurons hold `39.7%`, the top 32 hold `49.8%`, and the top 64 hold `55.5%`. Neuron 11 alone scores `0.134`, or 24% of the layer's positive score (`0.552`), ahead of the second-highest, neuron 1, at `0.028`.
+
+The 64th neuron in layer 15 scores `0.00136` and the 65th scores `0.00134`, so the cut at 64 goes through a smooth tail rather than a sharp drop to zero. It captures about half of the positive loop contribution while touching under 1.8% of the neurons. In Lab 7, `rebuilt` matches the released `p50c` edit: both shift WikiText-2 perplexity (prediction uncertainty on ordinary text, lower is better) by `+0.79%` and give a WikiText KL divergence (change in next-token probabilities) of `0.0333` (`0.0096` vs `0.0097` on productive texts, `0.0238` vs `0.0241` on looping texts). On `dev256`, both score `61.5%`.
+
+## Check yourself
+
+1. Why does the neuron score multiply `write = u @ Wd[L]` by `gapA24[L]` instead of selecting neurons by `|u @ Wd[L]|` alone?
+<details><summary>Answer</summary>
+
+Even if `W[:, j]` has a large dot product with `u`, editing it does nothing during loops if neuron `j` never fires there, or disturbs normal tokens as much as looping tokens if it fires equally on both. Multiplying `write[j]` by `gapA24[j]` measures what neuron `j` adds along `u` to the looping-minus-productive gap.
+</details>
+
+2. In layer 19, `cos(rebuilt u, released u)` is `-0.99998`. Since the signs are opposite, why doesn't the sign matter for the edit formula?
+<details><summary>Answer</summary>
+
+From Lab 0, the projection edit `W' = W - alpha * u (u^T W_S)` multiplies `u` by `u^T`, so `+u` and `-u` produce the same matrix `W'`. The negative cosine only means that Lab 1's SVD picked the productive sign for layer 19, whereas Lab 3's `gapH[L + 1]` points toward looping. Because the cosine magnitude is `0.99998` and not `1`, the rebuilt weights are very close to the released ones, though not bit-for-bit identical.
+</details>
+
+3. Look at the curve in `lab3_neuron_scores.png`. Why did the agent stop at 64 neurons per layer instead of editing only neuron 11, or editing all 3,584 neurons?
+<details><summary>Answer</summary>
+
+In the agent's notes (`exp8_math/runs/p50c/explanations.md`), switching off neuron 11 of layer 19 alone gained only +6.25 percentage points because one neuron carries only part of the push along `u`. Editing all 3,584 neurons at strength 1 or 2 gained +24.2 and +34.8 points. At strength 4 with a 16,384-token cap (`fwd_41` and `fwd_42`), both the all-neuron and top-64 edits gained `+26.56` points, and the all-neuron edit left 10 answers ending in whitespace runs against 2 for top-64. Keeping the top 64 neurons (about half of the positive score: `50%` at layer 15, `55%` at layer 19) let the agent raise `alpha` to `6`.
+</details>
+
+## Try this next
+
+You can test recipe changes without editing `labs/lab3_rebuild.py`. Passing any non-default flag (`--pairs N`, `--windows STREAM NEURONS`, or `--layers ...`) writes to `results/lab3/try-<name>/`, named after the flags you changed, and leaves `results/lab3/rebuild.json`, `configs/rebuilt.json`, and `results/figures/` untouched.
+
+Each try folder holds `rebuild.json`, the two figures (`lab3_where_the_loop_grows.png` and `lab3_neuron_scores.png`), `variant.json` (edit `rebuilt-<name>`), and `plan.json` (a one-GPU `dev256` run `dev/rebuilt-<name>`). The script prints the two commands to build `/nvme/wsp-lab/models/qwen35-rebuilt-<name>` on CPU in a few seconds and test it on the 256 unseen practice problems (about 20 minutes on one GPU):
+
+```bash
+CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants results/lab3/try-<name>/variant.json
+python scripts/run_jobs.py results/lab3/try-<name>/plan.json --gpus 7 --only dev/rebuilt-<name>
+```
+
+We ran all three tries below, built their models, and evaluated them in about 11 minutes across 6 GPUs (2 shards per run) with `plans/lab3-tries.json`. We then scored their side effects on WikiText-2 and the 8 texts in Lab 7 (`results/lab7/try-lab3-tries/`) and checked how their 512 answers ended (`results/lab3/endings-tries.json`):
+
+```bash
+python scripts/run_jobs.py plans/lab3-tries.json --gpus 0,1,2,3,4,6
+CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models rebuilt rebuilt-pairs2 rebuilt-windows36-36 rebuilt-layers15-19-21 --try-name lab3-tries
+```
+
+(`scripts/run_jobs.py` skips any shard with a `.done.json` marker; move the existing run folder under `/nvme/wsp-lab/runs/dev/` aside first to generate fresh answers.)
+
+1. **Use only the agent's first 2 pairs (`--pairs 2`, folder `results/lab3/try-pairs2/`).**
+   - **Predict first:** With half the data (the first 4 texts instead of all 8), how far will `cos(rebuilt u, released u)` and the 64-neuron overlap drop, and will `dev256` accuracy suffer?
+   - **Run it:**
+     ```bash
+     CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py --pairs 2
+     CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants results/lab3/try-pairs2/variant.json
+     python scripts/run_jobs.py results/lab3/try-pairs2/plan.json --gpus 7 --only dev/rebuilt-pairs2
+     ```
+   - **What we saw:** Blocks 15 and 19 still lead in MLP growth (`[(15, 0.201), (19, 0.188), (22, 0.167), (14, 0.163)]`), while the direction and neuron list get rougher:
+     - Layer 15: `cos(rebuilt u, released u) = +0.82157` (`0.822`), `42/64` neurons match, and the top 8 / 32 / 64 neurons hold `[0.144, 0.278, 0.386]` (`14.4% / 27.8% / 38.6%`) of the positive score.
+     - Layer 19: `cos(rebuilt u, released u) = -0.78384` (`-0.784`), `31/64` neurons match, and the top 8 / 32 / 64 neurons hold `[0.219, 0.348, 0.419]` (`21.9% / 34.8% / 41.9%`).
+     - On `dev256`, `dev/rebuilt-pairs2` reaches `62.3% +/- 2.7` accuracy (`+50.2 +/- 2.6` points over `dev/base`, `129` rescued, `0` lost), matching the 4-pair rebuild (`61.5%`). What changes is how the edit works and how hard it pushes: more answers hit the 32,768-token cap (`25.2%` vs `17.6%`), and more cut answers end in repetition (`74.4%` vs `32.2%`, or `68.2%` vs `15.6%` for verbatim loops; across all 512 answers in `results/lab3/endings-tries.json`, `86` are cut while thinking and `74` close out of a loop, vs `55` and `118` for `rebuilt` in `results/lab3/endings.json`). In Lab 7, it disturbs WikiText-2 less (perplexity `+0.43%` vs `+0.79%`, KL `0.0194` vs `0.0333`) and is less selective on the 8 texts (looping KL over productive KL is `0.0109 / 0.0075 = 1.46`, vs `0.0238 / 0.0096 = 2.48` for `rebuilt`).
+
+2. **Average neuron activations over 36 tokens instead of 24 (`--windows 36 36`, folder `results/lab3/try-windows36-36/`).**
+   - **Predict first:** The agent used 36 tokens for the stream (`H_WIN = 36`) and 24 tokens for the neuron scores (`A_WIN = 24`). If you set `A_WIN` to `36`, will `u` change, and will the extra 12 tokens shift the 64 chosen neurons enough to affect `dev256`?
+   - **Run it:**
+     ```bash
+     CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py --windows 36 36
+     CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants results/lab3/try-windows36-36/variant.json
+     python scripts/run_jobs.py results/lab3/try-windows36-36/plan.json --gpus 7 --only dev/rebuilt-windows36-36
+     ```
+   - **What we saw:** Because `H_WIN` remains `36`, the block growth numbers and `u` are unchanged (`+0.99998` at layer 15, `-0.99998` at layer 19). The 36-token neuron window swaps a few tail neurons: `58/64` match at layer 15 (top 8 / 32 / 64 score shares `[0.187, 0.375, 0.481]`) and `51/64` match at layer 19 (shares `[0.378, 0.474, 0.533]`). On `dev256`, `dev/rebuilt-windows36-36` scores `61.9% +/- 2.8` (`+49.8 +/- 2.7` points, `21.3%` cut, `122` rescued, `0` lost), with a Lab 7 WikiText-2 perplexity change of `+0.71%` (KL `0.0326`). The 24-token window is not fragile.
+
+3. **Also edit layer 21, the third-strongest MLP (`--layers 15 19 21`, folder `results/lab3/try-layers15-19-21/`).**
+   - **Predict first:** Block 21 had the third-largest MLP addition to the loop gap (`0.164`, behind `0.263` at block 15 and `0.249` at block 19). Does applying the 64-neuron, `alpha = 6` edit to layer 21 cut more loops and raise accuracy, or only add side effects?
+   - **Run it:**
+     ```bash
+     CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py --layers 15 19 21
+     CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants results/lab3/try-layers15-19-21/variant.json
+     python scripts/run_jobs.py results/lab3/try-layers15-19-21/plan.json --gpus 7 --only dev/rebuilt-layers15-19-21
+     ```
+   - **What we saw:** Layers 15 and 19 match the main rebuild (`64/64` neurons in both). At layer 21, scores are flatter: the top 8 / 32 / 64 neurons hold `[0.173, 0.292, 0.371]` (`17.3% / 29.2% / 37.1%`) of the positive score (top 12: `[2, 1, 11, 99, 14, 85, 9, 109, 168, 335, 212, 169]`), and the edit's magnitude is `5.8%` of layer 21's `down_proj` matrix magnitude (the square root of the sum of squared entries, not the fraction of entries changed). On `dev256`, `dev/rebuilt-layers15-19-21` scores `61.3% +/- 2.8` (`+49.2 +/- 2.6` points, `21.1%` cut, `122` rescued, `0` lost, with `3` answers stopping mid-thinking before `</think>`), showing no gain over the 2-layer rebuild (`61.5%`, difference `-0.2 +/- 1.9` points). In Lab 7, WikiText-2 perplexity rises by `+1.49%` (nearly double the 2-layer rebuild's `+0.79%`) and KL climbs from `0.0333` to `0.0394`.
+
+The main rebuild and the three tries line up as follows across Lab 3, `dev256`, and Lab 7:
+
+| Edit | Cosine with released `u` (L15 / L19) | Neurons shared with released 64 (L15 / L19) | `dev256` accuracy % | Cut % | Cut answers ending in repetition % | Gain vs `dev/base` (pp), rescued / lost | Lab 7 perplexity change | KL WikiText / productive / looping |
+|---|---|---|---|---:|---:|---|---:|---|
+| `rebuilt` (`--pairs 4 --windows 36 24 --layers 15 19`) | `+0.99998 / -0.99998` | `64 / 64` | `61.5 +/- 2.8` | `17.6%` | `32.2%` | `+49.4 +/- 2.6`, `122 / 0` | `+0.79%` | `0.0333 / 0.0096 / 0.0238` |
+| `rebuilt-pairs2` (`--pairs 2`) | `+0.82157 / -0.78384` | `42 / 31` | `62.3 +/- 2.7` | `25.2%` | `74.4%` | `+50.2 +/- 2.6`, `129 / 0` | `+0.43%` | `0.0194 / 0.0075 / 0.0109` |
+| `rebuilt-windows36-36` (`--windows 36 36`) | `+0.99998 / -0.99998` | `58 / 51` | `61.9 +/- 2.8` | `21.3%` | `35.8%` | `+49.8 +/- 2.7`, `122 / 0` | `+0.71%` | `0.0326 / 0.0096 / 0.0241` |
+| `rebuilt-layers15-19-21` (`--layers 15 19 21`) | `+0.99998 / -0.99998` | `64 / 64` (+64 at L21) | `61.3 +/- 2.8` | `21.1%` | `38.0%` | `+49.2 +/- 2.6`, `122 / 0` | `+1.49%` | `0.0394 / 0.0114 / 0.0280` |
+
+**Lesson:** The recipe is forgiving. Using half the pairs (`--pairs 2`) or widening the neuron window (`--windows 36 36`) still reaches about `62%` accuracy on unseen problems. Those variants mainly shift side effects on ordinary text and the balance between preventing loops early and breaking them after they start. Editing a third layer with flatter neuron scores (`--layers 15 19 21`) increases ordinary-text damage without improving math accuracy.
+

code/labs/lab3_rebuild.py

@@ -0,0 +1,191 @@
+#!/usr/bin/env python
+"""Lab 3: rebuild the released edit from the agent's 8 sentences, using only hooks and arithmetic.
+
+    . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+    CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py      # about a minute on one GPU
+
+Experiments (each writes to its own folder, results/lab3/try-*, and leaves the main results alone; the script
+prints the commands that build the try's edit and test it on the 256 unseen practice problems):
+    CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py --pairs 2           # only the agent's first 2 pairs
+    CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py --windows 36 36     # neurons averaged over 36 final tokens, not 24
+    CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py --layers 15 19 21   # also edit layer 21, the third-strongest MLP
+(plans/lab3-tries.json tests exactly these three: runs dev/rebuilt-pairs2, dev/rebuilt-windows36-36 and
+dev/rebuilt-layers15-19-21.)
+
+What it does:
+  1. Runs the 8 texts in labs/data/astra_pairs.json (4 productive, 4 looping) through the unedited model and
+     records every block's output (the residual stream) and every MLP's neuron activations.
+  2. For every layer, measures how far apart looping and productive text sit (averaged over the last 36 tokens)
+     and how much of that gap each block adds, split into the MLP's share and the mixer's share. This is how the
+     agent picked layers 15 and 19.
+  3. Direction u at layer L = the looping-minus-productive gap after block L, scaled to length 1.
+  4. Neuron score = (how much neuron j writes along u) x (how much more neuron j fires in looping text, last 24
+     tokens). The 64 highest are the neurons the edit touches.
+  5. Compares all of it with the released edit and writes configs/rebuilt.json, an edit you can build and test.
+"""
+import argparse
+import json
+import os
+import sys
+
+import numpy as np
+import torch
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+from wsp import hf  # noqa: E402
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--pairs", type=int, default=4, choices=[1, 2, 3, 4], help="use only the agent's first N pairs")
+ap.add_argument("--windows", type=int, nargs=2, default=[36, 24], metavar=("STREAM", "NEURONS"),
+                help="final tokens to average for the stream and for the neurons (the agent used 36 and 24)")
+ap.add_argument("--layers", type=int, nargs="+", default=[15, 19], help="layers to build the edit for")
+a = ap.parse_args()
+LAYERS = tuple(a.layers)
+H_WIN, A_WIN = a.windows  # the agent's windows: last 36 tokens for the stream, last 24 for neurons
+TRY = "try-" + "-".join(p for p in (f"pairs{a.pairs}" if a.pairs != 4 else "",
+                                     f"windows{H_WIN}-{A_WIN}" if a.windows != [36, 24] else "",
+                                     "layers" + "-".join(map(str, LAYERS)) if list(LAYERS) != [15, 19] else "") if p)
+TRY = None if TRY == "try-" else TRY
+OUT = os.path.join(ROOT, "results", "lab3", *([TRY] if TRY else []))
+FIG = OUT if TRY else os.path.join(ROOT, "results", "figures")
+os.makedirs(OUT, exist_ok=True)
+os.makedirs(FIG, exist_ok=True)
+
+pairs = json.load(open(os.path.join(ROOT, "labs", "data", "astra_pairs.json")))
+texts = pairs["texts"][:2 * a.pairs]
+N = len(texts)
+m, tok = hf.load("base")
+n_layers = len(hf.blocks(m))
+# hidden_states[-1] has the final norm applied, so the last block's raw output is not available to measure
+assert all(0 <= L < n_layers - 1 for L in LAYERS), f"--layers must be between 0 and {n_layers - 2}"
+
+# 1. record. hidden[l] is the stream entering block l (hidden[0] = token embeddings, hidden[l+1] = block l's output)
+hidden, acts, lens = [], [], []
+with torch.no_grad(), hf.record_outputs(m, range(n_layers)) as outs, hf.record_mlp_acts(m, range(n_layers)) as A:
+    for t in texts:
+        ids = hf.encode(tok, t)
+        lens.append(ids.shape[1])
+        o = m(ids, output_hidden_states=True)
+        hidden.append([h[0].float().cpu() for h in o.hidden_states])
+    # the hook copy of block l's output must equal transformers' hidden_states[l+1]
+    hook_check = max(float((outs[l][i] - hidden[i][l + 1]).abs().max()) for i in range(N) for l in range(n_layers - 1))
+assert lens == pairs["agent_token_lengths"][:N], (lens, pairs["agent_token_lengths"])
+print(f"token counts match the agent's: {lens}; hooks vs hidden_states max diff {hook_check:.2e}")
+
+
+def pair_gap(per_text):  # per_text: [N, d] -> mean over the pairs of (looping - productive)
+    x = torch.stack(per_text)
+    return (x[1::2] - x[0::2]).mean(0)
+
+
+gapH = [pair_gap([hidden[i][l][-H_WIN:].mean(0) for i in range(N)]) for l in range(n_layers + 1)]
+gapA36 = [pair_gap([A[l][i][-H_WIN:].mean(0) for i in range(N)]) for l in range(n_layers)]
+gapA24 = {L: pair_gap([A[L][i][-A_WIN:].mean(0) for i in range(N)]) for L in LAYERS}
+Wd = {l: hf.down_proj(m, l).weight.detach().float().cpu() for l in range(n_layers)}
+
+# 2. where does the gap grow? (the agent's table, pass 3)
+growth = []
+for l in range(n_layers - 1):
+    d = gapH[l + 1] / gapH[l + 1].norm()
+    total = float((gapH[l + 1] - gapH[l]) @ d)
+    mlp = float((Wd[l] @ gapA36[l]) @ d)
+    growth.append({"layer": l, "gap_size_after": float(gapH[l + 1].norm()), "growth": total, "mlp": mlp,
+                   "mixer": total - mlp})
+top_mlp = sorted(growth, key=lambda g: -g["mlp"])[:4]
+print("layers whose MLP adds most to the gap:", [(g["layer"], round(g["mlp"], 3)) for g in top_mlp])
+
+# 3-4. direction and neurons, then compare with the released edit
+rel = hf.released_edit()
+rebuilt, report = {}, {"token_lengths": lens, "hook_vs_hidden_states_max_diff": hook_check, "growth": growth,
+                       "layers": {}}
+for L in LAYERS:
+    u = gapH[L + 1] / gapH[L + 1].norm()  # points from productive toward looping
+    write = u @ Wd[L]                       # [3584]: how much each neuron writes along u per unit of activation
+    score = write * gapA24[L]
+    order = torch.argsort(score, descending=True)
+    top = sorted(order[:64].tolist())
+    pos = score.clamp_min(0).sum()
+    cos = float(u @ rel[L]["u"] / rel[L]["u"].norm()) if L in rel else None  # the released edit: layers 15, 19
+    report["layers"][str(L)] = {
+        "cos_with_released_u": cos,
+        "released_u_points_to": None if cos is None else "looping" if cos > 0 else "productive",
+        "overlap_with_released_64": len(set(top) & set(rel[L]["cols"])) if L in rel else None,
+        "top12": order[:12].tolist(),
+        "positive_score_share_top8_32_64": [float(score[order[:k]].clamp_min(0).sum() / pos) for k in (8, 32, 64)],
+        "gap_size": float(gapH[L + 1].norm()),
+        "u_loop_sign": [round(float(x), 8) for x in u],
+        "cols": top,
+        "scores_sorted": [round(float(x), 6) for x in score[order]],
+    }
+    rebuilt[str(L)] = {"alpha": 6, "dir": [round(float(x), 8) for x in u], "cols": top}
+    vs = (f"cos(rebuilt u, released u) = {cos:+.5f}; neurons shared with the released 64: "
+          f"{report['layers'][str(L)]['overlap_with_released_64']}/64; " if L in rel
+          else "the released edit leaves this layer alone; ")
+    print(f"layer {L}: {vs}top-8/32/64 share of positive score: "
+          f"{[round(x, 3) for x in report['layers'][str(L)]['positive_score_share_top8_32_64']]}")
+if 15 in LAYERS and 19 in LAYERS:
+    u15, u19 = (torch.tensor(report["layers"][str(L)]["u_loop_sign"]) for L in (15, 19))
+    report["cos_u15_u19_loop_sign"] = float(u15 @ u19)
+    print(f"cos(u15, u19) with both pointing toward looping: {report['cos_u15_u19_loop_sign']:+.3f}")
+
+json.dump(report, open(os.path.join(OUT, "rebuild.json"), "w"), indent=1)
+if TRY:  # its own variant name, so building it never replaces models/qwen35-rebuilt
+    name = f"rebuilt-{TRY[4:]}"
+    variants, plan = os.path.join(OUT, "variant.json"), os.path.join(OUT, "plan.json")
+    json.dump({name: {"about": f"Lab 3 experiment: {TRY[4:]}", "layers": rebuilt}}, open(variants, "w"), indent=1)
+    json.dump({"about": f"Lab 3 experiment {TRY[4:]}: the rebuilt edit on the 256 unseen practice problems",
+               "runs": [{"name": f"dev/{name}", "model": f"qwen35-{name}", "data": "dev256", "n": 2, "shards": 1,
+                         "est": 8}]}, open(plan, "w"), indent=1)
+else:
+    name = "rebuilt"
+    variants, plan = os.path.join(ROOT, "configs", "rebuilt.json"), os.path.join(ROOT, "plans", "extra.json")
+    json.dump({"rebuilt": {"about": "Lab 3: the edit rebuilt from the agent's 8 texts (alpha 6, my u, my top 64)",
+                           "layers": rebuilt}}, open(variants, "w"), indent=1)
+
+# figures
+import matplotlib  # noqa: E402
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+
+fig, ax = plt.subplots(1, 2, figsize=(13, 4.2))
+xs = [g["layer"] for g in growth]
+# leave out the last position: hidden_states[-1] has gone through the final norm, which rescales everything
+ax[0].plot(range(n_layers), [float(g.norm()) for g in gapH[:n_layers]], "o-", color="#0969da")
+ax[0].text(0.02, 0.95, f"not shown: after the last block and the final norm\nthe gap is {float(gapH[-1].norm()):.0f}",
+           transform=ax[0].transAxes, va="top", fontsize=8, color="#57606a")
+ax[0].set_xlabel("position in the stream (0 = embeddings, l+1 = after block l)")
+ax[0].set_ylabel("size of looping - productive gap")
+ax[0].set_title("The loop signal builds up through the network")
+ax[1].bar([x - 0.2 for x in xs], [g["mlp"] for g in growth], 0.4, label="added by the MLP", color="#cf222e")
+ax[1].bar([x + 0.2 for x in xs], [g["mixer"] for g in growth], 0.4, label="added by attention / DeltaNet",
+          color="#8c959f")
+for L in LAYERS:
+    ax[1].annotate(f"layer {L}", (L - 0.2, growth[L]["mlp"]), xytext=(0, 6), textcoords="offset points",
+                   ha="center", fontsize=9)
+ax[1].set_xlabel("block")
+ax[1].set_ylabel("growth of the gap along its own direction")
+ax[1].set_title("Which blocks push text toward the loop")
+ax[1].legend()
+fig.tight_layout()
+fig.savefig(os.path.join(FIG, "lab3_where_the_loop_grows.png"), dpi=130)
+
+fig, ax = plt.subplots(1, len(LAYERS), figsize=(6.5 * len(LAYERS), 3.8), squeeze=False)
+ax = ax[0]
+for k, L in enumerate(LAYERS):
+    s = np.array(report["layers"][str(L)]["scores_sorted"])
+    ax[k].plot(np.arange(len(s)), s, color="#8c959f", lw=1)
+    ax[k].plot(np.arange(64), s[:64], color="#cf222e", lw=2, label="the 64 neurons the edit touches")
+    ax[k].set_xscale("symlog", linthresh=10)
+    ax[k].axhline(0, color="black", lw=0.5)
+    ax[k].set_title(f"layer {L}: neuron scores, highest first")
+    ax[k].set_xlabel("neuron rank (of 3,584)")
+    ax[k].set_ylabel("writes along u x fires more in loops")
+    ax[k].legend()
+fig.tight_layout()
+fig.savefig(os.path.join(FIG, "lab3_neuron_scores.png"), dpi=130)
+print("wrote", OUT, "(an experiment: configs/rebuilt.json and results/figures are untouched)" if TRY
+      else "and configs/rebuilt.json and two figures")
+print("next, build the edit and test it on the 256 unseen practice problems (about 20 minutes on one GPU):")
+print(f"  CUDA_VISIBLE_DEVICES=\"\" python scripts/build_models.py --variants {os.path.relpath(variants, ROOT)}")
+print(f"  python scripts/run_jobs.py {os.path.relpath(plan, ROOT)} --gpus 7 --only dev/{name}")

code/labs/lab4_dose.md

@@ -0,0 +1,369 @@
+# Lab 4: Turn the knob, and the controls
+
+**Big thing:** You will sweep the edit strength `alpha` from `-4` to `12` across 256 unseen practice problems. You will also test each part of the edit (layer 15 vs layer 19, 64 neurons vs all 3,584 neurons, and same-size random controls) and classify how every answer ended.
+
+**Why it matters:** One benchmark gain shows that an edit helped, while a dose sweep and matched controls test why it worked and where it breaks. In these runs, the gain needed both the loop direction `u` and neurons chosen because they fire during loops. At `alpha = -4`, accuracy even rises while the edit pushes the model harder into loops.
+
+**Time and hardware:** Building the 12 variant models takes about 2 minutes on CPU. Running one variant on the 256 practice problems (`dev256`, 512 answers) takes 17 to 45 minutes on 1 NVIDIA A100 GPU, and the full `plans/sampler.json` plan takes about 10.6 GPU-hours across 7 GPUs. Classifying endings and inspecting answers with `labs/lab4_endings.py` and `labs/peek.py` takes a few seconds on CPU.
+
+**Theory guide:** Section 8.5.
+
+## Before you run: predict
+
+Write down a short guess for each question before running the commands or reading the tables.
+
+1. As `alpha` grows through `0, 1, 2, 4, 6, 8, 12`, does accuracy keep rising to `12`, or plateau and drop? Why?
+2. Keep the same 64 neurons and weight-change size (Frobenius norm) as the released edit, and replace only the loop direction `u` with a random direction at right angles to `u` (`randdir`). How many percentage points of the `+49.4` gain will survive?
+3. What if you keep the loop direction `u` and the same weight-change size, and apply it to 64 random neurons instead of the chosen 64 (`randcols`)?
+4. At `alpha = -4`, the edit multiplies the 64 neurons' push along the loop direction by `1 - (-4) = +5`, pushing 5 times harder toward loops. Will `dev256` accuracy be lower than the unedited model (`12.1%`), about the same, or higher?
+
+## Run it
+
+### 1. How edit variants are defined
+
+Every weight edit in this lab is a JSON entry in `configs/variants.json`, where `_doc` explains the schema:
+
+- Each entry becomes a model folder `/nvme/wsp-lab/models/qwen35-<name>`.
+- Inside `"layers"`, each edited layer (`"15"` or `"19"`) specifies `"alpha"` (the strength knob).
+- `"dir"` chooses the 1,024-number unit vector (a list of numbers scaled to have length one) in the residual stream. The residual stream is the running list of 1,024 numbers per token that every layer reads from and adds to. A token is a word piece. `"released"` uses the paper's direction `u` for that layer (default), and `"random_orth:SEED"` picks a random unit vector at right angles to `u`.
+- `"cols"` chooses which columns of `down_proj` to edit. This is the MLP output matrix of shape `1,024 x 3,584`; the MLP is the layer's bank of neurons. Column `j` is the vector neuron `j` adds to the stream per unit of activation, or firing strength. `"released"` uses the paper's 64 neurons (default), `"all"` edits all 3,584 neurons, and `"random:SEED"` picks 64 other neurons at random.
+- `"norm_match": true` rescales `alpha` so the Euclidean size (Frobenius norm) of the matrix change `W' - W` matches the released edit at that `alpha`.
+
+Two entries from `configs/variants.json`, for `a4` (`alpha = 4` on the released direction and neurons) and the random-direction control `randdir`:
+
+```json
+ "a4": {"about": "flip and triple the loop component",
+        "layers": {"15": {"alpha": 4}, "19": {"alpha": 4}}},
+ "randdir": {"about": "control: same 64 neurons, same size of change, but a random direction instead of u",
+             "layers": {"15": {"alpha": 6, "dir": "random_orth:15", "norm_match": true},
+                        "19": {"alpha": 6, "dir": "random_orth:19", "norm_match": true}}}
+```
+
+### 2. Build the edited model folders on CPU
+
+Connect to `sampler`, activate the environment, and run `scripts/build_models.py` on CPU (`CUDA_VISIBLE_DEVICES=""` hides the GPUs so PyTorch stays on CPU):
+
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+CUDA_VISIBLE_DEVICES="" python scripts/build_models.py
+```
+
+For each entry in `configs/variants.json`, `scripts/build_models.py` symlinks the tokenizer and config files from `/nvme/wsp-lab/models/Qwen3.5-0.8B` and writes `model.safetensors-00001-of-00001.safetensors` with the edited `down_proj` tensors in layers 15 and 19. If a folder already matches its spec in `edit_note.json`, the script prints `exists <name>` and skips it.
+
+### 3. Run the generation plan on `sampler`
+
+The generation runs are in `plans/sampler.json`. Every `dev/*` run tests one model on `dev256` (128 GSM8K-train and 128 MATH-train practice problems the agent never touched, with `n = 2` answers per problem, or `512` answers per run). One entry from `plans/sampler.json`:
+
+```json
+  {"name": "dev/a4", "model": "qwen35-a4", "data": "dev256", "n": 2, "shards": 1, "est": 8}
+```
+
+Use `nvidia-smi` to pick a free GPU. One `dev256` run on one A100 GPU takes 17 to 45 minutes depending on answer length (`dev/a12` finishes in 16.8 minutes because its answers are shorter; `dev/base` takes 44.8 minutes because almost every answer hits the 32,768-token cap). The full `plans/sampler.json` plan (22 runs, including Lab 5 decoding runs and `build/base`) takes about 10.6 GPU-hours:
+
+```bash
+nvidia-smi
+python scripts/run_jobs.py plans/sampler.json --gpus 0-6
+python scripts/summarize.py /nvme/wsp-lab/runs
+```
+
+**Note on rerunning finished jobs:** `scripts/run_jobs.py` skips any shard that already has `shard*.done.json`, so the command above finishes immediately without generating new text. To regenerate a run yourself, move our finished folder aside first:
+
+```bash
+mv /nvme/wsp-lab/runs/dev/a4 /nvme/wsp-lab/runs/dev/a4-ours
+python scripts/run_jobs.py plans/sampler.json --gpus 7 --only dev/a4
+```
+
+Then `dev/a4` holds your run and `dev/a4-ours` keeps ours.
+
+### 4. Classify how every answer ended and inspect single answers
+
+`scripts/pull_runs.sh` copies only per-answer summary numbers to the laptop, so the full answer texts (`shard*.texts.jsonl.gz`) stay on the VM that generated them (all `dev/*` runs in `plans/sampler.json` are on `sampler`). Run `labs/lab4_endings.py` on CPU on `sampler` to classify every answer in the sweep and controls, and to print examples where `dev/a-4` closed its thinking out of a loop:
+
+```bash
+export CUDA_VISIBLE_DEVICES=""
+python labs/lab4_endings.py
+python labs/lab4_endings.py dev/base dev/p50c dev/l15only dev/l19only dev/all1 dev/all2 dev/randdir dev/randcols \
+    --out lab4/endings-controls
+python labs/lab4_endings.py dev/a-4 --show 2
+```
+
+The first two commands write `results/lab4/endings.json` and `results/lab4/endings-controls.json`; the third writes `results/lab4/try-dev_a-4/endings.json`.
+
+To inspect single answers on `sampler`, use `labs/peek.py` (CPU, a few seconds). Without `--id`, it lists all 512 answers in a run; with `--id`, it prints the problem, whether the answer looped, the token where the loop started, the text around `</think>`, and the ending:
+
+```bash
+python labs/peek.py dev/p50c
+python labs/peek.py dev/p50c --id gsm8k_train_5628
+python labs/peek.py dev/a-4 --id gsm8k_train_5628
+```
+
+## What you should see
+
+### 1. The dose curve (`alpha` from `-4` to `12`)
+
+Every row below tests the same 64 neurons per layer and loop direction `u` at layers 15 and 19 on `dev256` (256 problems, 2 answers each = 512 answers), varying only `alpha`. Accuracy and gain include their standard error (`SE`, the typical sampling uncertainty across 256 problems; a gap of at least 2 SE is unlikely to be chance). Gains in percentage points (`pp`) and rescued or lost problem counts are paired by problem against `dev/base` (`alpha = 0`). A rescued problem has at least one correct edited answer and no correct base answer; a lost problem is the reverse. The last column, from Lab 7 (`results/lab7/side_effects.json`), gives the percentage change in perplexity, a measure of how surprised the model is by the text. Perplexity is $e$ to the average negative log-probability of the true next token across 65,536 tokens of WikiText-2, so lower is better and `+0.0%` is the unedited base score of `20.009`.
+
+| `alpha` | What it does to the 64 neurons' push along `u` | Accuracy % | Cut at cap % | Mean tokens | Correct when finished % | Gain vs base (pp), rescued / lost | WikiText perplexity change (Lab 7) |
+|---|---|---|---|---|---|---|---|
+| `-4` (`dev/a-4`) | 5 times stronger toward loops | `19.7 +/- 1.9` | `65.8` | `26,063` | `55.4` | `+7.6 +/- 2.2`, `54 / 24` | `+5.39%` |
+| `0` (`dev/base`) | Unedited | `12.1 +/- 1.6` | `88.7` | `29,683` | `100.0` | `-` | `0.00%` |
+| `1` (`dev/a1`) | Erased (`1 - alpha = 0`) | `22.7 +/- 2.1` | `77.1` | `26,964` | `96.6` | `+10.5 +/- 1.8`, `42 / 7` | `-0.49%` |
+| `2` (`dev/a2`) | Mirrored (`1 - alpha = -1`) | `35.2 +/- 2.6` | `62.7` | `23,911` | `92.1` | `+23.0 +/- 2.3`, `71 / 5` | `-0.69%` |
+| `4` (`dev/a4`) | Flipped, 3 times (`1 - alpha = -3`) | `59.2 +/- 2.8` | `29.1` | `17,730` | `82.9` | `+47.1 +/- 2.6`, `118 / 0` | `-0.47%` |
+| `6` (`dev/p50c`) | Flipped, 5 times (released edit) | `61.5 +/- 2.8` | `19.7` | `16,104` | `76.6` | `+49.4 +/- 2.6`, `125 / 0` | `+0.79%` |
+| `8` (`dev/a8`) | Flipped, 7 times (`1 - alpha = -7`) | `60.7 +/- 2.8` | `14.1` | `14,589` | `70.7` | `+48.6 +/- 2.6`, `119 / 0` | `+3.33%` |
+| `12` (`dev/a12`) | Flipped, 11 times (`1 - alpha = -11`) | `51.8 +/- 2.8` | `12.3` | `13,515` | `59.0` | `+39.6 +/- 2.7`, `102 / 0` | `+14.34%` |
+
+![Four panels against edit strength alpha from -4 to 12: accuracy, percent of answers cut at the token cap, mean answer length, and WikiText perplexity change.](../results/figures/lab4_dose.png)
+
+How to read `lab4_dose.png`: the four panels plot accuracy, percentage of answers cut at the 32,768-token cap, mean answer length (thousand tokens), and Lab 7's WikiText perplexity change across `alpha = -4, 0, 1, 2, 4, 6, 8, 12`. The dose sweep is blue in the first three panels and red in the fourth. The three open markers to the right of `alpha = 6` (a red square at `x = 6.35` for `rebuilt`, and purple triangles at `x = 6.7` and `x = 7.05` for Lab 8's `capA-a6` and `capB-a6`) also use `alpha = 6`; the plot shifts them right of `6` so their error bars do not overlap `p50c`.
+
+### 2. Which parts of the edit matter (layers, neuron subset, and random controls)
+
+All runs below are at `alpha = 6` unless noted, on the same 256 practice problems (`512` answers per run):
+
+| Model | What changes | Accuracy % | Cut at cap % | Mean tokens | Correct when finished % | Gain vs base (pp), rescued / lost | WikiText perplexity change (Lab 7) |
+|---|---|---|---|---|---|---|---|
+| `dev/p50c` | Released edit (`alpha = 6`, 64 neurons, layers 15 and 19) | `61.5 +/- 2.8` | `19.7` | `16,104` | `76.6` | `+49.4 +/- 2.6`, `125 / 0` | `+0.79%` |
+| `dev/l15only` | Released edit on layer 15 only | `60.4 +/- 2.9` | `28.5` | `17,362` | `84.2` | `+48.2 +/- 2.7`, `117 / 0` | `+0.05%` |
+| `dev/l19only` | Released edit on layer 19 only | `46.9 +/- 2.7` | `49.0` | `20,729` | `91.2` | `+34.8 +/- 2.4`, `100 / 0` | `+1.65%` |
+| `dev/all1` | `alpha = 1` on all 3,584 neurons (layers 15 and 19) | `28.7 +/- 2.3` | `71.9` | `25,609` | `97.9` | `+16.6 +/- 2.0`, `61 / 6` | `-0.42%` |
+| `dev/all2` | `alpha = 2` on all 3,584 neurons (layers 15 and 19) | `49.4 +/- 2.8` | `46.7` | `20,117` | `91.9` | `+37.3 +/- 2.5`, `97 / 0` | `-0.36%` |
+| `dev/randdir` | Control: same 64 neurons, random orthogonal direction, same change size | `14.5 +/- 1.8` | `87.3` | `29,486` | `98.5` | `+2.3 +/- 1.6`, `22 / 18` | `+0.30%` |
+| `dev/randcols` | Control: direction `u`, 64 random neurons, same change size | `14.6 +/- 1.9` | `86.7` | `29,113` | `100.0` | `+2.5 +/- 1.6`, `23 / 20` | `-0.10%` |
+
+![Horizontal bars of dev256 accuracy for each model: the unedited base in grey, the released edit and the Lab 3 rebuild in green, the two Lab 8 edits in purple, the ablations in blue, and the two random controls in red.](../results/figures/lab4_controls.png)
+
+How to read `lab4_controls.png`: each horizontal bar is one model on `dev256`, grouped by color. Grey is the unedited base model (`12.1%`), and green is the released edit (`61.5%`) and your Lab 3 rebuild (`61.5%`). Purple shows the two Lab 8 capstone edits built from the model's own loops (`capA-a6` at `43.0%` and `capB-a6` at `58.2%`). Blue shows the ablations (`l15only`, `l19only`, `all1`, `all2`). Red at the bottom shows the two norm-matched random controls (`randdir` at `14.5%` and `randcols` at `14.6%`), which barely pass the grey bar.
+
+### 3. How the agent discovered this path in its own notebook
+
+You can read the agent's notebook in `external/lm_understand-logs/exp8_math/runs/p50c/work_notes/experiment_history.md` (lines 55 to 112 and 225 to 374; online at <https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8_math/runs/p50c/work_notes/experiment_history.md>). The variants above retrace its main steps. Because failed requests did not count toward its budget, its pass number can trail the request file number in `external/lm_understand-logs/exp8_math/runs/p50c/requests/`. Each step lists both:
+
+- 4th forward pass (`fwd_5`): switching off one neuron (layer 19, neuron 11) gave `+6.25 +/- 4.35` percentage points, too noisy to trust on a small batch.
+- 10th forward pass (`fwd_13`): removing the loop direction (`alpha = 1`) from all 3,584 columns of layers 15 and 19 raised correct answers from `11 to 42 of 128` (`+24.22 +/- 3.96` points).
+- 19th forward pass (`fwd_27`): mirroring the direction (`alpha = 2`) across all 3,584 columns jumped from `21 to 110 of 256` (`+34.77 +/- 3.53` points).
+- 20th forward pass (`fwd_29`): raising `alpha` to `3` across all columns reached `+39.84` points.
+- 23rd forward pass (`fwd_35`): a random orthogonal direction of the same norm changed accuracy by only `+1.56 +/- 3.52` points, a sign that the gain came from `u` and not random weight noise.
+- 27th and 28th forward passes (`fwd_41` and `fwd_42`): at `alpha = 4` with a 16,384-token limit, editing the top 64 neurons per layer and editing all 3,584 both gained `+26.56` points, but the 64-neuron edit left fewer answers ending in long whitespace runs (`2` vs `10`).
+- 30th forward pass (`fwd_45`): raising the 64-neuron edit to `alpha = 6` jumped from `7/64 to 35/64` (`+43.75` points), with answers cut at the limit falling from `57` to `5`.
+- 33rd and 34th forward passes (`fwd_50` and `fwd_52`): the frozen edit raised 128 fresh practice problems from `9 to 74` correct (`+50.78` points). Later passes tested more fresh problems. On the 256 fresh problems of `final_1`, it went from `10.16%` to `58.98%` (`+48.83 +/- 3.23` points, `127` rescued, `2` lost). Across `704` checked practice items it went from `7.95%` to `57.53%` (line 244), and across the final `1,152` items from `10.59%` to `60.42%` (line 374).
+
+Our `all1` (`+16.6 +/- 2.0`) and `all2` (`+37.3 +/- 2.5`) apply the released direction `u` to all 3,584 columns, so they are close copies of the agent's 10th and 19th passes (`fwd_13` and `fwd_27`) rather than exact ones (the same `u`, but tested on different problems). Our norm-matched controls `randdir` (`+2.3 +/- 1.6`) and `randcols` (`+2.5 +/- 1.6`) match the agent's 23rd pass (`fwd_35`, `+1.56 +/- 3.52`).
+
+### 4. How every answer ended (`results/lab4/endings.json` and `endings-controls.json`)
+
+`labs/lab4_endings.py` sorts the 512 answers in a run into four mutually exclusive groups by whether and how the model closed its `<think>` block:
+
+1. **Cut while thinking:** hit the 32,768-token limit before writing `</think>`. (A cut answer still counts as correct if its last `\boxed{}` holds the right answer, even inside its scratchpad; if it wrote no `\boxed{}`, the grader checks its last `\fbox{}`.)
+2. **Stopped while thinking:** emitted the end-of-sequence token before writing `</think>`, leaving no final answer.
+3. **Closed out of a loop:** wrote `</think>` right after verbatim repetition (the 60-character snippet ending 100 characters before `</think>` appears at least 3 times in the 6,000 characters before `</think>`).
+4. **Closed normally:** wrote `</think>` and the group 3 check found no repeated snippet. That check inspects only the last 6,000 characters before `</think>`, so an answer that recovered from a loop more than 6,000 characters before closing, or wrote `</think>` before its 6,000th character, still counts as "closed normally".
+
+Every cell below lists `answers (correct answers)`:
+
+| Run | Total correct (of 512) | Cut while thinking | Stopped while thinking | Closed out of a loop | Closed normally |
+|---|---|---|---|---|---|
+| `dev/base` (`alpha = 0`) | `62` | `453 (4)` | `0 (0)` | `3 (2)` | `56 (56)` |
+| `dev/a-4` (`alpha = -4`) | `101` | `255 (4)` | `47 (0)` | `197 (87)` | `13 (10)` |
+| `dev/a1` (`alpha = 1`) | `116` | `394 (3)` | `1 (0)` | `31 (29)` | `86 (84)` |
+| `dev/a2` (`alpha = 2`) | `180` | `306 (4)` | `0 (0)` | `70 (44)` | `136 (132)` |
+| `dev/a4` (`alpha = 4`) | `303` | `83 (1)` | `1 (0)` | `172 (70)` | `256 (232)` |
+| `dev/p50c` (`alpha = 6`) | `315` | `72 (0)` | `0 (0)` | `96 (29)` | `344 (286)` |
+| `dev/a8` (`alpha = 8`) | `311` | `51 (0)` | `28 (0)` | `69 (16)` | `364 (295)` |
+| `dev/a12` (`alpha = 12`) | `265` | `43 (0)` | `37 (0)` | `19 (3)` | `413 (262)` |
+| `dev/l15only` | `309` | `81 (1)` | `0 (0)` | `136 (47)` | `295 (261)` |
+| `dev/l19only` | `240` | `196 (2)` | `0 (0)` | `151 (82)` | `165 (156)` |
+| `dev/all1` | `147` | `362 (6)` | `0 (0)` | `30 (25)` | `120 (116)` |
+| `dev/all2` | `253` | `200 (1)` | `0 (0)` | `93 (49)` | `219 (203)` |
+| `dev/randdir` | `74` | `440 (10)` | `0 (0)` | `14 (7)` | `58 (57)` |
+| `dev/randcols` | `75` | `443 (7)` | `0 (0)` | `6 (5)` | `63 (63)` |
+
+### 5. Median length of correct answers and breakdown by problem source
+
+`dev256` is half GSM8K (grade-school word problems, 128 problems, 256 answers) and half MATH (competition math, 128 problems, 256 answers). `results/summary.json` breaks accuracy down by half and records the median token length of correct answers:
+
+| Run | GSM8K accuracy % / cut % | MATH accuracy % / cut % | Median tokens of correct answers |
+|---|---|---|---|
+| `dev/base` (`alpha = 0`) | `15.2 / 86.3` | `9.0 / 91.0` | `5,600.5` |
+| `dev/a-4` (`alpha = -4`) | `23.0 / 55.9` | `16.4 / 75.8` | `12,987` |
+| `dev/p50c` (`alpha = 6`) | `74.6 / 4.7` | `48.4 / 34.8` | `7,998` |
+| `dev/a12` (`alpha = 12`) | `55.5 / 0.8` | `48.0 / 23.8` | `5,799` |
+
+In the unedited model, only short derivations finish before a loop starts, giving a median correct length of `5,600` tokens (`5,600.5`). With `p50c`, longer derivations can finish without looping, raising the median to `7,998` tokens. At `alpha = -4`, most correct answers (`87` of `101`) close right after repeated text. The median length of all correct answers is `12,987` tokens; the endings check does not measure loop length.
+
+## Why it works
+
+### 1. The dose trade-off along `u`: why accuracy plateaus and then drops
+
+In [Lab 0](lab0_toy.md), `W' = W - alpha * u (u^T W_S)` scales the chosen 64 neurons' push along `u` by `1 - alpha`. Sweeping `alpha` from `0` to `12` takes that multiplier from `+1` (no edit) through `0` (erase at `alpha = 1`) and `-1` (mirror at `alpha = 2`) to `-5` (`alpha = 6`) and `-11` (`alpha = 12`).
+
+```mermaid
+xychart-beta
+  title "Accuracy (%) on dev256 across edit strength alpha"
+  x-axis "alpha" ["-4", "0", "1", "2", "4", "6", "8", "12"]
+  y-axis "Accuracy (%)" 0 --> 70
+  line [19.7, 12.1, 22.7, 35.2, 59.2, 61.5, 60.7, 51.8]
+```
+
+`alpha = 1` (erasing the 64 neurons' push along `u`) is not enough. In Lab 3, those 64 neurons held about half the positive loop score in layers 15 and 19 (`49.8%` and `55.5%`), while the other 22 blocks and the attention and DeltaNet mixers also pushed toward `u`. Setting `alpha = 1` zeroes out those 64 neurons' contribution (`22.7%` accuracy) and leaves the rest of the loop drift intact. Setting `alpha > 1` turns them into a counter-push: when they fire, they subtract `(alpha - 1)` times their original loop push, cancelling drift from other neurons and layers.
+
+Pushing `alpha` too high shows two signs of harm to reasoning; the first is clean and the second is partly muddied by which problems finish:
+
+1. **Premature stop tokens during thinking:** From `alpha = 0` to `alpha = 6`, at most `1` of 512 answers emits an end-of-sequence token inside `<think>`. At `alpha = 8`, `28` answers stop mid-thinking with no final answer; at `alpha = 12`, `37` do.
+2. **Lower accuracy among finished answers, and higher perplexity:** Among answers that finish before the cap, accuracy drops as `alpha` grows: `100.0%` (`alpha = 0`), `96.6%` (`alpha = 1`), `92.1%` (`alpha = 2`), `82.9%` (`alpha = 4`), `76.6%` (`alpha = 6`), `70.7%` (`alpha = 8`), and `59.0%` (`alpha = 12`). Among normal `<think>` closes without a detected loop, accuracy falls from `286 / 344` (`83.1%`) at `alpha = 6` to `262 / 413` (`63.4%`) at `alpha = 12`. These finished-answer rates mix reasoning harm with selection: at `alpha = 0` few answers finish, and they may come from easier problems. Higher `alpha` lets harder problems finish too, and some of them are wrong. The `alpha = 6` vs `alpha = 12` normal-close gap has the same issue because the two runs finish different problems. The cleaner signs of harm are the mid-thinking stops in item 1 and WikiText-2 perplexity (Lab 7): small edits (`alpha = 1, 2, 4`) slightly lower perplexity (`-0.49%`, `-0.69%`, `-0.47%`), `alpha = 6` adds `+0.79%`, `alpha = 8` adds `+3.33%`, and `alpha = 12` jumps to `+14.34%`.
+
+Across `alpha = 4` (`59.2%`), `alpha = 6` (`61.5%`), and `alpha = 8` (`60.7%`), fewer cut answers and lower finished-answer accuracy roughly balance out. The released `alpha = 6` sits on that plateau: paired by problem, it leads `alpha = 4` by `+2.3 +/- 1.6` and `alpha = 8` by `+0.8 +/- 1.6` points. Mid-thinking stops and perplexity rise at `alpha = 8` and `12`.
+
+### 2. Why both the direction `u` and the 64 neurons matter
+
+Changing 64 columns by `10.6%` and `8.1%` of the matrix norm could in principle work by adding weight noise that breaks repetition. The controls `randdir` and `randcols` test this with `"norm_match": true` in `wsp/edits.py`, which rescales `alpha` so the Frobenius norm $\|W' - W\|_F$ matches `p50c` at both layers:
+
+- `randdir` edits the same 64 neurons per layer by the same matrix norm along a random unit vector orthogonal to `u`. Accuracy is `14.5 +/- 1.8%` (`+2.3 +/- 1.6` points over `12.1%`, `22` rescued, `18` lost).
+- `randcols` applies `u` with the same matrix norm to 64 random neurons outside the chosen 64. Accuracy is `14.6 +/- 1.9%` (`+2.5 +/- 1.6` points, `23` rescued, `20` lost).
+
+Both controls sit within noise of the unedited model, with `87.3%` and `86.7%` of answers still cut at the cap. Among our 64-neuron edits, the gain appeared only when `u` went into neurons chosen because they fire during loops. Two caveats apply: each control is a single random draw (one random direction, one random set of 64 neurons), and the neurons do not have to be the exact released 64. In [Lab 3](lab3_rebuild.md), rebuilding from only 2 of the agent's 4 text pairs picked neurons that shared `42` and `31` of the released 64 neurons at layers 15 and 19 and still scored `62.3%`. In [Lab 8](lab8_capstone.md), recipe B picked neurons from real loops, shared `42` and `28` of the 64, and scored `58.2%`.
+
+Comparing `all1` and `all2` with the 64-neuron edits shows what the 64-neuron restriction does. Editing all 3,584 neurons at `alpha = 1` (`all1`, `28.7%`) and `alpha = 2` (`all2`, `49.4%`) beats the 64-neuron edits at the same `alpha` (`a1` at `22.7%` and `a2` at `35.2%`), which suggests the other 3,520 neurons also push along `u`. In Lab 7, KL divergence measures how far the edited next-token probabilities move from the base model's. At `alpha = 2`, `all2` shifts predictions on productive texts about `3.5` times as much as `a2` (KL `0.0063` vs `0.0018`). It is also less selective: its KL ratio between looping and productive texts is `1.45` (`0.0092` vs `0.0063`), compared with `2.49` for `p50c` (`0.0241` vs `0.0097`). We did not run the all-neuron edit above `alpha = 2`, so we do not know its peak accuracy, only that the 64-neuron edit concentrates more of its change on looping text.
+
+### 3. Two likely mechanisms: fewer loops and more escapes (and layer 15 vs layer 19)
+
+Compare `dev/base` and `dev/p50c` in the endings table:
+
+```mermaid
+flowchart LR
+  Start["512 answers on dev256"] --> Base["Unedited (dev/base):<br/>453 cut while thinking<br/>3 closed out of a loop (2 correct)<br/>56 closed normally (56 correct)"]
+  Start --> Edit["Edited alpha = 6 (dev/p50c):<br/>72 cut while thinking<br/>96 closed out of a loop (29 correct)<br/>344 closed normally (286 correct)"]
+```
+
+The endings suggest the edit helps in two ways, but do not prove it, because the check inspects only the text right before `</think>`:
+
+1. **Fewer answers end their thinking in a loop:** Normal `<think>` closes jump more than sixfold, from `56` in `dev/base` to `344` in `dev/p50c`. Normal closes are usually right: `56 / 56` (`100%`) in `dev/base` and `286 / 344` (`83.1%`) in `dev/p50c`. The simplest reading is that fewer loops start.
+2. **More answers escape from a loop:** Answers that repeat verbatim and then emit `</think>` ("closed out of a loop") rise from `3` in `dev/base` to `172` at `alpha = 4` and `96` at `alpha = 6`. Loop escapes are noisier than normal closes: in `dev/p50c`, `29 of 96` (`30.2%`) are correct, adding `29` to the `286` normal-close successes (`286 + 29 = 315`).
+
+Comparing `dev/l15only` and `dev/l19only` in the endings table:
+
+- **Layer 15 alone mostly adds normal closes:** `dev/l15only` scores `60.4%` (`309` correct, near `61.5%` for both layers). Normal closes rise from `56` to `295` (`261` correct), cut answers fall from `453` to `81`, and `136` answers close out of a loop (`47` correct).
+- **Layer 19 alone leaves more answers cut and gets more closes from loops:** `dev/l19only` scores `46.9%` (`240` correct), with `196` cut answers, `165` normal closes (`156` correct), and `151` closes out of a loop (`82` correct).
+
+[Lab 6](lab6_steering.md) offers a hint, though not a proof, for why the layers differ. Across 48 real loops, it measures the trigger `s(t)` (how hard the 64 chosen neurons push along `u` on each token). Relative to its in-loop level, layer 15's trigger builds earlier: in the last 512 tokens before the loop it reaches `42%` of its in-loop value (`0.178` of `0.424`), compared with `32%` at layer 19 (`0.241` of `0.758`). Inside the loop, layer 19's trigger is larger (`0.758` vs `0.424`). This fits layer 15 acting earlier and layer 19 pushing harder once a loop starts, though we did not test that directly.
+
+### 4. The `alpha = -4` surprise: why you must read the answers
+
+At `alpha = -4` (`dev/a-4`), the edit multiplies the 64 neurons' push along `u` by `1 - (-4) = +5`, pushing 5 times harder in the loop direction. On headline accuracy alone, `dev/a-4` scores `19.7 +/- 1.9%`, or `+7.6 +/- 2.2` points above the unedited model (`12.1%`).
+
+The endings table and `labs/peek.py` show what happened:
+
+```mermaid
+flowchart TD
+  Neg["alpha = -4 (dev/a-4):<br/>pushes 5x harder toward loops"] --> Collapse["Normal closes collapse:<br/>56 to 13 answers (10 correct)<br/>24 previously solved problems lost"]
+  Neg --> Stop["Mid-thinking stops spike:<br/>0 to 47 answers end with no answer"]
+  Neg --> Crash["Loops intensify and crash out of thinking:<br/>3 to 197 answers close out of a loop<br/>(87 of the 101 correct answers)"]
+```
+
+- **Normal reasoning collapses:** Normal `<think>` closes drop from `56` in `dev/base` to `13` in `dev/a-4` (`10` correct). `dev/a-4` loses `24` problems the base model solved (`54` rescued, `24` lost, while `a4`, `p50c`, `a8`, and `a12` lose `0`).
+- **Mid-thinking stops jump to 47:** `47` answers emit the end token inside `<think>` with no final answer, and WikiText perplexity worsens by `+5.39%`.
+- **Most correct answers close their thinking right after repeated text:** `197` answers close this way; of `101` correct answers in `dev/a-4`, `87` came from `closed out of a loop`.
+
+One problem shows the pattern (`gsm8k_train_5628`: *"Mr. Desmond bought three times as many toys for his younger son as he bought for his elder son. If the elder son received 60 toys, how many toys did Mr Desmond buy?"*, truth `240`):
+
+- **Unedited (`quick/base`, answer 0):** Computes `60 + 180 = 240` early, then enters an 83-token verbatim loop at token `8,568` of `32,768` (*"Let's check if the answer is 240. / Let's check if the answer is 180..."*) that repeats about `292` times to the cap.
+- **Released edit (`quick/p50c`, answer 0):** Finishes in `5,825` tokens, closes `</think>`, and writes `\boxed{240}`.
+- **Alpha -4 (`dev/a-4`, answer 0, on `sampler`):** Run `python labs/peek.py dev/a-4 --id gsm8k_train_5628` on `sampler`. The model repeats *"Wait, is it possible $A$ and $C$ is the same?"* over and over, breaks off mid-line with a stray backtick and `</think>`, and writes `240` (`4,830` tokens total).
+
+Pushing harder in the loop direction cut normal closes from `56` to `13`, lost `24` previously solved problems, and dropped finished-answer accuracy to `55.4%`. Yet `87` correct answers closed their thinking right after repeated text, and total accuracy rose by `+7.6` points.
+
+## Check yourself
+
+1. Look at the dose table between `alpha = 6` (`61.5%` accuracy, `19.7%` cut) and `alpha = 12` (`51.8%` accuracy, `12.3%` cut). Fewer answers are cut at `alpha = 12`, so why is accuracy `9.7` percentage points lower?
+<details><summary>Answer</summary>
+
+Two things go wrong at `alpha = 12`. First, `37` answers emit an end token inside `<think>` (`stopped while thinking`), against `0` at `alpha = 6`, producing no final answer. Second, the `-11` multiplier likely hurts reasoning: WikiText-2 perplexity rises `+14.34%`, and finished-answer accuracy falls from `76.6%` at `alpha = 6` to `59.0%` at `alpha = 12` (normal closes fall from `286 / 344 = 83.1%` to `262 / 413 = 63.4%`), though selection may contribute because the two runs finish different problems.
+</details>
+
+2. Why do `randdir` and `randcols` both use `"norm_match": true` in `configs/variants.json`? What would happen if `randcols` used raw `alpha = 6` without `"norm_match": true`?
+<details><summary>Answer</summary>
+
+In Lab 1, the 64 chosen neurons aligned with `u` much more strongly than typical `down_proj` columns (median `|cos(u, column)|` is `0.095` on the 64 columns in layer 15 vs `0.018` on the rest). Without rescaling, plugging 64 random columns into `W' = W - 6 u (u^T W_S)` gives a Frobenius norm for `W' - W` several times smaller than the released edit's, so the control could fail from a smaller weight change alone. Setting `"norm_match": true` multiplies `alpha` by `||u_rel^T W[:, cols_rel]|| / ||u^T W[:, cols]||` so `W' - W` matches `p50c` in Frobenius norm.
+</details>
+
+3. Compare `dev/l15only` (`60.4%` accuracy, `+0.05%` perplexity change) and `dev/l19only` (`46.9%` accuracy, `+1.65%` perplexity change). Which single layer carries most of the benefit on `dev256`, and how do their answer endings differ?
+<details><summary>Answer</summary>
+
+Layer 15 alone (`60.4%`) nearly matches editing both layers (`61.5%`) with almost no WikiText perplexity change (`+0.05%`). In the endings table, `dev/l15only` mostly adds normal closes (`295` normal, `81` cut while thinking), while `dev/l19only` leaves `196` cut, gets `165` normal closes, and closes out of a loop more often (`151`).
+</details>
+
+4. In `dev/a-4`, `101` of `512` answers are correct (`19.7%`), beating `dev/base` (`62` of `512`, `12.1%`). Give three numbers from the tables that show `dev/a-4` damages the model instead of fixing the loop bug cleanly.
+<details><summary>Answer</summary>
+
+Any three of these confirm the damage:
+1. Normal closes fall from `56` (`56` correct) in `dev/base` to `13` (`10` correct) in `dev/a-4`.
+2. `dev/a-4` loses `24` problems that the unedited model solved (`54 / 24` rescued/lost), whereas `dev/p50c` loses `0` (`125 / 0`).
+3. `47` answers stop dead in the middle of thinking (`stopped while thinking`), compared with `0` in `dev/base` and `0` in `dev/p50c`.
+4. Accuracy among finished answers drops from `100.0%` in `dev/base` (`76.6%` in `dev/p50c`) to `55.4%`, and `87` of the `101` correct answers come from crashing out of a verbatim loop (`closed out of a loop`).
+5. WikiText-2 perplexity rises by `+5.39%` (compared with `+0.79%` for `p50c`).
+</details>
+
+## Try this next
+
+We did not run the experiments below. Write down your prediction first, then test on a 16-problem slice (`--limit 16`, about 3 to 5 minutes on 1 GPU) before running all 256 problems.
+
+1. **Test `alpha = 3` or `alpha = 5` on the dose curve (not run).** Between `alpha = 2` (`35.2%`) and `alpha = 4` (`59.2%`), accuracy jumps `24.0` points. From the project root on this machine, add `"a3"` or `"a5"` to a new file `configs/try_dose.json`:
+   ```json
+   {
+    "a3": {"about": "flip and double the loop component (1 - alpha = -2)",
+           "layers": {"15": {"alpha": 3}, "19": {"alpha": 3}}},
+    "a5": {"about": "between a4 and p50c (1 - alpha = -4)",
+           "layers": {"15": {"alpha": 5}, "19": {"alpha": 5}}}
+   }
+   ```
+   On this machine, create `plans/try_dose_quick.json` in the format of `plans/quick.json` (`--limit 16` runs the first 16 GSM8K problems of `dev256`, 2 answers each = 32 answers):
+   ```json
+   {
+    "about": "Quick 16-problem check of alpha 3 and alpha 5",
+    "runs": [
+     {"name": "quick/a3", "model": "qwen35-a3", "data": "dev256", "n": 2, "shards": 1, "est": 1, "args": ["--limit", "16"]},
+     {"name": "quick/a5", "model": "qwen35-a5", "data": "dev256", "n": 2, "shards": 1, "est": 1, "args": ["--limit", "16"]}
+    ]
+   }
+   ```
+   Sync both files:
+   ```bash
+   scripts/sync_vm.sh sampler
+   ```
+   Pick a free GPU with `nvidia-smi`. Connect to the VM, activate the environment, build both models on CPU, and run the plan (replace `7` with your free GPU):
+   ```bash
+   ssh sampler
+   . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+   CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/try_dose.json
+   python scripts/run_jobs.py plans/try_dose_quick.json --gpus 7
+   ```
+   **What to predict:** In the agent's 20th pass (`fwd_29`), raising `alpha = 2` to `alpha = 3` on all columns added about 5 points (`+34.77` to `+39.84`), about a third of the way to the roughly `+50` points its 64-neuron edit later reached. Our 64-neuron curve climbs faster, from `a2` at `35.2%` to `a4` at `59.2%`, so a fair guess for `a3` on full `dev256` is near the middle, around `45% to 50%`. Predict `a5` on full `dev256` to land around `60% to 61%` (within noise of `a4` at `59.2%` and `p50c` at `61.5%`). On these `16` problems alone, our full-run answers score `a2` at `37.5%` (`12/32` correct) and `a4`, `p50c`, and `a8` at `65.6%` (`21/32` each). With 16 problems, the standard error is roughly 9 points or more, so the quick check gives only a rough placement on the curve.
+
+2. **Edit only a subset of the 64 neurons at `alpha = 6` (not run).** In `wsp/edits.py`, `build_from_spec` accepts a list of integer column indices for `"cols"`. In `results/lab3/rebuild.json`, each layer stores `"cols"` (all 64 chosen neurons) and `"top12"` (the 12 highest-scoring neurons by loop score). From the project root on this machine, write a variant file with 32 of the 64 neurons (`"cols"[:32]`, the 32 lowest-numbered of the 64, since `cols` is sorted by index, not score) or with the top 12 (`"top12"`):
+   ```bash
+   python3 -c '
+   import json
+   r = json.load(open("results/lab3/rebuild.json"))["layers"]
+   spec = {
+    "halfcols-a6": {"about": "alpha 6 on 32 of the 64 neurons per layer",
+                    "layers": {L: {"alpha": 6, "cols": r[L]["cols"][:32]} for L in ("15", "19")}},
+    "top12-a6": {"about": "alpha 6 on the top 12 loop-score neurons per layer",
+                 "layers": {L: {"alpha": 6, "cols": r[L]["top12"]} for L in ("15", "19")}}
+   }
+   json.dump(spec, open("configs/try_subset.json", "w"), indent=1)
+   '
+   ```
+   Copy the quick plan from Try 1 to `plans/try_subset_quick.json`, replacing the two run names with `quick/halfcols-a6` and `quick/top12-a6` and the model names with `qwen35-halfcols-a6` and `qwen35-top12-a6`. Keep `"args": ["--limit", "16"]` and sync:
+   ```bash
+   scripts/sync_vm.sh sampler
+   ```
+   Build on CPU and test on a free GPU (replace `7` as needed):
+   ```bash
+   ssh sampler
+   . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+   CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/try_subset.json
+   python scripts/run_jobs.py plans/try_subset_quick.json --gpus 7
+   ```
+   **What to predict:** We did not measure the counter-push of each subset, so use Lab 3's loop scores as a rough guide. The 32 lowest-numbered columns are not an even half by score. At layer 19 they include the two highest scorers, neurons 1 and 11. Neuron 11 alone holds `24%` of layer 19's positive score, so this half probably carries well over half of layer 19's push. At layer 15 no neuron dominates (the top holds `5.6%`). If that half carried exactly half the push, `alpha = 6` on 32 columns would act like `alpha = 3` or `4` on all 64. Predict roughly `45% to 55%` on full `dev256` with under `+0.79%` WikiText perplexity change, and somewhat higher if layer 19's half carries more. The top 12 neurons hold more positive score than the top 8 (`19.3%` at layer 15, `39.7%` at layer 19) and less than the top 32 (`38.9%`, `49.8%`), compared with `49.8%` and `55.5%` for all 64. That share is large, especially at layer 19, so `top12-a6` could land anywhere from near `a2` (`35.2%`) to near `a4` (`59.2%`) on full `dev256`. The `16`-problem slice is not the full-set comparison: our full-run answers on these problems score `a2` at `37.5%` and `a4`, `p50c`, and `a8` at `65.6%`.

code/labs/lab4_endings.py

@@ -0,0 +1,119 @@
+#!/usr/bin/env python
+"""Lab 4 (also used in Labs 2 and 5): how did each answer end? Reads the full answer texts of finished runs.
+
+    . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+    export CUDA_VISIBLE_DEVICES=""                                # CPU only, a few seconds per run
+    # sampler VM
+    python labs/lab4_endings.py                                   # Lab 4, the alpha sweep -> results/lab4/endings.json
+    python labs/lab4_endings.py dev/base dev/p50c dev/l15only dev/l19only dev/all1 dev/all2 dev/randdir dev/randcols \
+        --out lab4/endings-controls                               # Lab 4, which parts matter
+    python labs/lab4_endings.py dev/base dev/base+temp1.0 dev/base+pp0 dev/base+pp2 dev/base+rep1.1 dev/base+max4k \
+        dev/base+nothink dev/p50c+nothink --out lab5/endings      # Lab 5 (the two thinking-off runs are skipped)
+    python labs/lab4_endings.py dev/base dev/p50c dev/rebuilt --out lab3/endings                     # Lab 3
+    python labs/lab4_endings.py dev/base dev/p50c dev/capA-a6 dev/capB-a6 --out lab8/endings          # Lab 8
+    # trainer VM
+    python labs/lab4_endings.py bench/base/math500 bench/p50c/math500 bench/base/gsm8k bench/p50c/gsm8k \
+        bench/base/ifeval bench/p50c/ifeval --out lab2/endings    # Lab 2
+    python labs/lab4_endings.py dev/rebuilt-pairs2 dev/rebuilt-windows36-36 dev/rebuilt-layers15-19-21 \
+        --out lab3/endings-tries                                  # Lab 3's experiments
+    # either VM
+    python labs/lab4_endings.py dev/a-4 --show 2                  # also print where 2 answers closed their thinking
+
+Without --out, any run list other than the sweep is written to results/lab4/try-<names>/endings.json, so the main
+results stay as they are.
+
+It runs on the VM that made the runs, on the CPU, because the answer texts stay there (scripts/pull_runs.sh copies
+only the per-answer numbers to the laptop). Every answer lands in exactly one of four groups:
+  cut while thinking       it hit the 32,768-token cap before it ever wrote </think>
+  stopped while thinking   the model emitted its end token before it ever wrote </think>, so there is no final answer
+  closed out of a loop     it wrote </think> right after a stretch of verbatim repetition
+  closed normally          it wrote </think> after ordinary reasoning
+"Closed out of a loop" is a rough text check: the 60 characters that end 100 characters before </think> appear
+at least 3 times in the 6,000 characters before </think>. An answer that closed its thinking can still be cut
+later, while it writes the final answer; it stays in its "closed" group.
+"""
+import argparse
+import glob
+import hashlib
+import json
+import os
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+from wsp import hf, loops, protocol  # noqa: E402
+
+SWEEP = ["dev/base", "dev/a-4", "dev/a1", "dev/a2", "dev/a4", "dev/p50c", "dev/a8", "dev/a12"]
+GROUPS = ["cut while thinking", "stopped while thinking", "closed out of a loop", "closed normally"]
+SNIP, GAP, LOOK, MIN_COUNT = 60, 100, 6000, 3
+
+ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ap.add_argument("runs", nargs="*", default=SWEEP, help="run names under runs/ (default: the alpha sweep)")
+ap.add_argument("--show", type=int, default=0, metavar="N",
+                help="print the text around </think> for N correct answers that closed out of a loop, per run")
+ap.add_argument("--out", metavar="PATH",
+                help="write results/PATH.json instead (e.g. lab5/endings)")
+a = ap.parse_args()
+
+
+def closed_out_of_loop(text, i):
+    """i = where </think> starts. True if the text just before it is a verbatim loop (rough check)."""
+    if i < LOOK:
+        return False
+    snip = text[i - GAP - SNIP:i - GAP]
+    return text[i - LOOK:i].count(snip) >= MIN_COUNT
+
+
+def group(sample, text):
+    i = text.find("</think>")
+    if i < 0:
+        return GROUPS[0] if sample["cut"] else GROUPS[1]
+    return GROUPS[2] if closed_out_of_loop(text, i) else GROUPS[3]
+
+
+res = {}
+for run in a.runs:
+    run_dir = os.path.join(hf.LAB, "runs", run)
+    done = sorted(glob.glob(os.path.join(run_dir, "shard*.done.json")))
+    if not done:
+        sys.exit(f"no finished shards in {run_dir}; is this the VM that ran {run}?")
+    if not protocol.PRESETS[json.load(open(done[0]))["preset"]]["thinking"]:
+        print(f"skipping {run}: thinking is off, so its answers have no </think> to look for")
+        continue
+    counts = {g: {"answers": 0, "correct": 0} for g in GROUPS}
+    shown = 0
+    for rec, j, s, text in loops.iter_samples(run_dir):
+        g = group(s, text)
+        counts[g]["answers"] += 1
+        counts[g]["correct"] += bool(s["correct"])
+        if g == GROUPS[2] and s["correct"] and shown < a.show:
+            shown += 1
+            i = text.find("</think>")
+            print(f"--- {run}: {rec['id']} answer {j}, {s['tokens']:,} tokens, closed out of a loop and correct")
+            print(text[i - 600:i + 300].rstrip() + "\n")
+    n = sum(c["answers"] for c in counts.values())
+    if n == 0:
+        sys.exit(f"no answer texts found in {run_dir}; is this the VM that ran {run}?")
+    res[run] = {"answers": n, "correct": sum(c["correct"] for c in counts.values()), "groups": counts}
+
+w = max(len(r) for r in res)
+print(f"{'run':{w}s} answers correct | " + " | ".join(GROUPS) + "   (each: answers, of which correct)")
+for run, r in res.items():
+    cells = [f"{r['groups'][g]['answers']:>{len(g) - 6}d} ({r['groups'][g]['correct']:3d})" for g in GROUPS]
+    print(f"{run:{w}s} {r['answers']:7d} {r['correct']:7d} | " + " | ".join(cells))
+
+if not res:
+    sys.exit("nothing to report")
+if a.out:
+    path = os.path.join(ROOT, "results", a.out.removesuffix(".json") + ".json")
+elif a.runs != SWEEP:
+    name = "-".join(r.replace("/", "_") for r in a.runs)
+    if len(name) > 60:
+        name = name[:50] + "-" + hashlib.md5(name.encode()).hexdigest()[:6]
+    path = os.path.join(ROOT, "results", "lab4", "try-" + name, "endings.json")
+else:
+    path = os.path.join(ROOT, "results", "lab4", "endings.json")
+os.makedirs(os.path.dirname(path), exist_ok=True)
+json.dump({"check": {"snippet": SNIP, "gap": GAP, "look_back": LOOK, "min_count": MIN_COUNT}, "runs": res},
+          open(path, "w"), indent=1)
+print(f"\nwrote {os.path.relpath(path, ROOT)}")

code/labs/lab5_decoding.md

@@ -0,0 +1,320 @@
+# Lab 5: can decoding settings do the same job?
+
+**Big thing:** We compare the 131,072-weight edit against six decoding changes on unedited Qwen3.5-0.8B (temperature `1.0`, presence penalty `0` and `2.0`, repetition penalty `1.1`, a shorter `4,096`-token cap, and `<think>` mode off) across all 512 `dev256` answers. Penalties and temperature raise accuracy at most from `12.1%` to `34.4%`, turning thinking off reaches `52.1%` in 13 times fewer tokens than the unedited thinking baseline, and the weight edit with thinking on reaches `61.5%` (`74.6%` on the GSM8K half).
+
+**Why it matters:** Always test whether a weight edit beats simpler decoding changes that leave the weights alone. Ending counts suggest penalties mostly add normal closes, while the weight edit adds both normal closes and loop escapes. These counts do not prove which loops were prevented, because the check reads only the text right before `</think>`.
+
+**Time and hardware:** Already run on `sampler` (`results/summary.md`, `results/lab5/endings.json`, and `results/figures/lab5_decoding.png`). Re-running one unedited `dev256` job on 1 NVIDIA A100 GPU takes `34` to `49` minutes with thinking on at the `32,768`-token cap, or about `3` minutes with thinking off or a `4,096`-token cap.
+
+**Theory guide:** Section 13.2.
+
+## Before you run: predict
+
+Write down a guess for each question before looking at the table.
+
+1. The paper's baseline (`"think35"`) uses `presence_penalty = 1.5`, which subtracts `1.5` from the score of any token that has already appeared in the answer. If you turn that penalty off (`presence_penalty = 0`), what happens to the share of answers cut at the 32,768-token cap (now `88.7%`)?
+2. The unedited model's correct `dev256` answers have a median length of `5,600` tokens, while looping answers run to `32,768` tokens. What happens to accuracy if you lower the token cap to `4,096` tokens?
+3. What if you turn `<think>` mode off (`enable_thinking=False`), so the model writes a direct step-by-step solution instead of an open-ended internal scratchpad? Will it beat the unedited thinking model (`12.1%`), and will the weight edit (`p50c`) still help when thinking is off?
+
+## Run it
+
+All seven decoding runs are at the bottom of `plans/sampler.json` and ran on `sampler` alongside the Lab 4 variants (`dev/base` and `dev/p50c` at the top of `plans/sampler.json` are the baselines).
+
+Open `plans/sampler.json` and check the last seven entries:
+
+```json
+  {"name": "dev/base+temp1.0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["temperature=1.0"]},
+  {"name": "dev/base+pp0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=0"]},
+  {"name": "dev/base+pp2", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=2.0"]},
+  {"name": "dev/base+rep1.1", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 14, "set": ["repetition_penalty=1.1"]},
+  {"name": "dev/base+max4k", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "set": ["max_tokens=4096"]},
+  {"name": "dev/base+nothink", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"},
+  {"name": "dev/p50c+nothink", "model": "qwen35-p50c", "data": "dev256", "n": 2, "shards": 1, "est": 1, "preset": "nothink35"}
+```
+
+Each entry starts from a preset in `wsp/protocol.py`:
+
+- `"think35"` (the default when `"preset"` is omitted): `temperature = 0.6`, `top_p = 0.95`, `top_k = 20`, `presence_penalty = 1.5`, `repetition_penalty = 1.0`, `max_tokens = 32768`, `thinking = True`.
+- `"nothink35"` (Qwen's model-card settings for non-thinking mode): `temperature = 0.7`, `top_p = 0.8`, `top_k = 20`, `presence_penalty = 1.5`, `repetition_penalty = 1.0`, `max_tokens = 8192`, `thinking = False`.
+- A `"set"` list overrides fields in the preset (`scripts/run_jobs.py` passes each `"key=value"` string to `scripts/gen_worker.py` as `--set`).
+
+These runs finished on `sampler` and are in `results/runs/` on this laptop. On 1 A100 GPU, a full `dev256` run takes `17` to `49` minutes (`33.8` to `48.9` minutes for the long unedited runs; `dev/p50c` took `20.6` minutes), while `max_tokens=4096` and `nothink35` take about `3` minutes (`2.7` to `3.2` minutes).
+
+To inspect how answers ended or read single answers on `sampler` (every VM command starts with `. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code`):
+
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+export CUDA_VISIBLE_DEVICES=""
+
+# Classify how every answer ended across the Lab 5 runs (writes results/lab5/endings.json)
+python labs/lab4_endings.py \
+  dev/base dev/base+temp1.0 dev/base+pp0 dev/base+pp2 dev/base+rep1.1 dev/base+max4k \
+  dev/base+nothink dev/p50c+nothink \
+  --out lab5/endings
+
+# List or read individual answers from any decoding run on sampler
+python labs/peek.py dev/base+rep1.1 --id gsm8k_train_5628
+python labs/peek.py dev/base+nothink --id gsm8k_train_5628
+```
+
+`labs/lab4_endings.py` skips `dev/base+nothink` and `dev/p50c+nothink` with a message (`thinking is off, so its answers have no </think> to look for`).
+
+To redraw `results/figures/lab5_decoding.png` on this laptop:
+
+```bash
+# On the local laptop, inside ~/weight-space-programming/code
+python3 scripts/summarize.py results/runs
+uv run -q --with numpy --with matplotlib python scripts/plot_runs.py
+```
+
+## What you should see
+
+### 1. All nine runs on `dev256` (`512` answers per run)
+
+Every run uses the same 256 unseen practice problems (128 GSM8K-train + 128 MATH-train, 2 answers each = 512 answers). Paired gain compares each run against `dev/base`, except `dev/p50c+nothink`, which `scripts/summarize.py` pairs against `dev/base+nothink` to isolate the weight edit when thinking is off:
+
+| Run | Setting | Accuracy % | Cut at the cap % | Mean tokens | Correct when finished % | Cut answers ending in repetition % | Paired gain vs baseline (pp) | Rescued / lost problems |
+|---|---|---|---|---|---|---|---|---|
+| `dev/base` | Unedited, paper settings (`think35`) | `12.1 +/- 1.6` | `88.7` | `29,683` | `100.0` | `96.9` | `-` | `-` |
+| `dev/base+temp1.0` | Temperature `1.0` (from `0.6`) | `27.9 +/- 2.5` | `71.3` | `27,430` | `95.9` | `59.5` | `+15.8 +/- 2.1` | `52 / 9` |
+| `dev/base+pp0` | Presence penalty `0` (from `1.5`) | `4.5 +/- 0.9` | `99.0` | `32,472` | `100.0` | `98.4` | `-7.6 +/- 1.6` | `11 / 42` |
+| `dev/base+pp2` | Presence penalty `2.0` (from `1.5`) | `16.2 +/- 1.9` | `84.6` | `28,614` | `98.7` | `96.8` | `+4.1 +/- 1.7` | `24 / 13` |
+| `dev/base+rep1.1` | Repetition penalty `1.1` (from `1.0`) | `34.4 +/- 2.6` | `61.7` | `23,301` | `89.8` | `88.3` | `+22.3 +/- 2.2` | `62 / 3` |
+| `dev/base+max4k` | Cap at `4,096` tokens (from `32,768`) | `5.9 +/- 1.1` | `95.5` | `4,019` | `100.0` | `18.2` | `-6.2 +/- 1.2` | `3 / 30` |
+| `dev/base+nothink` | Thinking off, cap `8,192` (`nothink35`) | `52.1 +/- 2.7` | `18.0` | `2,270` | `63.6` | `7.6` | `+40.0 +/- 2.5` | `115 / 2` |
+| `dev/p50c` | Edited (`p50c`), paper settings | `61.5 +/- 2.8` | `19.7` | `16,104` | `76.6` | `24.8` | `+49.4 +/- 2.6` | `125 / 0` |
+| `dev/p50c+nothink` | Edited (`p50c`), thinking off | `53.5 +/- 2.8` | `16.4` | `2,306` | `64.0` | `0.0` | `+1.4 +/- 1.9` (vs `base+nothink`) | `22 / 26` |
+
+![Three bar charts on the same 256 practice problems: accuracy, percent of answers cut at the token cap, and mean answer length, for the unedited model with six decoding changes and for the edited model with thinking on and off.](../results/figures/lab5_decoding.png)
+
+### 2. Breakdown by problem source (GSM8K vs MATH) and median length of correct answers
+
+Splitting `dev256` into its 128 GSM8K problems and 128 harder MATH problems (`results/summary.json`) shows where the tokens go and where thinking mode pays off:
+
+| Run | GSM8K accuracy % / cut % | MATH accuracy % / cut % | Median length of correct answers (tokens) |
+|---|---|---|---|
+| `dev/base` (unedited, thinking on) | `15.2` / `86.3` | `9.0` / `91.0` | `5,600` (`5,600.5`) |
+| `dev/base+temp1.0` | `35.9` / `62.1` | `19.9` / `80.5` | `12,548` |
+| `dev/base+pp0` (presence penalty `0`) | `6.2` / `98.4` | `2.7` / `99.6` | `32,768` |
+| `dev/base+pp2` (presence penalty `2.0`) | `20.7` / `80.1` | `11.7` / `89.1` | `5,353` |
+| `dev/base+rep1.1` (repetition penalty `1.1`) | `46.5` / `48.8` | `22.3` / `74.6` | `5,700.5` |
+| `dev/base+max4k` (cap `4,096`) | `7.4` / `94.1` | `4.3` / `96.9` | `3,307` |
+| `dev/base+nothink` (thinking off) | `58.2` / `4.3` | `46.1` / `31.6` | `452` |
+| `dev/p50c` (edited, thinking on) | `74.6` / `4.7` | `48.4` / `34.8` | `7,998` |
+| `dev/p50c+nothink` (edited, thinking off) | `62.1` / `2.0` | `44.9` / `30.9` | `480` (`479.5`) |
+
+In `dev/base+pp0`, removing the presence penalty sends `99.0%` of answers (`98.4%` on GSM8K, `99.6%` on MATH) to the `32,768`-token cap. The median length of its `23` correct answers is `32,768` tokens because `18` of those `23` were cut while thinking after boxing an answer mid-scratchpad. In `dev/base+max4k`, because the unedited model's median correct answer is `5,600` tokens, a `4,096`-token cap cuts off valid reasoning before completion and drops accuracy from `12.1%` to `5.9%` (`30` problems lost, `3` rescued).
+
+### 3. How every thinking-mode answer ended (`results/lab5/endings.json`)
+
+Ending counts from `results/lab5/endings.json` (plus `dev/p50c` from `results/lab4/endings.json`), with correct answers in parentheses:
+
+| Run (`512` answers each) | Total correct | Cut while thinking (correct) | Stopped while thinking (correct) | Closed out of a loop (correct) | Closed normally (correct) |
+|---|---|---|---|---|---|
+| `dev/base` | `62` | `453 (4)` | `0 (0)` | `3 (2)` | `56 (56)` |
+| `dev/base+temp1.0` | `143` | `352 (2)` | `0 (0)` | `3 (1)` | `157 (140)` |
+| `dev/base+pp0` | `23` | `507 (18)` | `0 (0)` | `1 (1)` | `4 (4)` |
+| `dev/base+pp2` | `83` | `433 (5)` | `0 (0)` | `4 (4)` | `75 (74)` |
+| `dev/base+rep1.1` | `176` | `294 (0)` | `0 (0)` | `28 (11)` | `190 (165)` |
+| `dev/base+max4k` | `30` | `484 (7)` | `0 (0)` | `0 (0)` | `28 (23)` |
+| `dev/p50c` (weight edit) | `315` | `72 (0)` | `0 (0)` | `96 (29)` | `344 (286)` |
+
+Compare the last two columns (`closed out of a loop` and `closed normally`):
+
+- Raising temperature to `1.0` increases `closed normally` from `56` to `157`, while `closed out of a loop` stays at `3`.
+- Raising presence penalty to `2.0` gives `4` loop escapes; adding repetition penalty `1.1` gives `28` loop escapes and `190` normal closes.
+- The weight edit (`dev/p50c`) produces `344` normal closes and `96` loop escapes. These endings suggest decoding tweaks mostly add normal closes, while the weight edit adds both. A normal close can still follow an earlier loop, because the check reads only the last `6,000` characters before `</think>`.
+
+## Why it works
+
+### 1. Decoding from first principles: how vLLM turns logits into a chosen token
+
+At each generation step, the language model outputs a vector of raw scores called logits, one number $z_v$ for every token $v$ in the vocabulary. Without decoding tweaks, the softmax function turns those logits into probabilities by exponentiating each score and dividing by the sum:
+
+$$p_v = \frac{e^{z_v}}{\sum_{w} e^{z_w}}$$
+
+Because $e^{z_v}$ grows exponentially, adding $+3$ to a token's logit multiplies its relative odds by $e^3 \approx 20$.
+
+In vLLM `0.33.0`, sampling follows the four steps below. The helper `pick_prob` in `wsp/protocol.py` copies this order for presence penalty, temperature, top-k, and top-p, and rejects non-default repetition penalties.
+
+```mermaid
+flowchart LR
+  L["Raw logits z_v<br/>from the model's final layer"] --> Pen["1. Penalties:<br/>repetition penalty (/r or *r),<br/>then presence penalty (-p)"]
+  Pen --> Temp["2. Temperature:<br/>divide all logits by T<br/>(T < 1 sharpens, T > 1 flattens)"]
+  Temp --> Filter["3. Top-k and Top-p:<br/>keep top 20 tokens, then keep<br/>smallest set with >= 95% mass"]
+  Filter --> Pick["4. Sample one token<br/>from the renormalized<br/>probabilities"]
+```
+
+Each step transforms a token's score:
+
+1. **Penalties (`repetition_penalty` and `presence_penalty`):**
+   - **Repetition penalty $r$ (`1.1` in `dev/base+rep1.1`):** For every token $v$ that has appeared anywhere in the prompt or generated answer so far, divide $z_v$ by $r$ ($z_v \leftarrow z_v / r$) if $z_v$ is positive, or multiply by $r$ ($z_v \leftarrow z_v \times r$) if negative. Both push the logit down in proportion to its size.
+   - **Presence penalty $p$ (`1.5` in `"think35"`):** For every token $v$ that has appeared in the generated answer so far (ignoring the prompt), subtract $p$ ($z_v \leftarrow z_v - p$). This penalty is binary: a token used once and a token used 200 times both lose `1.5`.
+2. **Temperature $T$ (`0.6` in `"think35"`, `1.0` in `dev/base+temp1.0`):** Divide every logit by $T$ ($z_v \leftarrow z_v / T$). Dividing by $0.6$ multiplies logit gaps by $1 / 0.6 = 1.67$, favoring the top token. Setting $T = 1.0$ leaves gaps unstretched, giving runner-up tokens a larger chance.
+3. **Top-k (`20`) and Top-p (`0.95`):** Keep the `20` highest-logit tokens, compute their softmax probabilities, sort them, and drop the lowest-probability tokens whose cumulative probability is at most $1 - 0.95 = 0.05$ (always keeping the top token).
+4. **Renormalize and sample:** Divide the kept probabilities by their sum so they add to `1.0`, then pick one token at random.
+
+### 2. A worked example: why token penalties fail to break a strong loop
+
+Suppose the model is inside a loop and about to repeat `"Let's"` (using made-up numbers):
+
+- `"Let's"` has raw logit $z_{\text{loop}} = 12.0$, and `"</think>"` (not yet in the answer) has raw logit $z_{\text{close}} = 5.0$.
+- `"Let's"` and the other scratchpad words have already appeared in the answer.
+
+| Setting | Logit of `"Let's"` after penalties | Logit of `"</think>"` after penalties | Gap after dividing by temperature $T$ | Odds ratio $e^{\text{gap}}$ (`"Let's"` vs `"</think>"`) |
+|---|---|---|---|---|
+| `presence_penalty = 0`, $T = 0.6$ (`dev/base+pp0`) | $12.0$ | $5.0$ | $(12.0 - 5.0) / 0.6 = 11.67$ | $e^{11.67} \approx 117,000 : 1$ |
+| Paper settings: `presence_penalty = 1.5`, $T = 0.6$ (`dev/base`) | $12.0 - 1.5 = 10.5$ | $5.0$ | $(10.5 - 5.0) / 0.6 = 9.17$ | $e^{9.17} \approx 9,600 : 1$ |
+| `presence_penalty = 2.0`, $T = 0.6$ (`dev/base+pp2`) | $12.0 - 2.0 = 10.0$ | $5.0$ | $(10.0 - 5.0) / 0.6 = 8.33$ | $e^{8.33} \approx 4,150 : 1$ |
+| `presence_penalty = 1.5`, $T = 1.0$ (`dev/base+temp1.0`) | $12.0 - 1.5 = 10.5$ | $5.0$ | $(10.5 - 5.0) / 1.0 = 5.50$ | $e^{5.50} \approx 245 : 1$ |
+| `presence_penalty = 1.5`, `repetition_penalty = 1.1`, $T = 0.6$ (`dev/base+rep1.1`) | $(12.0 / 1.1) - 1.5 = 9.41$ | $5.0$ | $(9.41 - 5.0) / 0.6 = 7.35$ | $e^{7.35} \approx 1,550 : 1$ |
+
+This arithmetic explains three patterns in the data:
+
+1. **Why presence penalty does not grow as the loop repeats:** `"Let's"` appeared in the first paragraph, so it pays `-1.5` on cycle 1 and the same `-1.5` on cycle 200. Normal math words (`"Therefore"`, `"answer"`, `"total"`, `"240"`) also appeared earlier in the scratchpad, so each pays `-1.5` too, leaving the gap between `"Let's"` and `"Therefore"` unchanged.
+2. **Why `repetition_penalty = 1.1` does better (`34.4%`) than `presence_penalty = 2.0` (`16.2%`):** Dividing a positive logit by `1.1` takes more from a large logit ($12.0 \to 10.91$, losing $1.09$ on top of the $1.5$ presence penalty) than from a moderate runner-up ($8.0 \to 7.27$, losing $0.73$). Even so, `61.7%` of `dev/base+rep1.1` answers hit the cap.
+3. **How penalties might prevent some loops yet fail to break others:** When looping and productive continuations have similar logits, higher temperature (`1.0`) or `repetition_penalty = 1.1` can tip the choice away from the first repeat. Once repeated cycles fill the context, loop-token logits can grow high enough that `top_p = 0.95` drops every non-loop token to probability zero. Lab 6 Part 3 measures the per-token pick probability inside 48 real loops: `0.99995` in the unedited model vs `0.979` with the weight edit.
+
+### 3. Why turning `<think>` off scores 52.1%, and why the edit adds nothing there
+
+Switching from `"think35"` to `"nothink35"` (`dev/base+nothink`) raises `dev256` accuracy from `12.1%` to `52.1%` and cuts mean length from `29,683` tokens to `2,270` (median length on correct answers is `452` tokens).
+
+```mermaid
+flowchart TD
+  Q["Math problem in dev256"] --> TOn["Thinking ON (think35 preset)<br/>Open-ended self-checking scratchpad"]
+  Q --> TOff["Thinking OFF (nothink35 preset)<br/>Direct step-by-step solution"]
+  TOn --> BaseOn["Unedited (dev/base):<br/>88.7% cut (96.9% of those in repetition)<br/>12.1% acc (GSM8K 15.2%, MATH 9.0%)"]
+  TOn --> EditOn["Edited (dev/p50c):<br/>loops suppressed -> 61.5% acc<br/>(GSM8K 74.6%, MATH 48.4%)"]
+  TOff --> BaseOff["Unedited (dev/base+nothink):<br/>few loops (7.6% of cuts) -> 52.1% acc<br/>(GSM8K 58.2%, MATH 46.1%)"]
+  TOff --> EditOff["Edited (dev/p50c+nothink):<br/>+1.4 +/- 1.9 pp -> 53.5% acc<br/>(GSM8K 62.1%, MATH 44.9%)"]
+```
+
+1. **The weight edit helps only when loops occur.** With thinking off (`dev/base+nothink`), `18.0%` of answers hit the 8,192-token cap, and `7.6%` of those cut answers end in repetition (`5.4%` in a verbatim loop). Few answers loop, so with the weight edit and thinking off (`dev/p50c+nothink`) the paired difference against `dev/base+nothink` is `+1.4 +/- 1.9` percentage points (`22` problems rescued, `26` lost), which is indistinguishable from zero.
+2. **Once loops are repaired, the thinking preset beats the non-thinking preset, mostly on word problems.** Holding the edited weights fixed, `dev/p50c` (`think35`, `61.5%`) beats `dev/p50c+nothink` (`nothink35`, `53.5%`) by `+8.0 +/- 2.0` paired points (`27` problems rescued, `11` lost). Most of that gain comes from GSM8K (`74.6%` vs `62.1%`, `+12.5 +/- 2.8` points), while the MATH gap is small (`48.4%` vs `44.9%`, `+3.5 +/- 2.9` points). The thinking preset uses about seven times as many tokens (`16,104` vs `2,306` on average). The two presets also differ in sampling settings (`think35` uses temperature `0.6`, top-p `0.95`, and a `32,768`-token cap; `nothink35` uses temperature `0.7`, top-p `0.8`, and an `8,192`-token cap), so part of the gap could come from sampling changes instead of thinking itself.
+
+### 4. The paper's Gemma 3 270M follow-up points the same way
+
+In Theory Guide section 10 (and `external/lm_understand-logs/exp8_math/gemma_followup/README.md`), the authors ran the same weight-space programming setup on a different small model, Gemma 3 270M. On MATH-500, unedited Gemma 3 270M scored `6.8%` and hit the token cap on `8.5%` of answers (`91.5%` finished, yet only `6.8%` of all answers were right).
+
+Across multiple agent runs, no weight edit moved Gemma 3 270M's MATH-500 score outside `6.8%` to `7.7%`. As the blog puts it: *"Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."* That matches `dev/p50c+nothink`: when answers do not get trapped in loops, subtracting the loop direction `u` does not make the model better at math.
+
+## Check yourself
+
+1. Why did removing the presence penalty (`dev/base+pp0`) hurt accuracy (`12.1%` down to `4.5%`), whereas raising it from `1.5` to `2.0` (`dev/base+pp2`) barely helped (`12.1%` to `16.2%`)?
+<details><summary>Answer</summary>
+
+At `presence_penalty = 0`, no penalty discourages reusing earlier output words: `99.0%` of answers hit the cap (`98.4%` of those end in repetition), and only `4` of `512` close normally. With presence penalty `1.5`, already-used output tokens get a `-1.5` logit penalty, and `56` answers close normally (`62` correct in all, `12.1%` accuracy). Raising the flat penalty from `1.5` to `2.0` adds `-0.5` more, which lowers already-used looping tokens and already-used reasoning tokens by the same amount.
+</details>
+
+2. In `results/summary.md`, `dev/base+nothink` shows a `+40.0 +/- 2.5` percentage-point gain, while `dev/p50c+nothink` shows `+1.4 +/- 1.9` percentage points even though its accuracy (`53.5%`) exceeds `dev/base+nothink` (`52.1%`). Why do the two gains differ?
+<details><summary>Answer</summary>
+
+`scripts/summarize.py` pairs each run against the baseline that isolates one change (`baseline(name)` in `scripts/summarize.py`). `dev/base+nothink` uses `dev/base` (`52.1% - 12.1% = +40.0` points, the effect of turning thinking off on the unedited model). `dev/p50c+nothink` changes both the model and the preset, so `scripts/summarize.py` pairs it against `dev/base+nothink` (`53.5% - 52.1% = +1.4` points, the effect of the weight edit when thinking is already off).
+</details>
+
+3. If a teammate suggests saving compute on the unedited thinking model by lowering `max_tokens` from `32,768` to `4,096` (`dev/base+max4k`), what do the numbers show?
+<details><summary>Answer</summary>
+
+Mean tokens drop by a factor of 7.4 (`29,683` to `4,019`), and accuracy falls in half (`12.1%` to `5.9%`, losing `30` problems and rescuing `3`). Because the unedited model's correct thinking answers have a `5,600`-token median length, a `4,096`-token cap cuts off over half of the answers that would have finished and solved the problem. For short answers without weight edits, turning `<think>` off (`dev/base+nothink`: `2,270` mean tokens, `52.1%` accuracy) works much better than truncating `<think>` at `4,096` tokens.
+</details>
+
+4. In vLLM's sampler (`wsp/protocol.py`), does `presence_penalty` get applied before or after dividing by `temperature`? Why does that order matter when `temperature = 0.6`?
+<details><summary>Answer</summary>
+
+`presence_penalty` is subtracted before dividing by `temperature`: `z = (logits - presence_penalty * present) / temperature`. Because `temperature = 0.6` divides the penalized logits by `0.6`, a presence penalty of `1.5` subtracts `1.5 / 0.6 = 2.5` from the post-temperature logit of every token already in the answer. When `dev/base+temp1.0` raises temperature from `0.6` to `1.0`, that effective penalty drops from `2.5` to `1.5`, and every post-temperature logit gap becomes `0.6` times as large for the same raw logits and token history. The raw logits are unchanged.
+</details>
+
+## Try this next
+
+1. **Compare how `gsm8k_train_5628` ends across four decoding settings and the edited `dev/p50c` on `sampler`.** Run `labs/peek.py` (CPU only, a few seconds) on the Mr. Desmond problem (`truth: 240`) across `dev/base`, `dev/base+temp1.0`, `dev/base+rep1.1`, `dev/base+nothink`, and `dev/p50c`:
+   ```bash
+   ssh sampler
+   . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+   export CUDA_VISIBLE_DEVICES=""
+   for r in dev/base dev/base+temp1.0 dev/base+rep1.1 dev/base+nothink dev/p50c; do
+     echo "=== $r ==="
+     python labs/peek.py "$r" --id gsm8k_train_5628 --chars 600
+   done
+   ```
+   Compare the length of `dev/base+nothink` against `dev/p50c`, and check whether `dev/base+rep1.1` solved `gsm8k_train_5628` or looped.
+2. **Test a new decoding combination on 16 problems (not run by us; write down your prediction first).** Combining the two best decoding tweaks on the unedited model (`temperature = 1.0` and `repetition_penalty = 1.1`), or combining the weight edit (`qwen35-p50c`) with `repetition_penalty = 1.1`, was not in `plans/sampler.json`. You can test both in about 5 minutes on 2 GPUs of `sampler` using `"args": ["--limit", "16"]` (as in `plans/quick.json`) and a `"try/"` prefix so `runs/dev/` stays untouched:
+   ```json
+   {
+    "about": "Quick 16-problem test of combined decoding settings.",
+    "runs": [
+     {
+      "name": "try/base+temp1.0+rep1.1",
+      "model": "base",
+      "data": "dev256",
+      "n": 2,
+      "shards": 1,
+      "est": 1,
+      "set": ["temperature=1.0", "repetition_penalty=1.1"],
+      "args": ["--limit", "16"]
+     },
+     {
+      "name": "try/p50c+rep1.1",
+      "model": "qwen35-p50c",
+      "data": "dev256",
+      "n": 2,
+      "shards": 1,
+      "est": 1,
+      "set": ["repetition_penalty=1.1"],
+      "args": ["--limit", "16"]
+     }
+    ]
+   }
+   ```
+   Save this JSON on this machine as `plans/try_lab5.json`. From the project root on this machine, sync the plan, run it on `sampler`, and pull the results:
+   ```bash
+   scripts/sync_vm.sh sampler
+   ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5.json --gpus 0,1' </dev/null
+   scripts/pull_runs.sh sampler
+   ```
+   The single shard runs the first `16` rows of `dev256` (all GSM8K problems, `2` answers each). The `try/` prefix gets no paired column from `scripts/summarize.py`, so compare the shared problem IDs directly on this machine:
+   ```bash
+   python3 - <<'PY'
+   from wsp.summary import compare, load_run
+   for name in ("base+temp1.0+rep1.1", "p50c+rep1.1"):
+       trial = load_run("results/runs/try/" + name)
+       for baseline in ("base", "p50c"):
+           print(name, "vs dev/" + baseline,
+                 compare(load_run("results/runs/dev/" + baseline), trial))
+   PY
+   ```
+   **What we predict:** Combining temperature `1.0` and repetition penalty `1.1` may reduce repetition further, though a gain is not guaranteed. On these `16` problems, our separate runs score `40.6%` (`13/32` answers) at temperature `1.0`, `46.9%` (`15/32`) with repetition penalty `1.1`, and `65.6%` (`21/32`) with `p50c`, with standard errors across problems of about `11` to `12` percentage points, so predictions from this small check are rough. Adding repetition penalty to `p50c` might reduce cuts, but it also penalizes repeated numbers and variables in valid solutions. Because this slice has no MATH problems, it cannot test hard MATH answers.
+3. **Test `nothink35` on the three full benchmarks (not run by us; write down your prediction first).** We ran `nothink35` on `dev256` (`52.1%` unedited, `53.5%` edited), not on `math500`, `gsm8k`, or `ifeval`. Scaling from `dev/base+nothink` (`512` answers in `2.73` minutes on `1` GPU) by answer count gives a rough wall-clock estimate of `6` to `15` minutes per benchmark with one shard on `1` GPU (`1,082` IFEval, `2,000` MATH-500, or `2,638` GSM8K answers); answer lengths and throughput may differ. Save this plan on this machine as `plans/try_lab5_nothink.json` (matching the benchmark files and answer counts of the `bench/` runs in `plans/trainer.json`):
+   ```json
+   {
+    "about": "Thinking off (nothink35) on the three full benchmarks, unedited and edited.",
+    "runs": [
+     {"name": "try/nothink/base/math500", "model": "base", "data": "math500", "n": 4, "shards": 1, "est": 5, "preset": "nothink35"},
+     {"name": "try/nothink/base/gsm8k", "model": "base", "data": "gsm8k", "n": 2, "shards": 1, "est": 6, "preset": "nothink35"},
+     {"name": "try/nothink/base/ifeval", "model": "base", "data": "ifeval", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"},
+     {"name": "try/nothink/p50c/math500", "model": "qwen35-p50c", "data": "math500", "n": 4, "shards": 1, "est": 5, "preset": "nothink35"},
+     {"name": "try/nothink/p50c/gsm8k", "model": "qwen35-p50c", "data": "gsm8k", "n": 2, "shards": 1, "est": 6, "preset": "nothink35"},
+     {"name": "try/nothink/p50c/ifeval", "model": "qwen35-p50c", "data": "ifeval", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"}
+    ]
+   }
+   ```
+   Sync, run on six free GPUs (replace `0-5` with the free GPU indices), and pull the results:
+   ```bash
+   scripts/sync_vm.sh sampler
+   ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5_nothink.json --gpus 0-5' </dev/null
+   scripts/pull_runs.sh sampler
+   ```
+   Print each score and the paired gain of the edit on this machine:
+   ```bash
+   python3 - <<'PY'
+   import json
+   from wsp.summary import compare, load_run
+   for d in ("gsm8k", "math500", "ifeval"):
+       for m in ("base", "p50c"):
+           s = json.load(open(f"results/runs/try/nothink/{m}/{d}/summary.json"))
+           print(f"{m} {d}: {100 * s['acc']:.1f} +/- {100 * s['acc_se']:.1f}%")
+       c = compare(load_run(f"results/runs/try/nothink/base/{d}"), load_run(f"results/runs/try/nothink/p50c/{d}"))
+       print(f"  p50c minus base, paired: {100 * c['gain']:+.1f} +/- {100 * c['gain_se']:.1f} points")
+   PY
+   ```
+   **What we predict:** On GSM8K test and MATH-500, `base` with `preset: "nothink35"` should mirror the `dev256` split: about `55%` to `60%` on GSM8K (above `14.2%` unedited thinking, below `66.9%` edited thinking) and `42%` to `46%` on MATH-500 (near edited thinking's `51.8%`). On IFEval, which tests formatting constraints instead of long math derivations, `base` with `nothink35` should approach `bench/p50c/ifeval` (`63.4%`), and `p50c` with `nothink35` should match `base` with `nothink35` within noise.

code/labs/lab6_steering.md

@@ -0,0 +1,302 @@
+# Lab 6: The weight edit is steering in disguise
+
+**Big thing:** Changing the 64 chosen columns of `down_proj` in the weight file gives the same outputs, up to rounding, as leaving the weights alone and subtracting $\alpha \cdot s(t) \cdot u$ from that layer's output during the forward pass. The single number $s(t)$ stays small while the model reasons well (about `0.1` at layer 19 on answers that finished correctly) and grows several times larger once it repeats itself (about `0.76` inside real loops). A fixed weight edit therefore acts like steering whose strength the text sets on each token.
+
+**Why it matters:** Weight editing changes the numbers stored in a model, while activation steering adds a vector to internal activations as the model runs. This edit is both at once: a steering vector $u$, scaled on every token by how hard 64 neurons push along it. That explains how changing 131,072 stored numbers pushes hard against loops and much less on ordinary reasoning, though it still pushes a little.
+
+**Time and hardware:** On one A100 GPU, parts 1 and 2 take about 1 minute, part 3 takes about 2 minutes, and part 4 takes under a minute. Every part runs on `sampler` or `trainer`, which both hold the base model's practice answers (`runs/build/base`) for parts 3 and 4. Redrawing the part 3 figures (`--plot-only`) runs on CPU in a few seconds.
+
+**Theory guide:** Section 12, part D (weight editing versus activation steering).
+
+## Before you run: predict
+
+Write down a short guess for each question before running the script.
+
+1. Load Qwen3.5-0.8B in 32-bit floats (`fp32`) and run the same texts two ways: by editing the 64 columns of `down_proj` in memory, and by leaving the weights untouched while a hook subtracts $\alpha \cdot s(t) \cdot u$ from `down_proj`'s output. Will the final next-token scores (logits) match to 16 decimal places, to about 2 decimal places, or only roughly?
+2. On one MLP matrix (`layer 15`) on CPU in 64-bit floats (`fp64`), how big is the maximum difference between the edited matrix output and the hook output?
+3. On the agent's 4 productive math texts versus its 4 looping math texts, is $s(t)$ (how hard the 64 neurons push along the loop direction $u$) large on both and canceled by later layers, or near zero on productive text and positive on looping text?
+
+## Run it
+
+Connect to `sampler` or `trainer`, since every part runs on either one. Then use `nvidia-smi` to pick a free GPU index and pass it in `CUDA_VISIBLE_DEVICES`.
+
+The script reads three files from the repository: `results/edit_info.json` (the released edit's $u$, $\alpha$ and 64 columns, from `scripts/build_models.py`), `results/lab3/rebuild.json` (from Lab 3, giving the sign of $u$ that points toward looping), and `labs/data/astra_pairs.json` (the agent's 8 texts).
+
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+nvidia-smi
+CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py
+```
+
+Parts 1 and 2 (the default, `--parts 1,2`) finish in about 1 minute and write two files:
+- `results/lab6/equivalence.json` (comparing the in-memory weight edit, the steering hook, and the released `bf16` weight file)
+- `results/lab6/s_on_pairs.json` ($s(t)$ on every token of the 8 texts in `labs/data/astra_pairs.json`)
+
+Part 3 traces real loops token by token from the unedited model's answers in `/nvme/wsp-lab/runs/build/base`:
+
+```bash
+CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3
+```
+
+Part 3 finishes in about 2 minutes and writes four files:
+- `results/lab6/real_loops.json` (summary statistics across 48 looping answers)
+- `results/lab6/trace_one_loop.json` (per-token arrays for one answer whose loop starts near the middle)
+- `results/figures/lab6_real_loop.png` and `results/figures/lab6_loops_summary.png`
+
+To measure the same signals on answers that finished without looping, run part 4 on the same VM as part 3 (or pass `--parts 3,4` to run both):
+
+```bash
+CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 4
+```
+
+Part 4 takes under a minute and reads part 3's `results/lab6/real_loops.json` on the same machine to compare the two groups and draw the figure; without that file, it writes only the good-answer numbers. With `--try-name` or `--max-samples`, part 4 uses the same `results/lab6/try-<name>/` folder as part 3 and compares against the loops saved there, or against the main `real_loops.json` if part 3 saved none. It writes two files:
+- `results/lab6/finished_answers.json` (summary statistics and per-answer numbers for 26 finished answers)
+- `results/figures/lab6_finished_vs_loops.png` (three-panel box-plot comparison of good answers and loops)
+
+To redraw the part 3 figures from the saved JSON files on CPU (`--plot-only` does not redraw part 4's figure), run:
+
+```bash
+python labs/lab6_steering.py --plot-only
+```
+
+## What you should see
+
+Parts 1 and 2 print:
+
+```text
+fp32_hook_vs_edit_max_abs_logit_diff                 0.00774
+fp32_base_vs_edit_max_abs_logit_diff                 11.4
+fp32_hook_vs_edit_mean_kl                            1.41e-08
+fp32_base_vs_edit_mean_kl                            0.0178
+bf16_hook_vs_released_file_max_abs_logit_diff        0.438
+bf16_hook_vs_released_file_mean_kl                   0.000539
+bf16_base_vs_released_file_mean_kl                   0.0184
+bf16_same_top_token_hook_vs_file                     0.994
+fp64_one_layer_max_abs_diff                          3.33e-16
+fp64_one_layer_output_scale                          0.65
+layer 15: s over the last 36 tokens, productive [0.0, 0.04, 0.04, 0.01] vs looping [0.19, 0.19, 0.42, 0.25]
+layer 19: s over the last 36 tokens, productive [-0.03, -0.04, 0.02, -0.01] vs looping [0.09, 0.11, 0.44, 0.3]
+```
+
+Part 1 (`results/lab6/equivalence.json`) compares the gap between the hook and the weight edit against the edit's effect on the base model:
+
+| Comparison | Max absolute difference | Mean KL divergence | What it checks |
+|---|---:|---:|---|
+| `fp64`, layer 15 `down_proj`, hook vs edit | `3.33e-16` (on outputs of scale `0.65`) | not applicable (single layer) | Exact match up to 64-bit rounding |
+| `fp32`, full model, hook vs in-memory edit | `0.00774` (`0.0077` in logits) | `1.41e-08` (`1.4e-8`) | Full-model rounding across 24 layers (largest logit gap about 1,500 times smaller than the edit's effect, mean KL about 1.3 million times smaller; the gap drops to `0.00134` with full IEEE `fp32`) |
+| `fp32`, full model, base vs in-memory edit | `11.4` (logits) | `0.0178` | Actual size of the edit's effect in `fp32` |
+| `bf16`, hook on base vs released `qwen35-p50c` file | `0.438` (`0.4375` in logits) | `0.000539` (`5.4e-4`) | Same top token on `99.4%` (`0.994`) of positions |
+| `bf16`, unedited base vs released `qwen35-p50c` file | not measured by script | `0.0184` | Actual size of the released `bf16` edit's effect (34 times larger than the hook-vs-file KL) |
+
+Part 2 (`results/lab6/s_on_pairs.json`) measures mean $s(t)$ over the last 36 tokens of each of the 8 texts, with $u$ oriented so positive values point toward looping:
+
+| Layer | Productive texts (4 texts) | Looping texts (4 texts) |
+|---|---|---|
+| Layer 15 | `0.00, 0.04, 0.04, 0.01` | `0.19, 0.19, 0.42, 0.25` |
+| Layer 19 | `-0.03, -0.04, 0.02, -0.01` | `0.09, 0.11, 0.44, 0.30` |
+
+On the four short productive texts, the 64 neurons barely push along $u$, or push slightly the other way. On the four looping texts, their push along $u$ reaches `0.09` to `0.44`. Because the edit subtracts $6 \cdot s(t) \cdot u$, the text sets the push strength, unlike a constant nudge. Part 4 shows that on long real answers that finish correctly, $s(t)$ is small but not zero.
+
+Each pair shares an opening (`75`, `85`, `70` and `69` tokens). On that shared prefix, $s(t)$ matches in the first three pairs (`0.0000` max difference) and differs by at most `0.0018` at layer 15 and `0.0042` at layer 19 in the fourth pair, from `bf16` rounding on the GPU when the two texts have different total lengths. After the split, $s(t)$ climbs on the looping text to peaks of `0.46, 0.47, 0.59, 0.49` at layer 15 (`0.4558, 0.4713, 0.5897, 0.4873`) and `0.36, 0.43, 0.74, 0.63` at layer 19 (`0.3620, 0.4300, 0.7376, 0.6322`).
+
+Part 3 (`results/lab6/real_loops.json` and `results/lab6/trace_one_loop.json`) takes `48` base-model answers from `runs/build/base` whose verbatim loop starts at least `1,024` tokens in. It feeds each fixed token sequence through the base and edited models (teacher forcing) and records at every token:
+- $s_{15}$ and $s_{19}$: how hard the 64 neurons push along $u$ at layers 15 and 19.
+- $\text{KL}(\text{base} \parallel \text{edited})$: how far the edited next-token probabilities moved from the base model's.
+- `p_next`: the probability of the actual next token under plain softmax at temperature 1.
+- `p_pick`: the chance that vLLM's sampler picks that next token under the paper's decoding settings (presence penalty `1.5` on tokens already used in the answer, temperature `0.6`, top-k `20`, top-p `0.95`, applied in vLLM's order by `wsp/protocol.py` `pick_prob`).
+
+Means across the `48` looping answers by stretch:
+
+| Stretch of the answer | `s15` | `s19` | `KL` | `p_next` base | `p_next` edit | `p_pick` base | `p_pick` edit |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Early (first half before the loop) | `0.096` | `0.116` | `0.021` | `0.768` (`0.7675`) | `0.750` | `0.834` | `0.821` |
+| Late (last 512 tokens before the loop) | `0.178` | `0.241` | `0.038` | `0.862` | `0.825` | `0.904` | `0.881` |
+| Inside the loop | `0.424` | `0.758` | `0.069` | `0.99975` | `0.946` | `0.99995` | `0.979` |
+
+Multiplying per-token probabilities across one repeating block of length `period` gives the cycle odds, the chance of writing one more identical cycle:
+
+| Scoring rule | Unedited, mean | Unedited, median | Edited, mean | Edited, median |
+|---|---:|---:|---:|---:|
+| Temperature 1, plain softmax | `0.992` | `0.995` | `0.205` | `0.097` |
+| The paper's sampler | `0.999` | `0.999` | `0.487` | `0.625` |
+
+- Under the paper's sampler and the edit, `23 of the 48` loops have odds below `0.5` of completing one more cycle.
+- Loop periods range from `10 to 224` tokens (median `43.5`). Longer loops break more easily: the correlation between $\log(\text{period})$ and edited sampler cycle odds is `-0.66` (`-0.657`), because a small per-token cut (`0.99995` to `0.979`) compounds across a long cycle.
+- The trigger rises before the verbatim loop starts. Across the early, late, and in-loop stretches, $s_{15}$ climbs `0.10 -> 0.18 -> 0.42` (`0.096 -> 0.178 -> 0.424`), $s_{19}$ climbs `0.12 -> 0.24 -> 0.76` (`0.116 -> 0.241 -> 0.758`), and $\text{KL}(\text{base} \parallel \text{edited})$ climbs `0.021 -> 0.038 -> 0.069`. Even in the early stretch, $s_{19}$ averages `0.12` and KL averages `0.021`, so the edit is never fully off; part 4 checks answers that never looped.
+- In the traced loop (`trace_one_loop.json`, plotted in `lab6_real_loop.png`), problem `math_train_intermediate_algebra_1137`, answer `0`, the loop starts at token `14,213` of `32,768` with period `15`. At temperature 1, one-cycle odds drop from `0.995` (unedited) to `0.64` (`0.6415`, edited), and ten-cycle odds drop from `0.988` to `0.013`. Under the paper's sampler, one-cycle odds are `0.997` (unedited) versus `0.991` (edited), and ten-cycle odds are `0.996` versus `0.970`. The ten-cycle numbers average the chance of writing each 150-token window inside the loop unchanged. Because probabilities vary along the loop, that differs from raising the one-cycle odds to the tenth power (`0.954`, `0.012`, `0.974` and `0.914`). This 15-token loop survives the edit under the paper's sampler because low temperature (`0.6`) and top-p (`0.95`) keep the repeated token far ahead of the runner-up.
+
+![One real loop from the base model, token by token: the push s(t) at layers 15 and 19, the KL change in next-token predictions, and the probability of the token that came next for the base and edited models. All of them jump where the loop starts at token 14,213.](../results/figures/lab6_real_loop.png)
+
+![Four panels for 48 real loops from the base model: s(t) at layer 15, the KL change and the next-token probability, each early, late and inside the loop, and the chance of one more identical cycle against loop length for the base and edited models.](../results/figures/lab6_loops_summary.png)
+
+Part 3 only examined answers that eventually looped, so it could not tell us how $s(t)$ behaves when reasoning goes well from start to finish. Part 4 (`results/lab6/finished_answers.json`) checks that, using all `26` answers in `runs/build/base` that closed `</think>` on their own and gave the right final number. These answers think for `750` to `9,698` tokens (median `3,770`), followed by a written answer of `190` to `740` tokens after `</think>` (median about `320`). Teacher forcing the `26` good answers through the unedited and edited models and comparing them with the `48` loops from part 3 gives these stretch means:
+
+| Group and stretch | `s15` | `s19` | `KL` |
+|---|---:|---:|---:|
+| Good answers (`26`), early (first half of thinking) | `0.062` | `0.091` | `0.011` |
+| Good answers (`26`), late (last 512 tokens of thinking) | `0.079` | `0.116` | `0.017` |
+| Good answers (`26`), written answer after `</think>` | `0.066` | `0.108` | `0.007` |
+| Looping answers (`48`), early (first half before loop) | `0.096` | `0.116` | `0.021` |
+| Looping answers (`48`), late (last 512 tokens before loop) | `0.178` | `0.241` | `0.038` |
+| Looping answers (`48`), inside the loop | `0.424` | `0.758` | `0.069` |
+
+- **Small on good answers, but not zero.** Across the three stretches of the `26` good answers, $s_{19}$ averages `0.091` to `0.116`, about one seventh of the in-loop mean (`0.758`). Since $u$ has length `1` and $\alpha = 6$, the push per token has length $6 \cdot |s(t)|$. At these average levels, that is about `0.5` to `0.7` at layer 19 on good answers versus `4.5` inside loops. Mean $\text{KL}(\text{base} \parallel \text{edited})$ on good answers is `0.007` to `0.017` per token, about a tenth to a quarter of the in-loop KL (`0.069`).
+- **Never at loop level.** For each good answer, the script finds the hardest 64-token stretch of thinking (the highest 64-token moving average of $s$). Across the `26` answers, this peak is `0.082` to `0.316` at layer 15 (median `0.159`) and `0.147` to `0.465` at layer 19 (median `0.223`). By comparison, the median loop averages `0.427` (layer 15) and `0.754` (layer 19) inside the loop, and even the weakest of the `48` loops averages `0.374` and `0.671`. Thus `0 of the 26` good answers have any 64-token stretch reaching even the weakest loop's in-loop mean.
+- **Early overlap, late separation.** The 25th / 50th / 75th percentiles of $s_{19}$ are `0.076 / 0.086 / 0.109` (good early), `0.078 / 0.105 / 0.123` (good late), and `0.082 / 0.099 / 0.122` (good written), versus `0.096 / 0.111 / 0.127` (loop early), `0.166 / 0.219 / 0.267` (loop late), and `0.719 / 0.754 / 0.785` (loop inside). The two groups overlap early in thinking; in the late stretch, most loops sit above most good answers; inside the loop, there is a wide gap. The signal $s(t)$ is a graded level: low on good reasoning, higher as an answer circles, and highest inside a loop.
+- **What this comparison does not control for.** The `26` good answers come from `18` problems (`8` where both practice answers finished, and `10` where the other answer ran to the token limit). Because they are not matched to the problems behind the `48` loops, and problems that finish are likely easier than those that loop, problem difficulty can explain part of the gap.
+
+![Box plots of s(t) at layers 15 and 19 and of the KL change, for 26 answers that finished with the right number (green) and 48 that ended in a loop (red). The loop boxes climb from early to inside the loop, while the good answers stay low.](../results/figures/lab6_finished_vs_loops.png)
+
+The good answer closest to loop level is `gsm8k_train_6504`, answer `1` ("Bob bought 2 show dogs for \$250.00 each to breed as a side business. The female just had a litter of 6 puppies. If he sells each puppy for \$350.00, what is his total profit?", truth `1600`). It thinks for `6,585` tokens (`6,795` total). Its late stretch averages $s_{19}$ `0.299`, above the loops' late median (`0.219`), and its peak 64-token window reaches `0.465` at layer 19 and `0.316` at layer 15. In its last 512 thinking tokens, the model circles through near-copies of the same check ("Wait, is it possible the "2 show dogs" are the *items*? Yes." / "Is it possible the "2 show dogs" are the *number* of puppies? No.") between restatements of `250 x 2 = 500`, `350 x 6 = 2100`, and `2100 - 500 = 1600`. It then writes "Okay, I'll proceed.", closes `</think>`, and gives the right answer, `1600.00`. Here $s(t)$ rose on circling text even though the answer recovered.
+
+## Why it works
+
+### 1. What `down_proj` computes on each token
+
+A language model reads and writes text one token (word piece) at a time. Inside Qwen3.5-0.8B, each token carries a residual stream: a running list of 1,024 numbers that passes through 24 repeated blocks (or layers). Each block has a sequence mixer (standard attention at layers 3, 7, 11, 15, 19, and 23, and a linear DeltaNet mixer at the other 18 layers) followed by an MLP (multi-layer perceptron).
+
+The MLP maps those 1,024 numbers to 3,584 intermediate neurons. Qwen's MLP is gated: it makes two different mixes of the stream, `gate_proj` and `up_proj`, passes the first through SiLU (a smooth cousin of $\max(0, x)$ that dips slightly below zero), and multiplies the two together, number by number. Because the `up_proj` number can have either sign, a neuron's activation can be positive or negative. An output matrix called `down_proj`, of shape $1,024 \times 3,584$, then writes a 1,024-number update back into the residual stream. Let $a_j(t)$ be the neuron activation of neuron $j$ on token $t$ (how strongly it fires), and let $W[:, j]$ be column $j$ of `down_proj`. The product $W a(t)$ sums the columns scaled by their activations:
+
+$$W a(t) = \sum_{j=0}^{3583} a_j(t) \, W[:, j]$$
+
+Column $W[:, j]$ is the 1,024-number vector that neuron $j$ adds to the residual stream per unit of activation.
+
+### 2. Why editing columns equals subtracting $\alpha \cdot s(t) \cdot u$
+
+Let $u$ be the unit direction in the 1,024-number residual stream (a vector of length 1 pointing toward looping), $S$ be the 64 chosen neuron indices ($|S| = 64$), and $\alpha = 6$ be the edit strength. A rank-one matrix change modifies every selected column along the single direction $u$. For each neuron $j$ in $S$, the dot product $u \cdot W[:, j]$ measures how much column $j$ writes along $u$. The weight edit subtracts $\alpha$ times that projection from each column in $S$ and leaves the other 3,520 columns unchanged:
+
+$$W'[:, j] = \begin{cases} W[:, j] - \alpha \, u \, (u \cdot W[:, j]) & \text{if } j \in S \\ W[:, j] & \text{if } j \notin S \end{cases}$$
+
+Multiply the edited matrix $W'$ by the activations $a(t)$ on token $t$. Because summation is linear, we can pull $\alpha$ and $u$ outside the sum over the 64 chosen neurons:
+
+$$W' a(t) = \sum_{j=0}^{3583} a_j(t) \, W[:, j] - \alpha \, u \sum_{j \in S} a_j(t) \, (u \cdot W[:, j]) = W a(t) - \alpha \, s(t) \, u$$
+
+Here $s(t)$ is one number per token:
+
+$$s(t) = \sum_{j \in S} a_j(t) \, (u \cdot W[:, j]) = u \cdot (W_S \, a_S(t))$$
+
+In words, $W_S a_S(t)$ is the vector the 64 chosen neurons alone add to the residual stream on token $t$, and $s(t)$ is how much of that vector points along the loop direction $u$. In `wsp/hf.py`, `steer_down` attaches a PyTorch hook (a function that intercepts a layer's output during the forward pass) to `down_proj`. It precomputes $r_j = u \cdot W[:, j]$ for the 64 neurons, computes $s(t) = a_S(t) \cdot r$ on each token, and returns `output - alpha * s(t) * u`. The script gives the hook the released file's own $u$, $\alpha$ and 64 columns (from `results/edit_info.json`), so the hook and the weight edit are the same edit written two ways.
+
+```mermaid
+flowchart LR
+    A["Neuron activations a(t)<br/>3,584 numbers per token"] --> B["Path A: Edited matrix W'<br/>W' = W - alpha u u^T W_S"]
+    A --> C["Path B: Unedited matrix W<br/>y = W a(t)"]
+    A --> D["64 chosen neurons a_S(t)<br/>s(t) = u . (W_S a_S(t))"]
+    C --> E["Hook subtracts<br/>alpha * s(t) * u"]
+    D --> E
+    B --> F["Same MLP output<br/>to floating-point rounding"]
+    E --> F
+```
+
+### 3. Why float64 is exact while float32 and bf16 show small rounding gaps
+
+The four number formats in part 1 keep different numbers of bits of precision: `fp64` has 53 bits, `fp32` has 24, `TF32` has 11, and `bf16` has 8.
+
+When part 1 runs layer 15 alone on CPU in `fp64`, `y_edit` and `y_hook` agree to `3.33e-16` on outputs of scale `0.65`. That is 3 times `2^-53` (about `1.1e-16`), where `2^-53` is one 64-bit rounding step for numbers between `0.5` and `1`. The two paths differ only because they sum terms in a different order.
+
+Across all 24 layers on GPU in `fp32`, the final logits (raw next-token scores before softmax turns them into probabilities) differ by at most `0.00774` (`0.0077`), with mean KL divergence (how far one probability distribution moved from another, where `0` is identical) of `1.41e-8` (`1.4e-8`). This gap comes mainly from the linear DeltaNet mixers, whose Triton GPU kernels default to `TF32` (11-bit precision) for `fp32` matrix multiplies. Small differences after layer 15 then pass through layers 16 to 23. Forcing full 24-bit IEEE `fp32` in Triton (`try-ieee` below) cuts the max logit gap `5.8`-fold, to `0.00134` (`0.0013`). Even with default `TF32`, the `0.00774` gap is about `1,500` times smaller than the edit's effect on the base model (`11.4`), and the `1.4e-8` mean KL is about 1.3 million times smaller than the edit's (`0.0178`).
+
+The released `qwen35-p50c` file stores weights in bf16 (16-bit brain floating point, with 8 bits of precision, or about 3 significant decimal digits). Rounding $W'$ to `bf16` on disk introduces one rounding step of up to `0.00049` per weight entry (Lab 1). Against the released `bf16` file, the `bf16` hook on the base model has a max logit gap of `0.438` (`0.4375`), mean KL `0.000539` (`5.4e-4`), and the same top token on `99.4%` of positions; the mean KL between the base model and the released file in `bf16` (`0.0184`) is `34` times larger.
+
+### 4. How static weights act like a token-dependent trigger
+
+Adding a constant steering vector $-c \cdot u$ at layers 15 and 19 on every token would push the residual stream equally hard during clean algebra and during a loop. Because the weight edit multiplies $u$ by $-\alpha \cdot s(t)$, the token itself sets the steering dose through the activations $a_S(t)$ of the 64 chosen neurons.
+
+```mermaid
+flowchart LR
+    T["Current token t"] --> N["64 chosen MLP neurons<br/>fire with activations a_S(t)"]
+    N --> P["Finished real answers<br/>s(t) small (layer 19: 0.09 to 0.12)<br/>Edit push -6 s(t) u is small"]
+    N --> L["Inside real loops<br/>s(t) large (0.42 L15, 0.76 L19)<br/>Edit push -6 s(t) u opposes u"]
+```
+
+Part 2 shows this on the 8 paired texts in `labs/data/astra_pairs.json`. Each pair shares its problem prompt and opening calculation before splitting into a productive check or a repeated sentence. In `results/lab6/s_on_pairs.json`, the `per_token` arrays match on the shared opening, apart from tiny `bf16` rounding differences in the fourth pair, then $s(t)$ stays near zero on the productive branch and climbs on the looping branch. Over the last 36 tokens of these short hand-picked texts, $s(t)$ is near zero on the productive continuations and large on the loops; on long real answers in part 4, productive $s(t)$ is higher, though still far below loop level.
+
+Part 3 tests this trigger on long base-model answers (`runs/build/base` on `build128`). It selects 48 answers that were cut at the 32,768-token limit and end in a verbatim loop (the same token block repeated at least 3 times across at least 64 tokens, where one block's length is the period) starting after at least 1,024 tokens, to focus on answers that reasoned normally before getting stuck. Using teacher forcing, which feeds a fixed token sequence through the model instead of generating token by token, it runs each answer once with the edit hooks off and once with them on. That lets us line up every token position $t$ and compare $s(t)$, $\text{KL}(\text{base} \parallel \text{edited})$, `p_next`, and `p_pick` across the early answer, the last 512 tokens before the loop starts, and the tokens inside the loop. Part 4 runs the same check on the `26` base-model answers that closed `</think>` and got the right answer, where $s_{19}$ averages `0.09` to `0.12` per stretch and `0 of 26` reach even the weakest loop's in-loop mean (`0.671`) on any 64-token window.
+
+**Lesson:** The weight edit and subtracting $\alpha \cdot s(t) \cdot u$ at layers 15 and 19 are the same computation. The text sets the dose through $s(t)$: small on ordinary reasoning (about one seventh of the in-loop mean at layer 19), rising as the model starts to circle, and largest inside a loop, where a small per-token cut adds up to a large chance of breaking the cycle. Because $s(t)$ stays above zero on good answers, the edit still nudges them a little; [Lab 7](lab7_side_effects.md) measures that cost.
+
+## Check yourself
+
+1. In Lab 3, the released file's singular vector $u$ at layer 19 had the opposite sign from the mean-difference vector $u_{\text{loop}}$. Why does replacing $u$ with $-u$ flip the sign of $s(t)$ without changing the steering push $-\alpha s(t) u$ or the weight edit $W'$?
+<details>
+<summary>Answer</summary>
+
+Both $s(t) = u \cdot (W_S a_S(t))$ and $u$ appear once in $s(t) u$. Replacing $u$ with $-u$ turns $s(t)$ into $-s(t)$, and the two minus signs cancel in $(-s(t))(-u) = s(t) u$. In $W' = W - \alpha u u^T W_S$, $u$ multiplies its transpose $u^T$, and $(-u)(-u)^T = u u^T$.
+
+</details>
+
+2. In part 1, the maximum `fp32` logit difference between the hook and the in-memory edit is `0.00774` (`0.0077`). Which numbers in `equivalence.json` and `try-ieee/equivalence.json` show this comes from floating-point kernel rounding and not from a mistake in the hook formula?
+<details>
+<summary>Answer</summary>
+
+Running layer 15's `down_proj` alone in `fp64` on CPU, where no GPU kernel rounds intermediate sums, gives `fp64_one_layer_max_abs_diff` of `3.33e-16` on outputs of scale `0.65`. Forcing full 24-bit IEEE `fp32` in the DeltaNet Triton kernels (`results/lab6/try-ieee/equivalence.json`) cuts `fp32_hook_vs_edit_max_abs_logit_diff` `5.8`-fold from `0.00774` to `0.00134`. Across the full `fp32` model, the edit itself shifts base logits by `11.4` (`fp32_base_vs_edit_max_abs_logit_diff`) with mean KL `0.0178`, while the hook and in-memory edit differ by `0.00774` (`1,500` times smaller) with mean KL `1.41e-8` (1.3 million times smaller).
+
+</details>
+
+3. On the four productive texts at layer 15, mean $s(t)$ over the last 36 tokens stays between `0.00` and `0.04`, versus `0.19`, `0.19`, `0.42`, and `0.25` on the four looping texts. How does $-\alpha \cdot s(t) \cdot u$ explain why the edit pushes much less on productive steps?
+<details>
+<summary>Answer</summary>
+
+Since $u$ has length `1` and $\alpha = 6$, the vector subtracted on token $t$ has length $|6 \cdot s(t)|$: at most about `0.24` (`6` times `0.04`) on the productive texts versus about `1.1` to `2.5` (`6` times `0.19` to `6` times `0.42`) on the looping texts. As the 64 neurons fire together in a loop, the push grows with $s(t)$ against $u$. Part 4 finds a smaller gap in the same direction on long real answers, so the push on good answers is small but not zero.
+
+</details>
+
+4. Why use teacher forcing in part 3 instead of letting the edited model generate freely from the prompt?
+<details>
+<summary>Answer</summary>
+
+Free generation diverges as soon as the edit changes one sampled token, leaving the base and edited models on different sentences. Teacher forcing feeds the same base-model loop into both models so we can compare $s(t)$, KL divergence, and loop-continuation odds at identical token positions.
+
+</details>
+
+5. Part 4 finds that `0 of 26` good answers have any 64-token stretch where $s(t)$ reaches loop level. Why does that not show that the edit leaves good answers unchanged?
+<details>
+<summary>Answer</summary>
+
+The push is proportional to $s(t)$, which on good answers is about one seventh of the in-loop mean (`0.091` to `0.116` at layer 19 versus `0.758`), not zero. That shifts next-token probabilities by a mean KL of `0.007` to `0.017` per token on good answers, about a tenth to a quarter of the in-loop KL (`0.069`). Across thousands of thinking tokens, those small shifts add up and can change which tokens get sampled; [Lab 7](lab7_side_effects.md) measures those side effects directly.
+
+</details>
+
+## Try this next
+
+Non-default flags write to `results/lab6/try-<name>/` and leave the main result files and figures untouched.
+
+1. **Inspect where $s(t)$ splits inside the 4 pairs.** Run this one-liner on the VM or locally to check the shared openings (`75`, `85`, `70` and `69` tokens) and post-split peaks in `results/lab6/s_on_pairs.json`:
+   ```bash
+   python3 -c 'import json; d=json.load(open("results/lab6/s_on_pairs.json")); k="per_token"; [print(f"L{L} pair {p}: shared diff={max(abs(a-b) for a,b in zip(d[L][k][2*p][:sh], d[L][k][2*p+1][:sh])):.4f}, loop peak={max(d[L][k][2*p+1]):.4f}") for L in ("15","19") for p,sh in enumerate((75,85,70,69))]'
+   ```
+   This prints `0.0000` on the first three shared openings, `0.0018` (layer 15) and `0.0042` (layer 19) on the fourth from `bf16` rounding, and loop peaks of `0.4558, 0.4713, 0.5897, 0.4873` at layer 15 and `0.3620, 0.4300, 0.7376, 0.6322` at layer 19.
+
+2. **Double the edit strength to $\alpha = 12$ with `--alpha-scale 2`.** The `--alpha-scale` flag scales $\alpha$ in both the hook and the in-memory edit (skipping the `bf16` file comparison because the file on disk was saved at $\alpha = 6$) and writes `results/lab6/try-alpha-scale-2/equivalence.json`:
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 1 --alpha-scale 2
+   ```
+   In `results/lab6/try-alpha-scale-2/equivalence.json`, the hook and in-memory edit still agree (`0.00916` max logit diff, mean KL `2.71e-8`). Against the base model, the max logit diff rises to `15.9` (`15.93`) and mean KL rises to `0.0745`, or `4.2` times the $\alpha = 6$ KL (`0.0178`). Doubling a small change roughly quadruples the KL divergence, because KL grows like the square of a small change.
+
+3. **Force full IEEE `fp32` in the Triton DeltaNet kernels (`try-ieee`).** Triton defaults to 11-bit `TF32` for `fp32` matrix multiplies on A100 Tensor Cores. Pass `TRITON_F32_DEFAULT=ieee` with a fresh cache directory to recompile in 24-bit `fp32` and write `results/lab6/try-ieee/equivalence.json`:
+   ```bash
+   TRITON_F32_DEFAULT=ieee TRITON_CACHE_DIR=/nvme/wsp-lab/tmp/triton-ieee \
+     CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 1 --try-name ieee
+   ```
+   The max `fp32` logit gap drops from `0.00774` to `0.00134` (`0.0013`), `5.8` times smaller, showing that `TF32` rounding in the DeltaNet mixers caused most of the `fp32` gap.
+
+4. **Run part 3 on 10 loops (`--max-samples 10`) and redraw with less smoothing (`--smooth 16`).** Passing `--max-samples 10` runs part 3 in about a minute on 10 loops and writes `real_loops.json`, `trace_one_loop.json`, and both figures to `results/lab6/try-max-samples-10/`:
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3 --max-samples 10
+   ```
+   Across those 10 loops, edited `p_pick` inside the loop averages `0.984`, mean cycle odds at temperature 1 drop from `0.991` to `0.150`, and under the paper's sampler cycle odds drop from `0.999` to `0.511`. To redraw the 48-loop figure with a 16-token moving average instead of 64 (CPU only, writing to `results/lab6/try-smooth-16/`):
+   ```bash
+   python labs/lab6_steering.py --plot-only --smooth 16
+   ```
+   Comparing `results/lab6/try-smooth-16/lab6_real_loop.png` with `results/figures/lab6_real_loop.png` shows the token-by-token oscillations inside each cycle that the 64-token window smooths out.
+
+5. **Read the good answer that came closest to loop level.** Sort `samples` in `results/lab6/finished_answers.json` by `max_s19_64_tokens`:
+   ```bash
+   python3 -c 'import json; d=json.load(open("results/lab6/finished_answers.json")); [print(s["id"], s["j"], round(s["max_s19_64_tokens"], 3)) for s in sorted(d["samples"], key=lambda s: -s["max_s19_64_tokens"])[:3]]'
+   ```
+   This prints `gsm8k_train_6504 1 0.465`, `math_train_intermediate_algebra_605 0 0.353`, and `math_train_prealgebra_141 1 0.31`. Inspect the last 512 thinking tokens of `gsm8k_train_6504` answer `1` (tokens `6073` to `6585` from the start of the answer, on `sampler`, CPU only):
+   ```bash
+   CUDA_VISIBLE_DEVICES="" python labs/peek.py build/base --id gsm8k_train_6504 --answer 1 --tokens 6073 6585
+   ```
+   The header `answer 1: 6,796 tokens, finished` shows the generator's token count (`6,795` when re-tokenized from saved text). On this stretch, where $s_{19}$ averages `0.299` (this answer's hardest 64-token window of thinking reaches `0.465` at layer 19 and `0.316` at layer 15), the model circles through the repeated check and arithmetic described in part 4 before closing `</think>` and giving the right answer `1600.00`. Does $s(t)$ respond only to fatal infinite loops, or to circling text even when the answer recovers?

code/labs/lab6_steering.py

@@ -0,0 +1,431 @@
+#!/usr/bin/env python
+"""Lab 6: the weight edit is steering in disguise.
+
+    . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+    CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py                 # parts 1 and 2, about a minute
+    CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3       # part 3, about 2 minutes, needs a finished base run
+    CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 4       # part 4, about a minute, needs part 3's results
+    python labs/lab6_steering.py --plot-only                            # redraw part 3's figures, no GPU
+
+Experiments (each writes to its own folder, results/lab6/try-*, and leaves the main results alone):
+    CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 1 --alpha-scale 2      # the same check at alpha 12
+    CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3 --max-samples 10     # part 3 on 10 loops
+    python labs/lab6_steering.py --plot-only --smooth 16                               # less smoothing, no GPU
+    TRITON_F32_DEFAULT=ieee TRITON_CACHE_DIR=/nvme/wsp-lab/tmp/triton-ieee \
+      CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 1 --try-name ieee    # DeltaNet kernels in full fp32
+
+Part 1. Same math, two ways. In float32, compare the logits of
+  (a) the model with down_proj edited in memory:  W' = W - alpha u u^T W[:, cols]
+  (b) the unedited model plus a hook that subtracts alpha * s(t) * u from down_proj's output,
+      where s(t) = u . (W[:, cols] a[cols](t)) is one number per token.
+  They agree to float32 rounding. Then compare the released bf16 file with (b) run in bf16.
+Part 2. s(t) on the agent's 8 texts: how hard the 64 neurons push along the loop direction.
+Part 3. Real loops. Teacher-force looping answers from the base model and follow, token by token:
+  s(t), how much the edit changes the next-token prediction (KL), and the probability of the token that
+  actually came next, both straight from the model (temperature 1) and as the chance that the paper's
+  sampler settings pick it (wsp/protocol.py, pick_prob). Inside the loop it also works out the chance
+  of writing one more full cycle unchanged.
+Part 4. Answers that went well. The same s(t) and KL on every answer in the run that finished on its own with
+  the right number, side by side with part 3's loops (from part 3's folder for the same flags, or the main results
+  if that folder has none). If the edit only reacted to loops, both would stay near zero here.
+"""
+import argparse
+import contextlib
+import json
+import os
+import sys
+
+import numpy as np
+import torch
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+from wsp import hf, loops, protocol  # noqa: E402
+from wsp.edits import projection_edit  # noqa: E402
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--parts", default="1,2")
+ap.add_argument("--run", default=os.path.join(hf.LAB, "runs", "build", "base"))
+ap.add_argument("--data", default="build128")
+ap.add_argument("--max-samples", type=int, default=48, help="part 3: how many looping answers to teacher-force")
+ap.add_argument("--alpha-scale", type=float, default=1.0,
+                help="part 1: multiply alpha by this in both the hook and the in-memory edit (2 means alpha 12)")
+ap.add_argument("--plot-only", action="store_true", help="redraw the part 3 figures from results/lab6/*.json")
+ap.add_argument("--smooth", type=int, default=64, help="moving-average window for the one-loop figure, in tokens")
+ap.add_argument("--try-name", help="write this run's results to results/lab6/try-<name>/ instead of the main files")
+a = ap.parse_args()
+parts = {int(x) for x in a.parts.split(",")}
+
+OUT = os.path.join(ROOT, "results", "lab6")
+FIG = os.path.join(ROOT, "results", "figures")
+# Experiments write to results/lab6/try-*/, so the main results and figures are never overwritten.
+# Changing --alpha-scale, --max-samples or --smooth picks a folder name for you.
+name1 = a.try_name or (f"alpha-scale-{a.alpha_scale:g}" if a.alpha_scale != 1.0 else None)
+name3 = a.try_name or (f"max-samples-{a.max_samples}" if a.max_samples != 48 else None)
+OUT1 = os.path.join(OUT, f"try-{name1}") if name1 else OUT       # part 1
+OUT2 = os.path.join(OUT, f"try-{a.try_name}") if a.try_name else OUT  # part 2
+OUT3 = os.path.join(OUT, f"try-{name3}") if name3 else OUT       # part 3, and where --plot-only reads it
+FIG3 = OUT3 if name3 else FIG
+if a.smooth != 64:
+    FIG3 = os.path.join(OUT, f"try-{name3 + '-' if name3 else ''}smooth-{a.smooth}")
+for d in (OUT, FIG, OUT1, OUT2, OUT3, FIG3):
+    os.makedirs(d, exist_ok=True)
+LAYERS = (15, 19)
+rel = hf.released_edit()
+lab3 = json.load(open(os.path.join(ROOT, "results", "lab3", "rebuild.json")))
+# the released direction u, signed so that +u points toward looping (Lab 3 measured which sign that is)
+u_loop = {}
+for L in LAYERS:
+    u_rel = rel[L]["u"] / rel[L]["u"].norm()
+    u_loop[L] = u_rel if float(u_rel @ torch.tensor(lab3["layers"][str(L)]["u_loop_sign"])) > 0 else -u_rel
+pairs = json.load(open(os.path.join(ROOT, "labs", "data", "astra_pairs.json")))
+
+
+@contextlib.contextmanager
+def steered(m, alpha_scale=1.0, record=None):
+    """The released edit as hooks at both layers. alpha_scale=0 only records s(t)."""
+    with contextlib.ExitStack() as st:
+        for L in LAYERS:
+            st.enter_context(hf.steer_down(m, L, u_loop[L], alpha_scale * rel[L]["alpha"], rel[L]["cols"],
+                                           record=None if record is None else record[L]))
+        yield
+
+
+def kl_rows(lp, lq):  # KL(p || q) per row, from log-probs
+    return (lp.exp() * (lp - lq)).sum(-1)
+
+
+def plot_part3():
+    """Draws part 3's two figures from real_loops.json and trace_one_loop.json in OUT3, into FIG3."""
+    import matplotlib
+    matplotlib.use("Agg")
+    import matplotlib.pyplot as plt
+
+    real = json.load(open(os.path.join(OUT3, "real_loops.json")))
+    trace = json.load(open(os.path.join(OUT3, "trace_one_loop.json")))
+    stats, names = real["samples"], list(real["mean"])
+
+    def smooth(v, w=a.smooth):  # moving average that stays honest at both ends of the answer
+        v, k = np.asarray(v, dtype=float), np.ones(w)
+        return np.convolve(v, k, mode="same") / np.convolve(np.ones_like(v), k, mode="same")
+
+    tm = trace["meta"]
+    xs = np.arange(len(trace["s15"]))  # s(t): one value per answer token
+    xk = np.arange(len(trace["kl"]))   # KL and p(next): one fewer, since position t predicts token t+1
+    has_pick = "p_pick_edit" in trace  # traces written before the sampler measurement have 3 panels
+    fig, ax = plt.subplots(4 if has_pick else 3, 1, figsize=(13, 11.5 if has_pick else 9), sharex=True)
+    ax[0].plot(xs, smooth(trace["s15"]), label="layer 15", color="#0969da")
+    ax[0].plot(xs, smooth(trace["s19"]), label="layer 19", color="#8250df")
+    ax[0].set_ylabel("s(t): push along\nthe loop direction")
+    ax[0].legend(loc="upper left")
+    ax[1].plot(xk, smooth(trace["kl"]), color="#cf222e")
+    ax[1].set_ylabel("KL(base || edited)\nnext-token change")
+    ax[2].plot(xk, smooth(trace["p_base"]), label="base model", color="#57606a")
+    ax[2].plot(xk, smooth(trace["p_edit"]), label="edited model", color="#1a7f37")
+    ax[2].set_ylabel("probability of the\ntoken that came next\n(temperature 1)")
+    ax[2].legend(loc="lower left")
+    if has_pick:
+        ax[3].plot(xk, smooth(trace["p_pick_base"]), label="base model", color="#57606a")
+        ax[3].plot(xk, smooth(trace["p_pick_edit"]), label="edited model", color="#1a7f37")
+        ax[3].set_ylabel("chance the paper's\nsampler picks it")
+        ax[3].legend(loc="lower left")
+    ax[-1].set_xlabel("token position in the answer")
+    for axi in ax:
+        axi.axvline(tm["loop_starts_at"], color="black", ls="--", lw=1)
+    ax[0].set_title(f"One real loop from the base model ({tm['id']})\nthe loop starts at token "
+                    f"{tm['loop_starts_at']} and repeats every {tm['period']} tokens; lines are "
+                    f"{a.smooth}-token moving averages")
+    fig.tight_layout()
+    fig.savefig(os.path.join(FIG3, "lab6_real_loop.png"), dpi=130)
+
+    has_cycle = "cycle_pick_edit" in stats[0][names[2]]
+    fig, ax = plt.subplots(1, 4 if has_cycle else 3, figsize=(18.5 if has_cycle else 14, 3.8))
+    short = ["early", "late", "inside loop"]
+    for i, (k, lab) in enumerate([("s15", "s(t) at layer 15"), ("kl", "KL(base || edited)"),
+                                  ("p_next_base", "p(next token), temperature 1")]):
+        vals = np.array([[s[z][k] for z in names] for s in stats])
+        ax[i].boxplot([vals[:, j] for j in range(3)], tick_labels=short, showfliers=False)
+        if k == "p_next_base":
+            ve = np.array([[s[z]["p_next_edit"] for z in names] for s in stats])
+            ax[i].plot(range(1, 4), ve.mean(0), "o-", color="#1a7f37", label="edited model (mean)")
+            ax[i].plot(range(1, 4), vals.mean(0), "o-", color="#57606a", label="base model (mean)")
+            ax[i].legend()
+        ax[i].set_title(lab)
+    if has_cycle:
+        per = [s["period"] for s in stats]
+        ax[3].scatter(per, [s[names[2]]["cycle_pick_base"] for s in stats], color="#57606a", s=18, label="base model")
+        ax[3].scatter(per, [s[names[2]]["cycle_pick_edit"] for s in stats], color="#1a7f37", s=18,
+                      label="edited model")
+        ax[3].set_xscale("log")
+        ax[3].set_ylim(-0.03, 1.03)
+        ax[3].set_xlabel("loop length (tokens per cycle)")
+        ax[3].set_title("chance of one more identical cycle\n(paper's sampler settings)")
+        ax[3].legend(loc="center left")
+    fig.suptitle(f"{len(stats)} real loops from the base model, three stretches of each answer")
+    fig.tight_layout()
+    fig.savefig(os.path.join(FIG3, "lab6_loops_summary.png"), dpi=130)
+    print("wrote lab6_real_loop.png and lab6_loops_summary.png in", FIG3)
+
+
+if a.plot_only:
+    plot_part3()
+    sys.exit(0)
+
+
+# ------------------------------------------------------------------ part 1
+if 1 in parts:
+    res = {}
+    m, tok = hf.load("base", dtype=torch.float32)
+    ids = [hf.encode(tok, t) for t in pairs["texts"]]
+    with torch.no_grad():
+        base = [m(x).logits[0].float() for x in ids]
+        with steered(m, a.alpha_scale):
+            hook = [m(x).logits[0].float() for x in ids]
+        saved = {}
+        for L in LAYERS:
+            W = hf.down_proj(m, L).weight
+            saved[L] = W.data.clone()
+            W.data = projection_edit(W.data, u_loop[L].to(W.device), a.alpha_scale * rel[L]["alpha"], rel[L]["cols"])
+        edit = [m(x).logits[0].float() for x in ids]
+        for L in LAYERS:
+            hf.down_proj(m, L).weight.data = saved[L]
+    res["fp32_hook_vs_edit_max_abs_logit_diff"] = max(float((h - e).abs().max()) for h, e in zip(hook, edit))
+    res["fp32_base_vs_edit_max_abs_logit_diff"] = max(float((b - e).abs().max()) for b, e in zip(base, edit))
+    res["fp32_hook_vs_edit_mean_kl"] = float(torch.cat([kl_rows(e.log_softmax(-1), h.log_softmax(-1))
+                                                        for h, e in zip(hook, edit)]).mean())
+    res["fp32_base_vs_edit_mean_kl"] = float(torch.cat([kl_rows(b.log_softmax(-1), e.log_softmax(-1))
+                                                        for b, e in zip(base, edit)]).mean())
+    del m
+    torch.cuda.empty_cache()
+
+if 1 in parts and a.alpha_scale == 1.0:  # the released file has alpha 6, so this check only makes sense at scale 1
+    m16, tok = hf.load("base")
+    ids = [hf.encode(tok, t) for t in pairs["texts"]]
+    p16, _ = hf.load("qwen35-p50c")
+    with torch.no_grad():
+        b16 = [m16(x).logits[0].float() for x in ids]
+        with steered(m16):
+            h16 = [m16(x).logits[0].float() for x in ids]
+        r16 = [p16(x).logits[0].float() for x in ids]
+    res["bf16_hook_vs_released_file_max_abs_logit_diff"] = max(float((h - r).abs().max()) for h, r in zip(h16, r16))
+    res["bf16_hook_vs_released_file_mean_kl"] = float(torch.cat([kl_rows(r.log_softmax(-1), h.log_softmax(-1))
+                                                                 for h, r in zip(h16, r16)]).mean())
+    res["bf16_base_vs_released_file_mean_kl"] = float(torch.cat([kl_rows(b.log_softmax(-1), r.log_softmax(-1))
+                                                                 for b, r in zip(b16, r16)]).mean())
+    res["bf16_same_top_token_hook_vs_file"] = float(torch.cat([(h.argmax(-1) == r.argmax(-1)).float()
+                                                               for h, r in zip(h16, r16)]).mean())
+    # one layer in float64, where no GPU kernel rounds anything: the identity holds to machine precision
+    with torch.no_grad(), hf.record_mlp_acts(m16, [15]) as acts:
+        m16(ids[1])
+    A64 = acts[15][0].double()                               # [tokens, 3584] neuron activations
+    W64 = hf.down_proj(m16, 15).weight.detach().double().cpu()
+    u64 = u_loop[15].double() / u_loop[15].double().norm()
+    al, cols = rel[15]["alpha"], rel[15]["cols"]
+    mask = torch.zeros(W64.shape[1], dtype=torch.float64)
+    mask[cols] = 1
+    W_edit = W64 - al * torch.outer(u64, mask * (u64 @ W64))  # the weight edit
+    s = A64[:, cols] @ (W64[:, cols].T @ u64)                 # one number per token
+    y_edit = A64 @ W_edit.T
+    y_hook = A64 @ W64.T - al * s[:, None] * u64[None, :]
+    res["fp64_one_layer_max_abs_diff"] = float((y_edit - y_hook).abs().max())
+    res["fp64_one_layer_output_scale"] = float(y_edit.abs().max())
+    del p16, m16
+    torch.cuda.empty_cache()
+if 1 in parts:
+    json.dump(res, open(os.path.join(OUT1, "equivalence.json"), "w"), indent=1)
+    for k, v in res.items():
+        print(f"{k:52s} {v:.3g}")
+    if a.alpha_scale != 1.0:
+        print("skipped the bf16 and fp64 checks against the released file, which was saved with alpha 6; wrote", OUT1)
+
+# ------------------------------------------------------------------ part 2
+if 2 in parts:
+    m, tok = hf.load("base")
+    rec = {L: [] for L in LAYERS}
+    with torch.no_grad(), steered(m, 0.0, rec):
+        for t in pairs["texts"]:
+            m(hf.encode(tok, t))
+    out = {}
+    for L in LAYERS:
+        tails = [float(s[-36:].mean()) for s in rec[L]]
+        out[str(L)] = {"productive_last36": tails[0::2], "looping_last36": tails[1::2],
+                       "per_token": [[round(float(x), 4) for x in s] for s in rec[L]]}
+        print(f"layer {L}: s over the last 36 tokens, productive {np.round(tails[0::2], 2).tolist()} "
+              f"vs looping {np.round(tails[1::2], 2).tolist()}")
+    json.dump(out, open(os.path.join(OUT2, "s_on_pairs.json"), "w"), indent=1)
+    del m
+    torch.cuda.empty_cache()
+
+# ------------------------------------------------------------------ part 3
+if 3 in parts:
+    m, tok = hf.load("base")
+    rows = loops.read_rows(hf.LAB, a.data)
+    found = loops.looping_samples(tok, a.run, rows)
+    print(f"{len(found)} samples in {a.run} end in a verbatim loop")
+    # keep loops with a real run-up before them, spread over the run
+    found = [f for f in found if f["onset"] - f["n_prompt"] >= 1024]
+    rng = np.random.default_rng(0)
+    pick = sorted(rng.choice(len(found), size=min(a.max_samples, len(found)), replace=False).tolist())
+    lm_head = m.lm_head
+    stats, best, trace = [], None, None
+    for k in pick:
+        f = found[k]
+        x = torch.tensor([f["ids"]], device="cuda")
+        rec_b = {L: [] for L in LAYERS}
+        rec_e = {L: [] for L in LAYERS}
+        n0, on = f["n_prompt"], f["onset"]
+        with torch.no_grad():
+            with steered(m, 0.0, rec_b):
+                hb = m.model(x, use_cache=False).last_hidden_state[0]
+            with steered(m, 1.0, rec_e):
+                he = m.model(x, use_cache=False).last_hidden_state[0]
+            T = x.shape[1]
+            # first[v] = where token v first shows up in the answer. The sampler's presence penalty
+            # hits token v at position t once first[v] <= t.
+            first = torch.full((lm_head.weight.shape[0],), T, device=x.device)
+            first.scatter_reduce_(0, x[0, n0:], torch.arange(n0, T, device=x.device), reduce="amin")
+            kl, pb, pe, qb, qe = [], [], [], [], []
+            nxt = x[0, 1:]
+            for c in range(0, T - 1, 1024):
+                zb = lm_head(hb[c:min(c + 1024, T - 1)]).float()
+                ze = lm_head(he[c:min(c + 1024, T - 1)]).float()
+                lb, le = zb.log_softmax(-1), ze.log_softmax(-1)
+                kl.append(kl_rows(lb, le).cpu())
+                ix = nxt[c:c + lb.shape[0]]
+                pb.append(lb.gather(1, ix[:, None])[:, 0].exp().cpu())
+                pe.append(le.gather(1, ix[:, None])[:, 0].exp().cpu())
+                present = first[None, :] <= torch.arange(c, c + lb.shape[0], device=x.device)[:, None]
+                qb.append(protocol.pick_prob(zb, present, ix).cpu())
+                qe.append(protocol.pick_prob(ze, present, ix).cpu())
+                del zb, ze, lb, le, present
+        kl, pb, pe = torch.cat(kl).numpy(), torch.cat(pb).numpy(), torch.cat(pe).numpy()
+        qb, qe = torch.cat(qb).numpy(), torch.cat(qe).numpy()
+        sb = {L: rec_b[L][0].numpy() for L in LAYERS}
+        # positions t predict token t+1; the three stretches of the answer
+        zones = {"early (first half before the loop)": slice(n0, n0 + (on - n0) // 2),
+                 "late (last 512 tokens before the loop)": slice(on - 512, on),
+                 "inside the loop": slice(on + f["period"], T - 1)}
+        st = {"id": f["id"], "j": f["j"], "answer_tokens": T - n0, "loop_starts_at": on - n0, "period": f["period"]}
+        for z, sl in zones.items():
+            st[z] = {"s15": float(sb[15][sl].mean()), "s19": float(sb[19][sl].mean()), "kl": float(kl[sl].mean()),
+                     "p_next_base": float(pb[sl].mean()), "p_next_edit": float(pe[sl].mean()),
+                     "p_pick_base": float(qb[sl].mean()), "p_pick_edit": float(qe[sl].mean())}
+        # Inside the loop the text repeats every `period` tokens. The chance of writing one more full
+        # cycle unchanged is the product of `period` token probabilities; average it over every start.
+        ins = zones["inside the loop"]
+        for n, v in (("base", pb), ("edit", pe), ("pick_base", qb), ("pick_edit", qe)):
+            lp = np.concatenate([[0.0], np.cumsum(np.log(np.maximum(v[ins].astype(np.float64), 1e-300)))])
+            st["inside the loop"][f"cycle_{n}"] = float(np.exp(lp[f["period"]:] - lp[:-f["period"]]).mean())
+        stats.append(st)
+        off_middle = abs(st["loop_starts_at"] / st["answer_tokens"] - 0.5)  # plot a loop that starts mid-answer
+        if best is None or off_middle < best:
+            best = off_middle
+            trace = {"meta": st, "s15": sb[15][n0:].tolist(), "s19": sb[19][n0:].tolist(), "kl": kl[n0:].tolist(),
+                     "p_base": pb[n0:].tolist(), "p_edit": pe[n0:].tolist(),
+                     "p_pick_base": qb[n0:].tolist(), "p_pick_edit": qe[n0:].tolist(), "text": f["text"]}
+        print(f"{f['id']} #{f['j']}: loop at token {on - n0} of {T - n0}, period {f['period']}; "
+              f"KL early/late/inside {[round(st[z]['kl'], 3) for z in zones]}")
+    agg = {z: {k: float(np.mean([s[z][k] for s in stats])) for k in stats[0][z]} for z in zones}
+    json.dump({"n": len(stats), "mean": agg, "samples": stats}, open(os.path.join(OUT3, "real_loops.json"), "w"),
+              indent=1)
+    json.dump(trace, open(os.path.join(OUT3, "trace_one_loop.json"), "w"))
+    print(json.dumps(agg, indent=1))
+
+    plot_part3()
+    print("wrote", OUT3)
+    del m
+    torch.cuda.empty_cache()
+
+# ------------------------------------------------------------------ part 4
+if 4 in parts:
+    m, tok = hf.load("base")
+    rows = loops.read_rows(hf.LAB, a.data)
+    end_think = tok.convert_tokens_to_ids("</think>")
+    done = []
+    for rec, j, s, text in loops.iter_samples(a.run):
+        if s.get("cut") or not s.get("correct"):
+            continue
+        prompt = tok(protocol.render(tok, rows[rec["id"]], True), add_special_tokens=False).input_ids
+        ans = tok(text, add_special_tokens=False).input_ids
+        if end_think in ans:
+            done.append({"id": rec["id"], "j": j, "ids": prompt + ans, "n_prompt": len(prompt),
+                         "think_end": len(prompt) + ans.index(end_think)})
+    print(f"{len(done)} answers in {a.run} finished on their own with the right number")
+    if not done:
+        sys.exit("part 4 needs at least one answer that finished on its own with the right number; this run has none")
+    box64 = np.ones(64) / 64
+    stats = []
+    for f in done:
+        x = torch.tensor([f["ids"]], device="cuda")
+        rec_b = {L: [] for L in LAYERS}
+        with torch.no_grad():
+            with steered(m, 0.0, rec_b):
+                hb = m.model(x, use_cache=False).last_hidden_state[0]
+            with steered(m, 1.0):
+                he = m.model(x, use_cache=False).last_hidden_state[0]
+            T = x.shape[1]
+            kl = []
+            for c in range(0, T - 1, 1024):
+                lb = m.lm_head(hb[c:min(c + 1024, T - 1)]).float().log_softmax(-1)
+                le = m.lm_head(he[c:min(c + 1024, T - 1)]).float().log_softmax(-1)
+                kl.append(kl_rows(lb, le).cpu())
+                del lb, le
+        kl = torch.cat(kl).numpy()
+        sb = {L: rec_b[L][0].numpy() for L in LAYERS}
+        n0, te = f["n_prompt"], f["think_end"]
+        zones = {"early (first half of the thinking)": slice(n0, n0 + (te - n0) // 2),
+                 "late (last 512 tokens of the thinking)": slice(max(n0, te - 512), te),
+                 "the written answer after </think>": slice(te, T - 1)}
+        st = {"id": f["id"], "j": f["j"], "answer_tokens": T - n0, "thinking_tokens": te - n0}
+        for z, sl in zones.items():
+            st[z] = {"s15": float(sb[15][sl].mean()), "s19": float(sb[19][sl].mean()), "kl": float(kl[sl].mean())}
+        for L in LAYERS:  # the hardest 64-token stretch of the thinking
+            st[f"max_s{L}_64_tokens"] = float(np.convolve(sb[L][n0:te], box64, mode="valid").max())
+        stats.append(st)
+        print(f"{f['id']} #{f['j']}: {te - n0} thinking tokens; s19 early/late/answer "
+              f"{[round(st[z]['s19'], 3) for z in zones]}; KL {[round(st[z]['kl'], 4) for z in zones]}")
+    agg = {z: {k: float(np.mean([s[z][k] for s in stats])) for k in stats[0][z]} for z in zones}
+    out = {"n": len(stats), "mean": agg, "samples": stats}
+    # Part 4 writes where part 3 does, and compares with that experiment's loops if part 3 wrote any there.
+    OUT4, FIG4 = OUT3, (OUT3 if name3 else FIG)
+    loops_file = next((p for p in (os.path.join(OUT3, "real_loops.json"), os.path.join(OUT, "real_loops.json"))
+                       if os.path.exists(p)), None)
+    if loops_file:  # line the answers up against part 3's loops
+        print("comparing with the loops in", os.path.relpath(loops_file, ROOT))
+        real = json.load(open(loops_file))
+        inside = list(real["mean"])[2]
+        out["loops_for_comparison"] = {"n": real["n"], "mean": real["mean"]}
+        for L in LAYERS:
+            med = float(np.median([s[inside][f"s{L}"] for s in real["samples"]]))
+            hit = sum(s[f"max_s{L}_64_tokens"] > med for s in stats)
+            out[f"layer {L}"] = {"median loop, mean s inside the loop": med,
+                                 "answers whose hardest 64-token stretch beats it": hit}
+            print(f"layer {L}: {hit} of {len(stats)} answers have a 64-token stretch with s above the median "
+                  f"loop's in-loop mean ({med:.3f})")
+    os.makedirs(OUT4, exist_ok=True)
+    json.dump(out, open(os.path.join(OUT4, "finished_answers.json"), "w"), indent=1)
+    print(json.dumps(agg, indent=1))
+    if "loops_for_comparison" in out:
+        import matplotlib
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+
+        fz, lz = list(agg), list(real["mean"])
+        fig, ax = plt.subplots(1, 3, figsize=(16, 4.2))
+        for i, (k, lab) in enumerate((("s15", "s(t) at layer 15"), ("s19", "s(t) at layer 19"),
+                                      ("kl", "KL(base || edited)"))):
+            data = [[s[z][k] for s in stats] for z in fz] + [[s[z][k] for s in real["samples"]] for z in lz]
+            bp = ax[i].boxplot(data, tick_labels=["right answer\nearly", "right answer\nlate",
+                                                  "right answer\nwritten", "loop\nearly", "loop\nlate",
+                                                  "loop\ninside"], showfliers=False, patch_artist=True)
+            for b, col in zip(bp["boxes"], ["#1a7f37"] * 3 + ["#cf222e"] * 3):
+                b.set_facecolor(col)
+                b.set_alpha(0.35)
+            ax[i].set_title(lab)
+            ax[i].tick_params(axis="x", labelsize=8)
+        fig.suptitle(f"The edit's trigger on {len(stats)} answers that finished with the right number (green) "
+                     f"and {real['n']} answers that ended in a loop (red), mean per stretch of each answer")
+        fig.tight_layout()
+        fig.savefig(os.path.join(FIG4, "lab6_finished_vs_loops.png"), dpi=130)
+        print("wrote lab6_finished_vs_loops.png in", FIG4)
+    print("wrote", OUT4)

code/labs/lab7_side_effects.md

@@ -0,0 +1,287 @@
+# Lab 7: What else did the edit change
+
+**Big thing:** You will score the base model and 14 edited versions on 65,536 tokens of Wikipedia articles (WikiText-2) and on the agent's 8 short math texts (4 productive, 4 looping). For each model, you measure how well it predicts the text (perplexity) and how far its next-token probabilities move from the base model's (KL divergence). The released edit (`p50c`) moves predictions 2.5 times more on looping texts than on productive ones (`0.0241` vs `0.0097`) and raises WikiText-2 perplexity by only `0.79%`.
+
+**Why it matters:** A loop fix is useless if it ruins ordinary language, since the model still has to read questions and write normal sentences. The 14 versions cover weaker and stronger edits ($\alpha$ from `-4` to `12`), single-layer edits, edits across all 3,584 neurons, and two random controls with the same weight-change size as the released edit. Comparing them shows what makes the edit act on loops and leave other text alone: the direction $u$, the 64 chosen neurons, or both.
+
+**Time and hardware:** About 1.5 minutes on one A100 GPU (78 seconds in our run), plus a few seconds on the first run to download WikiText-2.
+
+**Theory guide:** Section 13.4 (measure more than accuracy).
+
+## Before you run: predict
+
+Write down a short guess for each question before running the script.
+
+1. The released edit (`p50c`, $\alpha = 6$) flips the loop push of 64 neurons in layers 15 and 19 and makes it 5 times larger. On plain Wikipedia text (no chat template, no thinking), will perplexity get much worse (say `+15%` or more), change by less than `1%`, or improve?
+2. Two random controls test whether any change of the same size works. `randdir` edits the same 64 neurons along a random direction at right angles to $u$. `randcols` edits 64 random neurons along $u$. Both match `p50c`'s weight-change size (matched before `bf16` rounding, so they agree closely but not exactly; the build script raised their $\alpha$ to between about 20 and 31). On the agent's 8 texts, will they move predictions more on looping texts than on productive ones, or move both about equally?
+3. Compare how far WikiText-2 predictions move when editing only layer 15 (`l15only`) versus both layers 15 and 19 (`p50c`). Must the two-layer edit move WikiText-2 predictions more than the single-layer edit?
+
+## Run it
+
+Connect to `sampler` (or `trainer`). Then use `nvidia-smi` to pick a free GPU index and pass it in `CUDA_VISIBLE_DEVICES`.
+
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+nvidia-smi
+CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py
+```
+
+The script loads the base model from `/nvme/wsp-lab/models/Qwen3.5-0.8B`, reads `edit_note.json` from each `/nvme/wsp-lab/models/qwen35-*` folder (built by `scripts/build_models.py`), and loads the `down_proj` matrices at the union of edited layers (layers 15 and 19 for the 14 standard edits, plus any extra edited layers from models you built in Labs 3 and 8). Pass `--models` to score a subset. It evaluates each model on `Salesforce/wikitext` (`wikitext-2-raw-v1`, `test` split) and `labs/data/astra_pairs.json`, then writes:
+- `results/lab7/side_effects.json` (perplexity, percent perplexity change, and KL divergences for `base` and each edited model)
+- `results/figures/lab7_side_effects.png` (two-panel bar chart of WikiText-2 perplexity change and productive-versus-looping KL divergence)
+
+## What you should see
+
+The script lists the 15 models in the default run (`base` plus 14 standard edits) and the swapped `down_proj` layers, then prints one row per model and the output path and runtime (yours will differ a little):
+
+```text
+models: ['base', 'a-4', 'a1', 'a12', 'a2', 'a4', 'a8', 'all1', 'all2', 'l15only', 'l19only', 'p50c', 'randcols', 'randdir', 'rebuilt'] | down_proj swapped at layers [15, 19]
+model           ppl  ppl +%   KL wiki   KL prod   KL loop
+base         20.009    0.00   0.00000   0.00000   0.00000
+a-4          21.088    5.39   0.02058   0.00558   0.01466
+a1           19.912   -0.49   0.00160   0.00095   0.00138
+a12          22.878   14.34   0.16430   0.03661   0.10268
+a2           19.871   -0.69   0.00445   0.00178   0.00346
+a4           19.916   -0.47   0.01516   0.00485   0.01120
+a8           20.675    3.33   0.06168   0.01645   0.04416
+all1         19.925   -0.42   0.00248   0.00227   0.00289
+all2         19.937   -0.36   0.00782   0.00634   0.00919
+l15only      20.020    0.05   0.03476   0.00550   0.01938
+l19only      20.340    1.65   0.01025   0.00464   0.00755
+p50c         20.167    0.79   0.03327   0.00967   0.02405
+randcols     19.989   -0.10   0.00346   0.00446   0.00383
+randdir      20.070    0.30   0.00462   0.00479   0.00387
+rebuilt      20.167    0.79   0.03326   0.00961   0.02384
+wrote /nvme/wsp-lab/code/results/lab7 in 78 s
+```
+
+Grouping the 14 edited variants by what they test (rounded to 4 decimal places for KL, with base WikiText-2 perplexity `20.009`):
+
+| Model | What it is | Perplexity change | KL WikiText | KL productive texts | KL looping texts |
+|---|---|---:|---:|---:|---:|
+| `p50c` | Released edit ($\alpha = 6$, 64 neurons, layers 15 and 19) | `+0.79%` | `0.0333` | `0.0097` | `0.0241` |
+| `rebuilt` | Lab 3's rebuild from the agent's 8 texts | `+0.79%` | `0.0333` | `0.0096` | `0.0238` |
+| `a-4` | $\alpha = -4$ (amplifies 5x toward loops) | `+5.39%` | `0.0206` | `0.0056` | `0.0147` |
+| `a1` | $\alpha = 1$ (erase loop component on 64 neurons) | `-0.49%` | `0.0016` | `0.0010` | `0.0014` |
+| `a2` | $\alpha = 2$ (mirror loop component on 64 neurons) | `-0.69%` | `0.0044` | `0.0018` | `0.0035` |
+| `a4` | $\alpha = 4$ (flip and triple on 64 neurons) | `-0.47%` | `0.0152` | `0.0048` | `0.0112` |
+| `a8` | $\alpha = 8$ (flip and multiply by 7) | `+3.33%` | `0.0617` | `0.0164` | `0.0442` |
+| `a12` | $\alpha = 12$ (flip and multiply by 11) | `+14.34%` | `0.1643` | `0.0366` | `0.1027` |
+| `all1` | $\alpha = 1$ on all 3,584 neurons | `-0.42%` | `0.0025` | `0.0023` | `0.0029` |
+| `all2` | $\alpha = 2$ on all 3,584 neurons | `-0.36%` | `0.0078` | `0.0063` | `0.0092` |
+| `l15only` | Released edit ($\alpha = 6$, 64 neurons) on layer 15 only | `+0.05%` | `0.0348` | `0.0055` | `0.0194` |
+| `l19only` | Released edit ($\alpha = 6$, 64 neurons) on layer 19 only | `+1.65%` | `0.0102` | `0.0046` | `0.0075` |
+| `randcols` | Control: direction $u$, 64 random neurons, matched change size | `-0.10%` | `0.0035` | `0.0045` | `0.0038` |
+| `randdir` | Control: same 64 neurons, random orthogonal direction, matched size | `+0.30%` | `0.0046` | `0.0048` | `0.0039` |
+
+The script also saves `results/figures/lab7_side_effects.png`:
+
+![Left: WikiText-2 perplexity change for each edit, highest for a12. Right: KL change on productive texts (green) and looping texts (red) for each edit, on a log scale.](../results/figures/lab7_side_effects.png)
+
+## Why it works
+
+### 1. Testing 15 models (the base model plus 14 edits) by swapping matrices in memory
+
+A language model processes text one token (word piece) at a time through 24 blocks (or layers), each reading from and adding to the 1,024-number residual stream. Inside each block, the MLP (multi-layer perceptron) expands the stream to 3,584 neurons and projects them back through `down_proj` (a $1,024 \times 3,584$ matrix stored in bf16, 16-bit floating point).
+
+The base model has 488 weight tensors. In the default 15-model run (`base` plus 14 edits in `results/lab7/side_effects.json`), each edited model differs from the base model in at most two tensors: `layers.15.mlp.down_proj.weight` and `layers.19.mlp.down_proj.weight` (single-layer edits change only one). Instead of loading 15 separate 1.7 GB models from disk, `labs/lab7_side_effects.py` loads the base model onto the GPU once. It reads `edit_note.json` in each model folder, takes the union of edited layers (`LAYERS`), and swaps `down_proj` at those layers with `set_weights(w)` (printing `down_proj swapped at layers ...`). Any layer a model leaves untouched gets the base weights, which is how Lab 3's layer-21 experiment (`rebuilt-layers15-19-21`, layers 15, 19, and 21) is scored alongside 2-layer models.
+
+For each chunk of text, the script uses teacher forcing: it feeds a fixed sequence of $1,024$ tokens in one forward pass and reads next-token predictions at positions $0$ through $1,022$ (predicting tokens $1$ through $1,023$). On WikiText-2 test, 64 chunks of 1,024 tokens ($65,536$ tokens total) yield 65,472 predictions, since the first token of each chunk has no prior context.
+
+```mermaid
+flowchart LR
+    B["Load base model once<br/>save base W_15 and W_19"] --> C["Feed 64 x 1,024 WikiText tokens<br/>and the 8 math texts"]
+    V["14 variant folders<br/>load only W_15 and W_19"] --> S["Swap W_15 and W_19<br/>in GPU memory"]
+    S --> C
+    C --> M["Compare base and edited log-probs<br/>compute perplexity and KL"]
+```
+
+### 2. Perplexity and KL divergence from scratch
+
+At each token position, the model outputs logits (one raw score per vocabulary token), and softmax converts those scores into probabilities that sum to `1`. We compute two metrics from those probabilities.
+
+Imagine a two-token vocabulary, `"38"` and `"But"`, at a step where `"38"` is the true next token:
+
+- The base model predicts $p_{\text{base}}(\text{"38"}) = 0.80$ and $p_{\text{base}}(\text{"But"}) = 0.20$.
+- An edited model predicts $p_{\text{edit}}(\text{"38"}) = 0.50$ and $p_{\text{edit}}(\text{"But"}) = 0.50$.
+
+Perplexity measures how well a model predicts the true next token in ground-truth text. First take the negative natural log of the true token's probability (`"38"`), called the negative log-likelihood (NLL):
+- Base model: $\text{NLL} = -\ln(0.80) = 0.2231$.
+- Edited model: $\text{NLL} = -\ln(0.50) = 0.6931$.
+
+Average the NLL across all tokens and raise $e$ to that average: $\text{perplexity} = e^{\text{mean NLL}}$. On this single token, perplexity is $e^{0.2231} = 1 / 0.80 = 1.25$ for the base model and $e^{0.6931} = 1 / 0.50 = 2.00$ for the edited model. A model as uncertain as a fair $K$-sided die at every token (assigning probability $1/K$ to the true token) has perplexity $K$. Lower perplexity means better prediction. On the 65,536 WikiText-2 tokens, the base model's perplexity is `20.009`.
+
+KL divergence, written $\text{KL}(\text{base} \parallel \text{edited})$, measures how far the edited model's full probability distribution moved from the base model's, regardless of which token came next. At one position, it averages $\ln(p_{\text{base}}(v) / p_{\text{edit}}(v))$ weighted by the base probabilities:
+
+$$\text{KL}(p_{\text{base}} \parallel p_{\text{edit}}) = \sum_{v} p_{\text{base}}(v) \left(\ln p_{\text{base}}(v) - \ln p_{\text{edit}}(v)\right)$$
+
+In our two-token example:
+
+$$\text{KL} = 0.80 \ln\left(\frac{0.80}{0.50}\right) + 0.20 \ln\left(\frac{0.20}{0.50}\right) = 0.80(0.4700) + 0.20(-0.9163) = 0.3760 - 0.1833 = 0.1927$$
+
+Identical distributions have KL divergence `0`. Even when perplexity barely changes because gains on some tokens offset losses on others, KL divergence is strictly positive whenever predictions move.
+
+### 3. Reading the four findings in the table
+
+First, compare selectivity on looping versus productive text. Each variant in `configs/variants.json` modifies `down_proj` along a direction $u$ (a 1,024-number unit vector) with edit strength $\alpha$ (scaling the chosen neurons' push along $u$ by $1 - \alpha$). In the right panel of `lab7_side_effects.png`, compare the red bars (`kl_looping_texts`) with the green bars (`kl_productive_texts`). The released edit `p50c` ($\alpha = 6$) moves next-token predictions about 2.5 times more on looping texts (`0.0241`) than on productive ones (`0.0097`). Lab 3's rebuild (`rebuilt`) matches `p50c` closely (`0.0238` vs `0.0096`, `+0.79%` perplexity).
+
+Now compare `p50c` with the random controls `randdir` and `randcols`. Both set `norm_match: true`, rescaling $\alpha$ so the Frobenius size of the weight change $\|W' - W\|_F$ matches `p50c`'s change size before `bf16` rounding (`10.6%` of the matrix at layer 15 and `8.1%` at layer 19, from Lab 1). Despite equal change size, `randdir` (`0.0039` looping vs `0.0048` productive) and `randcols` (`0.0038` looping vs `0.0045` productive) move both text types by nearly the same small amount, shifting productive texts slightly more. Editing all 3,584 neurons (`all1`, `all2`) is also less selective than editing only the 64 chosen ones. At $\alpha = 2$, `a2` (64 neurons) moves productive text by `0.0018` and looping text by `0.0035` (ratio about 1.9), whereas `all2` (3,584 neurons) moves them by `0.0063` and `0.0092` (ratio about 1.5), shifting productive text 3.5 times more than `a2`.
+
+```mermaid
+flowchart TD
+    E["Same Frobenius weight change size<br/>at layers 15 and 19"] --> T["Targeted edit (p50c)<br/>top 64 loop neurons + loop direction u"]
+    E --> R1["Random direction control (randdir)<br/>same 64 neurons + random orthogonal u"]
+    E --> R2["Random columns control (randcols)<br/>64 other neurons + loop direction u"]
+    T --> TO["Selective shift:<br/>KL looping 0.0241 vs productive 0.0097 (about 2.5x)"]
+    R1 --> RO1["Unselective shift:<br/>KL looping 0.0039 vs productive 0.0048"]
+    R2 --> RO2["Unselective shift:<br/>KL looping 0.0038 vs productive 0.0045"]
+```
+
+KL has no sign: it measures how far predictions move, not which way. `a-4` pushes the 64 neurons toward loops instead of away, yet it is selective too (looping texts move 2.6 times more than productive ones) because it acts in the same place as `p50c`: where those 64 neurons push hard along $u$, which happens on looping text. Selectivity shows where an edit acts; Lab 4's accuracy runs show whether it helps or hurts.
+
+Second, trace the WikiText-2 perplexity dose curve across $\alpha$ (`a-4`, `a1`, `a2`, `a4`, `p50c` at $\alpha = 6$, `a8`, `a12`) in the left panel of `lab7_side_effects.png`:
+- Setting $\alpha$ to `-4` (`a-4`, pushing 5 times harder in the loop direction) raises perplexity by `+5.39%` (`21.088`).
+- Small positive edits (`a1` `-0.49%`, `a2` `-0.69%`, `a4` `-0.47%`, `all1` `-0.42%`, `all2` `-0.36%`) lower WikiText-2 perplexity slightly below the base model's `20.009`, reaching `19.871` at `a2`.
+- At the released strength $\alpha = 6$ (`p50c`), flipping and amplifying 5x costs `+0.79%` perplexity (`20.167`).
+- Above $\alpha = 6$, perplexity climbs fast: `a8` ($\alpha = 8$) adds `+3.33%` (`20.675`), and `a12` ($\alpha = 12$) adds `+14.34%` (`22.878`) with WikiText KL `0.1643`.
+
+Third, the single-layer rows `l15only` and `l19only` show that KL does not add across edits. On WikiText-2, editing layer 15 alone moves predictions slightly more (`wikitext_kl = 0.0348`) than editing both layers 15 and 19 (`p50c`, `0.0333`). Yet `l15only` barely changes perplexity (`+0.05%`), while `l19only` moves predictions less (KL `0.0102`) and raises perplexity more (`+1.65%`). This table cannot say why. One plausible story: layer 19 sits four blocks after layer 15, so the layer-15 edit changes the input reaching layer 19's 64 neurons. Because the two loop directions point along similar lines (absolute cosine similarity `0.849` in Lab 1, where `1` means the same direction and `0` means at right angles), the layer-19 edit may undo part of the layer-15 shift on ordinary text. `l15only` is also the most selective edit in the table, moving looping texts 3.5 times more than productive ones, and in Lab 4 it fixes almost as many problems as the full edit (`60.4%` correct on the dev set vs `61.5%` for `p50c`).
+
+Fourth, WikiText-2 is plain encyclopedia text scored with no chat template and no thinking. This table measures damage to ordinary language modeling, not to math reasoning or following instructions. Lab 4's dose runs measure the math side. Here, $\alpha = 12$ hurts ordinary text a lot and $\alpha = 6$ barely touches it.
+
+## Check yourself
+
+1. Suppose a model assigns probability `0.25` to the true next token at every position of a test passage. What is its average negative log-likelihood (NLL), and what is its perplexity?
+<details>
+<summary>Answer</summary>
+
+At every token, $\text{NLL} = -\ln(0.25) = \ln(4) \approx 1.3863$. Raising $e$ to the mean NLL gives $\text{perplexity} = e^{\ln(4)} = 4.0$. A constant true-token probability of $1/K$ always gives perplexity $K$.
+
+</details>
+
+2. How can `a2` ($\alpha = 2$) have a positive KL divergence on WikiText-2 (`0.0044`) while its WikiText-2 perplexity change is negative (`-0.69%`)?
+<details>
+<summary>Answer</summary>
+
+KL divergence measures how far the edited probabilities moved from the base probabilities in either direction, so any change makes KL positive. Perplexity measures how much probability lands on the actual next token in WikiText-2. For `a2`, the small shift puts slightly more probability on the true next token on average, so perplexity drops from `20.009` to `19.871` (`-0.69%`) while KL records the `0.0044` shift.
+
+</details>
+
+3. What specific alternative explanation does `randdir` rule out, and what specific alternative explanation does `randcols` rule out?
+<details>
+<summary>Answer</summary>
+
+`randdir` edits the same 64 neurons as `p50c` with the same change norm along a random direction orthogonal to $u$; its flat response (`0.0039` looping vs `0.0048` productive) argues against the idea that perturbing those 64 neurons in any direction would target loops. `randcols` uses the loop direction $u$ and the same change norm on 64 random neurons; its flat response (`0.0038` looping vs `0.0045` productive) argues against the idea that projecting $u$ out of any 64 neurons works just as well. In these tests, selectivity required both the loop direction $u$ and the neurons that fire during loops. Each control is a single random draw, so treat this as good evidence rather than proof.
+
+</details>
+
+4. Why can `l15only` have a larger `wikitext_kl` (`0.0348`) than `p50c` (`0.0333`), even though `p50c` includes the exact same layer-15 edit plus the layer-19 edit?
+<details>
+<summary>Answer</summary>
+
+KL doesn't add across edits. The layer-19 edit acts on a stream that the layer-15 edit already changed, so the two shifts can partly cancel instead of stacking. On WikiText-2 they seem to cancel a little, bringing predictions slightly back toward the base model's. This table shows that it happens, not how. Measuring the KL between `l15only` and `p50c` directly would show how much the layer-19 edit moves predictions once the layer-15 edit is in place.
+
+</details>
+
+## Try this next
+
+Each experiment writes its JSON file and figure to `results/lab7/try-<name>/`, leaving `results/lab7/side_effects.json` untouched. Our try folders are committed so you can compare your run with ours.
+
+1. **Rank the edits by how selective they are.** For each model, divide its KL on the looping texts by its KL on the productive texts, and sort:
+   ```bash
+   python -c 'import json; m=json.load(open("results/lab7/side_effects.json"))["models"]; rows=[(v["kl_looping_texts"]/v["kl_productive_texts"], k, v["ppl_change_pct"]) for k,v in m.items() if k != "base"]; print("\n".join(f"{k:10s} ratio={r:.2f}x  ppl_change={p:+.2f}%" for r,k,p in sorted(rows, reverse=True)))'
+   ```
+   Our output:
+   ```text
+   l15only    ratio=3.52x  ppl_change=+0.05%
+   a12        ratio=2.81x  ppl_change=+14.34%
+   a8         ratio=2.68x  ppl_change=+3.33%
+   a-4        ratio=2.63x  ppl_change=+5.39%
+   p50c       ratio=2.49x  ppl_change=+0.79%
+   rebuilt    ratio=2.48x  ppl_change=+0.79%
+   a4         ratio=2.31x  ppl_change=-0.47%
+   a2         ratio=1.95x  ppl_change=-0.69%
+   l19only    ratio=1.63x  ppl_change=+1.65%
+   all2       ratio=1.45x  ppl_change=-0.36%
+   a1         ratio=1.44x  ppl_change=-0.49%
+   all1       ratio=1.28x  ppl_change=-0.42%
+   randcols   ratio=0.86x  ppl_change=-0.10%
+   randdir    ratio=0.81x  ppl_change=+0.30%
+   ```
+   The layer-15-only edit ranks first, ahead of the released edit. The `a-4` edit, which pushes toward loops, also ranks above `p50c` because KL measures how far predictions move, not which way. Both random controls rank last, below 1, moving productive text slightly more than looping text.
+
+2. **Score only the last 36 tokens of each text.** In each pair, the productive and looping texts share their prompt and opening lines before the looping text starts repeating. Those shared tokens get the same KL, pulling the two KL columns together. `--last 36` scores only the last 36 tokens of each of the 8 texts (the shortest has 98 tokens):
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --last 36
+   ```
+   This writes `results/lab7/try-last-36/`, with WikiText-2 columns unchanged. Point the ranking one-liner at `results/lab7/try-last-36/side_effects.json` to compare:
+
+   | Model | Ratio over all tokens | Ratio over the last 36 tokens |
+   |---|---:|---:|
+   | `l15only` | `3.52` | `12.96` |
+   | `p50c` | `2.49` | `6.31` |
+   | `rebuilt` | `2.48` | `6.18` |
+   | `all1` | `1.28` | `2.07` |
+   | `randcols` | `0.86` | `0.49` |
+   | `randdir` | `0.81` | `0.34` |
+
+   Every edit along $u$ gets more selective, while the random controls get less selective. For `p50c`, productive KL barely moves (`0.0097` to `0.0093`), while looping KL more than doubles (`0.0241` to `0.0584`). This fits Lab 6, where the trigger $s(t)$ rises in the last 512 tokens before a loop and peaks inside the repetition ($s_{15}$ goes `0.096 -> 0.178 -> 0.424` from early in a looping answer, to just before the loop, to inside it).
+
+3. **Score less WikiText-2.** `--chunks 8` scores the first 8 chunks (8,192 tokens) instead of all 64:
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --chunks 8
+   ```
+   This writes `results/lab7/try-chunks-8/`. Base perplexity is `22.804` on these 8,192 tokens versus `20.009` on all 65,536, which is fine because every model is compared with the base model on the same tokens. The WikiText KL ranking matches the 64-chunk run except that `p50c` and `rebuilt` (tied to 4 decimal places) swap places. Perplexity changes are less stable: the four largest keep their order (`a12` `+12.21%`, `a-4` `+5.16%`, `a8` `+2.43%`, `l19only` `+1.47%`), while shifts under 1% shuffle (`p50c` drops from `+0.79%` to `+0.20%`, and `l15only` flips from `+0.05%` to `-0.54%`). Because KL averages non-negative per-position values, positions cannot cancel out, which helps explain why its ranking held up. A perplexity change subtracts two nearly equal averages, so when it is under 1%, even its sign depends on which text you score. The run took 70 seconds versus 78 for 64 chunks: scoring the text is the fast part.
+
+4. **Score Lab 3's rebuild experiments (`try-lab3-tries`).** After building the three Lab 3 experiment models (`rebuilt-pairs2`, `rebuilt-windows36-36`, and `rebuilt-layers15-19-21`), compare them against the main 4-pair rebuild:
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models rebuilt rebuilt-pairs2 rebuilt-windows36-36 rebuilt-layers15-19-21 --try-name lab3-tries
+   ```
+   Because `rebuilt-layers15-19-21` also edits layer 21, the script swaps `down_proj` at layers `[15, 19, 21]` and writes `results/lab7/try-lab3-tries/side_effects.json` (base perplexity `20.009`):
+
+   | Model | Perplexity | Change | KL WikiText | KL productive | KL looping |
+   |---|---:|---:|---:|---:|---:|
+   | `rebuilt` | `20.167` | `+0.79%` | `0.0333` | `0.0096` | `0.0238` |
+   | `rebuilt-pairs2` | `20.096` | `+0.43%` | `0.0194` | `0.0075` | `0.0109` |
+   | `rebuilt-windows36-36` | `20.151` | `+0.71%` | `0.0326` | `0.0096` | `0.0241` |
+   | `rebuilt-layers15-19-21` | `20.307` | `+1.49%` | `0.0394` | `0.0114` | `0.0280` |
+
+   All four models score between `61.3%` and `62.3%` on `dev256`, yet their side effects separate cleanly. Using only 2 pairs (`rebuilt-pairs2`) produces a gentler, less selective edit (`+0.43%` perplexity, looping-to-productive KL ratio `1.46` vs `2.48`), while widening the neuron window to 36 tokens (`rebuilt-windows36-36`) changes almost nothing. Adding layer 21 (`rebuilt-layers15-19-21`) nearly doubles the WikiText-2 perplexity cost (`+1.49%` vs `+0.79%`) without helping accuracy (`61.3%` vs `61.5%`).
+
+5. **Score your main capstone edits (after Lab 8).** Lab 8 adds two models built from the unedited model's 225 loops at $\alpha = 6$: `capA-a6` (loop minus just before) and `capB-a6` (loop minus well before). Compare them with the released edit and the most selective edit:
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models p50c l15only capA-a6 capB-a6
+   ```
+   This writes `results/lab7/try-models-p50c-l15only-capA-a6-capB-a6/`. Our output:
+   ```text
+   models: ['base', 'p50c', 'l15only', 'capA-a6', 'capB-a6']
+   model           ppl  ppl +%   KL wiki   KL prod   KL loop
+   base         20.009    0.00   0.00000   0.00000   0.00000
+   p50c         20.167    0.79   0.03327   0.00967   0.02405
+   l15only      20.020    0.05   0.03476   0.00550   0.01938
+   capA-a6      20.111    0.51   0.01778   0.01025   0.01624
+   capB-a6      20.194    0.92   0.02711   0.01166   0.02078
+   wrote /nvme/wsp-lab/code/results/lab7/try-models-p50c-l15only-capA-a6-capB-a6 in 34 s
+   ```
+   Both capstone edits move productive texts more than the released edit does, and looping texts less. Their selectivity ratios are `1.58` and `1.78` versus `2.49` for `p50c`. In Lab 8, `capA-a6` also helps less: it scores `43.0%` on the dev set with `52.5%` of answers cut off, versus `61.5%` accuracy and `19.7%` cut for `p50c` (`capB-a6` reaches `58.2%` accuracy with `30.7%` cut).
+
+6. **Score Lab 8's capstone experiments (`try-lab8-tries`).** Lab 8 tests building the capstone edits from only the first 16 loops (`capA-max-samples-16`, `capB-max-samples-16`) or raising the strength from $\alpha = 6$ to $\alpha = 8$ on all 225 loops (`capA-alpha-8`, `capB-alpha-8`):
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models capA-max-samples-16 capB-max-samples-16 capA-alpha-8 capB-alpha-8 --try-name lab8-tries
+   ```
+   This writes `results/lab7/try-lab8-tries/side_effects.json` in about 33 seconds. Here are those four models alongside the main $\alpha = 6$ capstone edits (`capA-a6` and `capB-a6`, base perplexity `20.009`):
+
+   | Model | Loops used | $\alpha$ | `dev256` accuracy % | Perplexity | Change | KL WikiText | KL productive | KL looping |
+   |---|---:|---:|---|---:|---:|---:|---:|---:|
+   | `capA-a6` (main) | `225` | `6` | `43.0 +/- 2.6` | `20.111` | `+0.51%` | `0.0178` | `0.0103` | `0.0162` |
+   | `capA-max-samples-16` | `16` | `6` | `17.6 +/- 1.9` | `20.029` | `+0.10%` | `0.0081` | `0.0063` | `0.0086` |
+   | `capA-alpha-8` | `225` | `8` | `42.8 +/- 2.8` | `20.331` | `+1.61%` | `0.0298` | `0.0183` | `0.0265` |
+   | `capB-a6` (main) | `225` | `6` | `58.2 +/- 2.9` | `20.194` | `+0.92%` | `0.0271` | `0.0117` | `0.0208` |
+   | `capB-max-samples-16` | `16` | `6` | `53.7 +/- 2.8` | `20.021` | `+0.06%` | `0.0204` | `0.0102` | `0.0173` |
+   | `capB-alpha-8` | `225` | `8` | `54.5 +/- 2.9` | `20.562` | `+2.76%` | `0.0468` | `0.0206` | `0.0340` |
+
+   Raising $\alpha$ from `6` to `8` roughly triples the WikiText-2 perplexity cost (`+0.51%` to `+1.61%` for recipe A, `+0.92%` to `+2.76%` for recipe B) and nearly doubles the KL shifts (`0.0178` to `0.0298` and `0.0271` to `0.0468` on WikiText; `0.0103` to `0.0183` and `0.0117` to `0.0206` on productive texts). Yet `dev256` accuracy does not improve: recipe A scores `42.8%` vs `43.0%` (paired change `-0.2 +/- 2.3` points), and recipe B scores `54.5%` vs `58.2%` (paired drop `-3.7 +/- 1.6` points). Pushing harder along an imperfect direction scales the off-target part of $u$ along with the useful part. Using only 16 loops (`capB-max-samples-16`) keeps ordinary-text side effects tiny (`+0.06%` perplexity, `0.0204` WikiText KL) while reaching `53.7%` on `dev256` (`+41.6` points over the unedited model).
+
+   `capA-max-samples-16` shows why you should not read this table on its own. It has some of the smallest side effects here (`+0.10%` perplexity, `0.0081` WikiText KL), yet it reaches only `17.6%` on `dev256` (`5.5 +/- 1.9` points above the unedited model) with `82.6%` of answers still cut off. By comparison, `all2` has a similar WikiText KL (`0.0078`) and reaches `49.4%`. Small side effects are good news only when the edit also does its job. Read side effects next to the accuracy column. [Lab 8](lab8_capstone.md) explains why this edit fails.
+

code/labs/lab7_side_effects.py

@@ -0,0 +1,175 @@
+#!/usr/bin/env python
+"""Lab 7: what else does the edit change?
+
+    . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+    CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py          # a few minutes on one GPU
+
+For the base model and every edited model built in models/:
+  1. Ordinary text. Perplexity on the first 64 x 1,024 tokens of WikiText-2 (test), and KL(base || edited):
+     how far the edited model's next-token prediction moves from the base model's, averaged over tokens.
+  2. The agent's 8 texts. KL on its productive texts vs its looping texts. A good edit moves predictions a lot
+     where the model loops and barely anywhere else.
+Every edited model differs from the base only in the down_proj weights of the layers its edit_note.json lists
+(15 and 19 for most of them), so the script loads the base once and swaps those matrices in and out.
+
+Experiments. Any flag other than the defaults writes to results/lab7/try-NAME/ (figure included), so the main
+results stay as they are:
+    python labs/lab7_side_effects.py --chunks 8                   # quick check on 8 x 1,024 WikiText tokens
+    python labs/lab7_side_effects.py --last 36                    # KL on only the last 36 tokens of the 8 texts
+    python labs/lab7_side_effects.py --models p50c capA-a6        # only some models (after Lab 8: your own edit)
+"""
+import argparse
+import json
+import os
+import sys
+import time
+
+import numpy as np
+import torch
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+from wsp import hf  # noqa: E402
+from wsp.edits import NAME, load  # noqa: E402
+
+ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ap.add_argument("--models", nargs="+", metavar="NAME",
+                help="only these edited models, named like their folders without 'qwen35-' (default: all of them)")
+ap.add_argument("--chunks", type=int, default=64, help="how many 1,024-token WikiText-2 chunks to score (default 64)")
+ap.add_argument("--last", type=int, default=0,
+                help="measure KL on the 8 texts over only their last N tokens (default 0: every token)")
+ap.add_argument("--try-name", help="folder name for an experiment; results go to results/lab7/try-NAME/")
+a = ap.parse_args()
+
+t_start = time.time()
+CHUNK = 1024
+parts = []
+if a.chunks != 64:
+    parts.append(f"chunks-{a.chunks}")
+if a.last:
+    parts.append(f"last-{a.last}")
+if a.models:
+    parts.append("models-" + ("-".join(a.models) if len(a.models) <= 4 else str(len(a.models))))
+try_name = a.try_name or "-".join(parts)
+OUT = os.path.join(ROOT, "results", "lab7", f"try-{try_name}" if try_name else "")
+FIG = OUT if try_name else os.path.join(ROOT, "results", "figures")
+os.makedirs(OUT, exist_ok=True)
+os.makedirs(FIG, exist_ok=True)
+
+MODELS = os.path.join(hf.LAB, "models")
+found = sorted(d[len("qwen35-"):] for d in os.listdir(MODELS)
+               if d.startswith("qwen35-") and os.path.exists(os.path.join(MODELS, d, "edit_note.json")))
+missing = [n for n in a.models or [] if n not in found]
+if missing:
+    sys.exit(f"no edited model folder for {missing}; found {found}")
+
+
+def edited_layers(n):
+    note = json.load(open(os.path.join(MODELS, "qwen35-" + n, "edit_note.json")))
+    return {int(L) for L in note["spec"]["layers"]} if "spec" in note else {15, 19}  # p50c: the released edit
+
+
+# swap every layer that any chosen model edits; a model that leaves one of them alone just carries the base weights
+LAYERS = sorted(set().union(*(edited_layers(n) for n in a.models or found)))
+m, tok = hf.load("base")
+mods = {L: hf.down_proj(m, L) for L in LAYERS}
+base_w = {L: mods[L].weight.detach().clone() for L in LAYERS}
+variants = {}
+for n in a.models or found:
+    t = load(os.path.join(MODELS, "qwen35-" + n), [NAME.format(L) for L in LAYERS])
+    variants[n] = {L: t[NAME.format(L)].to("cuda") for L in LAYERS}
+print("models:", ["base"] + list(variants), "| down_proj swapped at layers", LAYERS)
+
+
+def set_weights(w):
+    for L in LAYERS:
+        mods[L].weight.data.copy_(w[L])
+
+
+from datasets import load_dataset  # noqa: E402
+
+wiki = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1", split="test")
+ids = tok("".join(wiki["text"]), add_special_tokens=False).input_ids
+if a.chunks > len(ids) // CHUNK:
+    sys.exit(f"WikiText-2 test has only {len(ids) // CHUNK} chunks of {CHUNK} tokens")
+chunks = torch.tensor(ids[:CHUNK * a.chunks]).view(a.chunks, CHUNK)
+pairs = json.load(open(os.path.join(ROOT, "labs", "data", "astra_pairs.json")))
+pair_ids = [hf.encode(tok, t) for t in pairs["texts"]]
+
+names = ["base"] + list(variants)
+nll = {n: [] for n in names}
+kl = {n: [] for n in names}
+kl_pairs = {n: {"productive": [], "looping": []} for n in names}
+
+
+def logprobs(x):
+    return m(x).logits[0, :-1].float().log_softmax(-1)
+
+
+def kl_text(lb, lv):
+    k = (lb.exp() * (lb - lv)).sum(-1)
+    return (k[-a.last:] if a.last else k).mean().item()
+
+
+with torch.no_grad():
+    for c in range(a.chunks):
+        x = chunks[c:c + 1].cuda()
+        set_weights(base_w)
+        lb = logprobs(x)
+        tgt = x[0, 1:, None]
+        nll["base"].append(-lb.gather(1, tgt).mean().item())
+        kl["base"].append(0.0)
+        for n, w in variants.items():
+            set_weights(w)
+            lv = logprobs(x)
+            nll[n].append(-lv.gather(1, tgt).mean().item())
+            kl[n].append((lb.exp() * (lb - lv)).sum(-1).mean().item())
+    for i, x in enumerate(pair_ids):
+        kind = "looping" if i % 2 else "productive"
+        set_weights(base_w)
+        lb = logprobs(x)
+        kl_pairs["base"][kind].append(0.0)
+        for n, w in variants.items():
+            set_weights(w)
+            lv = logprobs(x)
+            kl_pairs[n][kind].append(kl_text(lb, lv))
+set_weights(base_w)
+
+res = {}
+for n in names:
+    res[n] = {"wikitext_ppl": float(np.exp(np.mean(nll[n]))),
+              "wikitext_kl": float(np.mean(kl[n])),
+              "kl_productive_texts": float(np.mean(kl_pairs[n]["productive"])),
+              "kl_looping_texts": float(np.mean(kl_pairs[n]["looping"]))}
+for n in names:
+    res[n]["ppl_change_pct"] = 100 * (res[n]["wikitext_ppl"] / res["base"]["wikitext_ppl"] - 1)
+json.dump({"tokens": CHUNK * a.chunks, "last": a.last, "models": res},
+          open(os.path.join(OUT, "side_effects.json"), "w"), indent=1)
+print(f"{'model':10s} {'ppl':>8s} {'ppl +%':>7s} {'KL wiki':>9s} {'KL prod':>9s} {'KL loop':>9s}")
+for n in names:
+    r = res[n]
+    print(f"{n:10s} {r['wikitext_ppl']:8.3f} {r['ppl_change_pct']:7.2f} {r['wikitext_kl']:9.5f} "
+          f"{r['kl_productive_texts']:9.5f} {r['kl_looping_texts']:9.5f}")
+
+import matplotlib  # noqa: E402
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+
+order = [n for n in names if n != "base"]
+fig, ax = plt.subplots(1, 2, figsize=(14, 4.5))
+ax[0].bar(order, [res[n]["ppl_change_pct"] for n in order], color="#0969da")
+ax[0].set_ylabel("WikiText-2 perplexity change vs base (%)")
+ax[0].set_title("Damage to ordinary text")
+xi = np.arange(len(order))
+ax[1].bar(xi - 0.2, [res[n]["kl_productive_texts"] for n in order], 0.4, label="productive texts", color="#1a7f37")
+ax[1].bar(xi + 0.2, [res[n]["kl_looping_texts"] for n in order], 0.4, label="looping texts", color="#cf222e")
+ax[1].set_xticks(xi, order)
+ax[1].set_yscale("log")
+ax[1].set_ylabel("KL(base || edited), mean per token")
+ax[1].set_title("Where the edit moves predictions")
+ax[1].legend()
+for a_ in ax:
+    a_.tick_params(axis="x", rotation=60)
+fig.tight_layout()
+fig.savefig(os.path.join(FIG, "lab7_side_effects.png"), dpi=130)
+print(f"wrote {OUT} in {time.time() - t_start:.0f} s")

code/labs/lab8_capstone.md

@@ -0,0 +1,336 @@
+# Lab 8 (capstone): build your own edit from the model's own loops
+
+**Big thing:** With no hand-written texts, the unedited model's own looping answers give a loop direction close to the agent's. A weight edit built from them ("well before the loop", recipe B) reaches `58.2%` accuracy on 256 unseen practice problems, against `61.5%` for the released edit.
+
+**Why it matters:** In Lab 3, we rebuilt the released edit from 8 short math scratchpads that the AI agent wrote and labeled by hand. Here you extract the direction and the 64 neurons per layer from the base model's failed answers on 128 practice problems, check your direction with four diagnostics before editing any weights, and see why the comparison window inside each answer changes unseen accuracy by `15.2` percentage points.
+
+**Time and hardware:** About 5 minutes on 1 NVIDIA A100 GPU to extract the edits, 1 minute on CPU to build the edited models, and 30 minutes on 2 A100 GPUs (one run per GPU) to test both on `dev256`.
+
+**Theory guide:** Sections 9 and 13.3.
+
+## Before you run: predict
+
+Write down a short guess for each question before you run the script.
+
+1. For each looping answer, we compare a 36-token window (a token is a word piece) inside the loop against an earlier 36-token window from the same answer. Recipe A ("just before the loop") ends 32 tokens before verbatim repetition begins. Recipe B ("well before the loop") sits halfway between the start of the answer and the start of the loop. Which recipe will produce a stronger edit on unseen problems, and why?
+2. The AI agent needed only 4 pairs of hand-written texts to find its direction. Cosine similarity measures how closely two directions align. With the model's own noisy, long answers, roughly how many random looping answers must we average before the direction $u$ settles down (say, cosine similarity above `0.95` with the direction from a held-out set of loops, kept separate from those used to estimate the direction)? Will recipe A need more answers than recipe B?
+3. In Lab 3, neuron 11 of layer 19 had a score nearly five times larger than any other neuron on the agent's 8 short texts. When we score all 3,584 neurons on 225 long math answers from the model itself, will neuron 11 of layer 19 still rank first?
+
+## Run it
+
+Connect to `sampler`, which holds the answer texts for `runs/build/base` and the `dev/*` runs.
+
+Run the four main commands from `labs/lab8_capstone.py`:
+
+```bash
+ssh sampler
+. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+nvidia-smi
+CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py                          # about 5 minutes on one GPU
+CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/capstone.json   # CPU, about a minute
+python scripts/run_jobs.py plans/extra.json --gpus 6,7 --only 'dev/cap*'    # about 30 minutes, one run per GPU
+python scripts/summarize.py /nvme/wsp-lab/runs                               # writes /nvme/wsp-lab/summary.md
+```
+
+`scripts/run_jobs.py` skips any shard with a `.done.json` marker. To generate `dev/capA-a6` and `dev/capB-a6` fresh instead of keeping our finished runs, move `/nvme/wsp-lab/runs/dev/capA-a6` and `/nvme/wsp-lab/runs/dev/capB-a6` aside first.
+
+What each step writes:
+
+- `labs/lab8_capstone.py` reads the base model's answers in `/nvme/wsp-lab/runs/build/base` (`build128`, 128 practice problems never used for testing). It writes `results/lab8/capstone.json`, `results/lab8/example_answer.json`, `configs/capstone.json` (defining `capA-a6` and `capB-a6`), and three figures: `results/figures/lab8_loop_onset.png`, `results/figures/lab8_data_needed.png`, and `results/figures/lab8_one_answer.png`.
+- `scripts/build_models.py --variants configs/capstone.json` writes two edited model folders in `/nvme/wsp-lab/models/`: `qwen35-capA-a6` and `qwen35-capB-a6`.
+- `scripts/run_jobs.py plans/extra.json --gpus 6,7 --only 'dev/cap*'` generates 2 answers per problem across the 256 unseen problems of `dev256` (`512` answers per model; `dev/capA-a6` took `31.8` minutes and `dev/capB-a6` took `26.1` minutes) and writes `merged.jsonl` and `summary.json` under `/nvme/wsp-lab/runs/dev/capA-a6` and `/nvme/wsp-lab/runs/dev/capB-a6`.
+- `scripts/summarize.py` updates the summary table.
+
+Two more scripts fill in the rest of the section 3 table. `labs/lab4_endings.py` sorts answers by how they ended ([Lab 4](lab4_dose.md) explains the four groups), and `labs/lab7_side_effects.py` measures side effects on WikiText-2 and the agent's 8 texts ([Lab 7](lab7_side_effects.md)). Run both on `sampler`:
+
+```bash
+CUDA_VISIBLE_DEVICES="" python labs/lab4_endings.py dev/base dev/p50c dev/capA-a6 dev/capB-a6 --out lab8/endings   # CPU, a few seconds per run
+CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models p50c l15only capA-a6 capB-a6                     # under a minute on one GPU
+```
+
+`--out lab8/endings` writes `results/lab8/endings.json`, and the side-effects run writes `results/lab7/try-models-p50c-l15only-capA-a6-capB-a6/`.
+
+Passing any non-default flag to `labs/lab8_capstone.py` (such as `--max-samples 16`, `--min-before 2048`, or `--alpha 8`) writes to `results/lab8/try-<flags>/` (`capstone.json`, `example_answer.json`, `variant.json`, `plan.json`, and three figures) without touching the main files or `configs/capstone.json`. Full workflow for an experiment:
+
+```bash
+CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --max-samples 16          # about 5 minutes on one GPU
+CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants results/lab8/try-max-samples-16/variant.json   # CPU
+python scripts/run_jobs.py results/lab8/try-max-samples-16/plan.json --gpus 0,1   # about 30 minutes, one run per GPU
+CUDA_VISIBLE_DEVICES="" python labs/lab4_endings.py dev/capA-max-samples-16 dev/capB-max-samples-16 --out lab8/try-max-samples-16/endings
+CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models capA-max-samples-16 capB-max-samples-16 capA-alpha-8 capB-alpha-8 --try-name lab8-tries
+```
+
+(`--alpha 8` works the same way in `try-alpha-8`.) We ran both `lab8_capstone.py` tries side by side on GPUs 6 and 7, then built both models and ran each plan on two GPUs. The `dev256` runs took `28 to 33` minutes each, except `dev/capA-max-samples-16`, which took `43.5` minutes because most answers ran to the 32,768-token cap (mean `28,525` tokens). The Lab 7 check of all four models took `33` seconds (`results/lab7/try-lab8-tries/`).
+
+## What you should see
+
+### 1. Loops found in `build128` and the extracted directions
+
+Running `labs/lab8_capstone.py` prints the loop counts, the percentiles of loop period and onset token, the diagnostics for both recipes, and the onset profile:
+
+```text
+256 answers in /nvme/wsp-lab/runs/build/base: 230 hit the token cap, 225 of those end in a verbatim loop
+using 225 loops (loop starts at least 512 answer tokens in)
+loop period, 10/25/50/75/90th percentile: [18.0, 27.0, 44.0, 78.0, 144.0] tokens
+loop starts at answer token, same percentiles: [2813.4, 4021.0, 5822.0, 8013.0, 10953.8]
+recipe A (just before the loop), layer 15: cos with released u +0.722; released neurons picked 38/64; split-half cos 0.978; agent's 4 pairs, looping minus productive along u: [0.78, 0.88, 1.69, 1.16]
+   answers needed: [(1, 0.6), (2, 0.628), (4, 0.712), (8, 0.841), (16, 0.919), (32, 0.952), (64, 0.97), (112, 0.978)]
+recipe A (just before the loop), layer 19: cos with released u +0.724; released neurons picked 28/64; split-half cos 0.978; agent's 4 pairs, looping minus productive along u: [1.15, 1.27, 2.4, 1.73]
+recipe B (well before the loop), layer 15: cos with released u +0.782; released neurons picked 42/64; split-half cos 0.988; agent's 4 pairs, looping minus productive along u: [0.9, 1.04, 1.7, 1.25]
+   answers needed: [(1, 0.676), (2, 0.781), (4, 0.853), (8, 0.912), (16, 0.958), (32, 0.97), (64, 0.984), (112, 0.988)]
+recipe B (well before the loop), layer 19: cos with released u +0.773; released neurons picked 28/64; split-half cos 0.989; agent's 4 pairs, looping minus productive along u: [1.32, 1.5, 2.35, 1.83]
+cos between recipe A's u and recipe B's u: {'15': 0.951, '19': 0.959}
+```
+
+The loop finder works on tokens: an answer counts if its tokens end in one block repeated back to back at least 3 times over at least 64 tokens. It finds `225` of the `230` cut answers; the summary table's word-level check counts `222` of `230` (`96.5%`), a 3-answer difference. In `results/lab8/capstone.json`, all `225` loops start at least 512 tokens into the answer, so all `225` are used. The loop period (the length in tokens of one repeated block) has 10th, 25th, 50th, 75th, and 90th percentiles of `18, 27, 44, 78, 144` tokens. The answer token where the loop starts has percentiles of `2,813; 4,021; 5,822; 8,013; 10,954`.
+
+How the two window recipes compare against the released edit (`p50c`) and each other:
+
+| Recipe (comparison window) and layer | Cosine with released $u$ | Released neurons picked (of 64) | Split-half cosine (stability) | Gap size ($\|\overline{\Delta h}\|$) | Agent's 4 pairs: looping minus productive along $u$ | Top neuron (score, next score) |
+|---|---|---|---|---|---|---|
+| Recipe A (just before the loop), layer 15 | `0.722` | `38 / 64` | `0.978` | `0.689` | `+0.78, +0.88, +1.69, +1.16` | `46` (`0.0120`, next `294` at `0.0108`) |
+| Recipe A (just before the loop), layer 19 | `0.724` | `28 / 64` | `0.978` | `1.037` | `+1.15, +1.27, +2.40, +1.73` | `11` (`0.121`, next `171` at `0.024`) |
+| Recipe B (well before the loop), layer 15 | `0.782` | `42 / 64` | `0.988` | `1.418` | `+0.90, +1.04, +1.70, +1.25` | `46` (`0.0261`, next `0` at `0.0187`) |
+| Recipe B (well before the loop), layer 19 | `0.773` | `28 / 64` | `0.989` | `2.234` | `+1.32, +1.50, +2.35, +1.83` | `11` (`0.210`, next `1` at `0.045`) |
+
+Four patterns in these numbers:
+
+- **Agreement with each other and with the released edit:** Both recipes give nearly the same unit direction (a length-1 vector in the 1,024-number stream; cosine similarity is the dot product of two unit vectors, where `1` means the same direction and `0` means at right angles). Their mutual cosine `cos_A_B` is `0.951` at layer 15 and `0.959` at layer 19, yet recipe B has more than double the raw gap size (the length of the average stream difference) and sits closer to the released edit's direction. The table above lists the gap sizes and cosines for each layer.
+- **Top neuron:** At layer 19, neuron `11` ranks first in both recipes with a score nearly five times the next neuron's. The table above lists both scores for each recipe. The AI agent switched off this same neuron in its 4th forward pass (request file `fwd_5` in `external/lm_understand-logs/exp8_math/runs/p50c/requests/`; an earlier failed request did not count), taking 32 practice problems from 3 to 5 correct (`+6.25` points, standard error `4.35`, which its notes call "weak positive, not confirmed"). At layer 15, both recipes' top three neurons (`46, 294, 0`) are among Lab 3's top four (`294, 371, 0, 46`).
+- **Transfer to the agent's 8 hand-written texts:** Both recipes were built only from the base model's long math loops, yet projecting the last 36 tokens of all 4 hand-written pairs (`labs/data/astra_pairs.json`) onto $u$ still separates the looping text from the productive text in every pair. The table above lists the gaps along the direction for each pair.
+- **Data needed:** At layer 15, the average cosine similarity between a direction from $N$ random answers and a direction from a held-out half (`113` answers) climbs fast:
+
+| Answers used ($N$) | `1` | `2` | `4` | `8` | `16` | `32` | `64` | `112` |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| Recipe A (just before the loop), layer 15 | `0.60` | `0.63` | `0.71` | `0.84` | `0.92` | `0.95` | `0.97` | `0.978` |
+| Recipe B (well before the loop), layer 15 | `0.68` | `0.78` | `0.85` | `0.91` | `0.96` | `0.97` | `0.98` | `0.988` |
+
+![Cosine against the number of answers used to build u, at layers 15 and 19, for recipes A (red) and B (purple). Solid lines compare with the other half of the answers and dashed lines with the released u.](../results/figures/lab8_data_needed.png)
+
+### 2. When the loop starts in the residual stream
+
+Aligning all `225` loops at the token where verbatim repetition begins (`token 0`) and averaging across answers shows when the residual stream and the 64 released neurons start drifting:
+
+| Quantity (mean over 225 loops) | `2,048 to 1,024` tokens before | `512 to 128` tokens before | Last `128` tokens before (`-128..0`) | First `128` tokens inside (`0..128`) | `128 to 256` tokens inside |
+|---|---:|---:|---:|---:|---:|
+| Stream along released $u$, layer 15 | `1.14` | `1.39` | `1.58` | `1.95` | `2.20` |
+| Stream along released $u$, layer 19 | `2.27` | `2.66` | `2.95` | `3.50` | `3.96` |
+| Trigger $s(t)$ (released 64 neurons), layer 15 | `0.135` | `0.173` | `0.206` | `0.288` | `0.366` |
+| Trigger $s(t)$ (released 64 neurons), layer 19 | `0.183` | `0.237` | `0.277` | `0.376` | `0.512` |
+
+Both the stream projection along $u$ and the 64 neurons' push $s(t) = u \cdot (W_S a_S(t))$ (the Lab 6 trigger measuring how hard those 64 neurons push along $u$ at token $t$) rise hundreds of tokens before repetition starts, then jump at token `0`.
+
+![Median and middle half of 225 loops lined up at the token where each loop starts. The stream along the released u and the 64 neurons' push s(t) rise slowly at layers 15 and 19, then jump once the loop starts.](../results/figures/lab8_loop_onset.png)
+
+In `results/lab8/example_answer.json` and `lab8_one_answer.png`, the median-onset answer (`math_train_algebra_1625`, sample `1`) runs to `32,768` tokens, and its loop starts at answer token `5,822` with period `51` tokens. The three shaded 36-token windows show where each recipe reads the stream: window B covers tokens `2,893 to 2,929`, window A covers `5,754 to 5,790`, and the loop window covers `5,924 to 5,960` (starting two periods after token `5,822`).
+
+![One answer that loops from token 5,822: the stream along the released u and the push s(t) at layers 15 and 19, with shaded strips for window B, window A and the loop window.](../results/figures/lab8_one_answer.png)
+
+### 3. Test results on the 256 unseen problems (`dev256`)
+
+The table below combines `dev256` results (`256` unseen problems, `2` answers each = `512` answers per run), answer endings from `results/lab8/endings.json`, and side effects from `results/lab7/try-models-p50c-l15only-capA-a6-capB-a6/side_effects.json`. From Lab 7, perplexity ($e$ to the mean negative log-probability of the true next token; base model `20.009` on WikiText-2, lower is better) and KL divergence ($\text{KL}(\text{base} \parallel \text{edited})$, how far next-token probabilities shift from the unedited model, where `0` means identical) measure how much an edit disturbs ordinary text and the agent's 8 texts:
+
+| Model / edit | Accuracy % | Cut at cap % | Mean tokens | Gain vs unedited (pp), rescued / lost | Correct when finished % | Endings: cut / stopped / escaped loop (correct) / normal (correct) | Perplexity change | KL: WikiText / productive / looping |
+|---|---:|---:|---:|---|---:|---|---:|---|
+| `dev/base` (unedited) | `12.1 +/- 1.6` | `88.7` | `29,683` | baseline | `100.0` | `453 (4) / 0 / 3 (2) / 56 (56)` | `0%` (`20.009`) | `0 / 0 / 0` |
+| `capA-a6` (just before the loop, recipe A) | `43.0 +/- 2.6` | `52.5` | `22,699` | `+30.9 +/- 2.3`, `95 / 1` | `89.3` | `237 (3) / 1 (0) / 141 (94) / 133 (123)` | `+0.51%` | `0.0178 / 0.0103 / 0.0162` |
+| `capB-a6` (well before the loop, recipe B) | `58.2 +/- 2.9` | `30.7` | `19,899` | `+46.1 +/- 2.6`, `113 / 0` | `83.7` | `108 (0) / 0 / 145 (67) / 259 (231)` | `+0.92%` | `0.0271 / 0.0117 / 0.0208` |
+| `p50c` (released edit, alpha 6) | `61.5 +/- 2.8` | `19.7` | `16,104` | `+49.4 +/- 2.6`, `125 / 0` | `76.6` | `72 (0) / 0 / 96 (29) / 344 (286)` | `+0.79%` | `0.0333 / 0.0097 / 0.0241` |
+
+Paired per-problem differences across the 256 problems:
+
+- `capB-a6` beats `capA-a6` by `+15.2 +/- 2.2` percentage points.
+- `capB-a6` trails the released edit `p50c` by `-3.3 +/- 1.5` percentage points.
+
+An answer closed normally if it wrote `</think>` with no repeated snippet in the 6,000 characters before it ([Lab 4](lab4_dose.md)). Both capstone edits escape a running loop about equally often (`141` for recipe A, `145` for recipe B), while `capB-a6` nearly doubles normal endings (`259` vs `133`) and more than halves answers trapped at the cap (`108` vs `237`). A likely reading, though these counts do not prove it: recipe B also seems to stop more loops from starting. Rescued / lost in the table counts problems, relative to the unedited model.
+
+## Why it works
+
+### 1. From the model's own failures to a rank-one weight edit
+
+A language model reads and writes one token (word piece) at a time. At each token, it updates a 1,024-number residual stream across 24 layers (blocks). Each block has a token mixer and an MLP (multi-layer perceptron) whose output matrix `down_proj` has 3,584 columns, one per neuron. In Lab 3, each of the agent's 4 text pairs shared the same prompt and opening lines, so subtracting the productive text from the looping text canceled the problem topic.
+
+In `runs/build/base`, each answer cut at the 32,768-token cap starts with productive reasoning before falling into a verbatim loop (the same token block repeating to the cap). Comparing a window inside the loop against an earlier window from the same answer via teacher forcing (feeding the saved tokens through the model in one pass and recording activations with PyTorch hooks) cancels the problem topic within each answer.
+
+```mermaid
+flowchart LR
+  S["Start of answer<br/>(token n0)"] --> WB["Well before the loop (recipe B)<br/>(36 tokens at midpoint of n0..onset)"]
+  WB --> WA["Just before the loop (recipe A)<br/>(36 tokens ending 32 tokens before onset)"]
+  WA --> ON["Loop starts<br/>(token onset, period p)"]
+  ON --> WL["Loop window: inside the loop<br/>(36 tokens starting at onset + 2p)"]
+```
+
+For every looping answer whose loop starts at least `512` tokens into the answer (`on - n0 >= 512`), `labs/lab8_capstone.py` slices three 36-token windows (`WIN = 36`):
+
+- **Inside the loop (`loop`):** 36 tokens starting two periods after loop onset (`on + 2 * p`), once verbatim repetition is underway.
+- **Just before the loop (recipe A):** 36 tokens ending `GAP = 32` tokens before loop onset (`on - 68` to `on - 32`).
+- **Well before the loop (recipe B):** 36 tokens centered halfway between the start of the answer and loop onset (`n0 + (on - n0) // 2 - 18` to `+ 18`).
+
+Because the model is causal (each token reads only earlier tokens), the script truncates each forward pass right after the needed windows (`end = max(w["loop"][1], min(on + 256, len(ids)))`) instead of running all `32,768` tokens. That finishes all 225 answers in about 5 minutes on one GPU.
+
+At layers $L \in \{15, 19\}$, the remaining steps follow Lab 3:
+
+1. **Direction $u$:** For each answer $i$ and recipe $k \in \{A, B\}$, subtract the mean residual stream leaving block $L$ over window $k$ from the mean over the loop window: $\Delta h_{i, L}^{(k)} = \bar{h}_{i, L}^{\text{loop}} - \bar{h}_{i, L}^{(k)}$. Average $\Delta h_{i, L}^{(k)}$ across all 225 answers and scale to unit length:
+   $$u_L^{(k)} = \frac{\overline{\Delta h}_L^{(k)}}{\|\overline{\Delta h}_L^{(k)}\|}$$
+2. **Neuron scores:** Over the last `AWIN = 24` tokens of each window, find the mean activation difference $\overline{\Delta a}_L^{(k)}[j]$ (loop window minus comparison window) for each of the 3,584 MLP neurons. Multiply by how much column $j$ of `down_proj` writes along $u_L^{(k)}$:
+   $$\text{score}_j = \left(u_L^{(k)} \cdot W_{\text{down}, L}[:, j]\right) \times \overline{\Delta a}_L^{(k)}[j]$$
+3. **Rank-one edit on the top 64 columns:** Sort the 3,584 neurons by $\text{score}_j$ descending, keep the top 64 indices $S$, and apply the rank-one update (one column vector $u$ times one row vector $u^T W_S$) $W' = W - \alpha \, u \, (u^T W_S)$ with edit strength $\alpha = 6$. This scales those 64 neurons' push along $u$ by $1 - \alpha = -5$.
+
+```mermaid
+flowchart LR
+  R["225 looping answers<br/>from build128 (runs/build/base)"] --> W["Take 36-token windows:<br/>loop minus window A or B"]
+  W --> U["Average stream difference<br/>across answers -> unit vector u"]
+  U --> N["Score 3,584 neurons:<br/>(u . W[:, j]) * mean(da_24)<br/>Keep top 64 at alpha = 6"]
+  N --> T["Build edited models and test<br/>on 256 unseen problems (dev256)"]
+```
+
+### 2. Why "well before the loop" beats "just before the loop"
+
+Recipe A and recipe B have cosine similarity `0.951` at layer 15 and `0.959` at layer 19, and both pick `28 of 64` released neurons at layer 19 (`38` vs `42` at layer 15). Yet recipe B beats recipe A on `dev256` by `15.2` percentage points (`58.2%` vs `43.0%`).
+
+In `lab8_loop_onset.png` and Lab 6 Part 3, hundreds of tokens before exact repetition starts at token `0`, the model is already re-checking the same numbers and drifting toward the loop:
+
+- At layer 15, the stream projection along the released $u$ climbs from `1.14` (`2,048 to 1,024` tokens before onset) to `1.58` in the last `128` tokens before onset, then jumps to `2.20` inside the loop.
+- In Lab 6 Part 3, the 64 neurons' push $s_{15}(t)$ nearly doubled from `0.096` in the early half of the pre-loop answer to `0.178` in the last 512 tokens before onset, then jumped to `0.424` inside the loop.
+
+Window A (`-68` to `-32` tokens before onset) sits inside that late drift zone, where the stream has already moved about halfway to the loop. When recipe A computes `stream(in the loop) - stream(just before the loop)`, it subtracts out the early drift and keeps mostly the final jump into verbatim repetition. Three results reflect this:
+
+1. **The raw gap vector is less than half as long:** Recipe A's $\|\overline{\Delta h}\|$ is `0.689` vs `1.418` for recipe B at layer 15 (`1.037` vs `2.234` at layer 19). Because part of the loop signal cancels out, background token differences between the two windows make up a larger share of the normalized unit vector $u_A$, lowering its cosine with the released $u$ (`0.722` vs `0.782`) and its layer-15 neuron overlap (`38` vs `42`).
+2. **Smaller neuron activation gaps:** Because loop neurons are already firing harder in the last 68 pre-loop tokens, subtracting window A's activations shrinks $\overline{\Delta a}_{24}$ on the neurons that build up the drift (the `score` of neuron 11 at layer 19 drops from `0.210` in B to `0.121` in A, and neuron 46 at layer 15 drops from `0.0261` to `0.0120`).
+3. **Fewer loops versus more escapes on `dev256`:** Both edits escape a loop about equally often (`141` for recipe A, `145` for recipe B), while recipe B nearly doubles normal endings (`259` vs `133`). A likely reading, which the endings counts fit but do not prove, is that recipe B's direction includes the pre-loop drift, so its edit pushes back before repetition starts, whereas recipe A seems to act mainly once a loop is underway.
+
+### 3. What split-half stability and the data-needed curve measure
+
+To check that the direction reflects a shared property across problems instead of sample noise, the script runs 20 random split-half checks. Each trial shuffles the 225 loops into two non-overlapping halves of `112` and `113` answers ($H_1$ and $H_2$), computes a unit direction from each half, and takes their cosine similarity. Recipe B's halves agree at `0.988 +/- 0.003` (layer 15) and `0.989 +/- 0.003` (layer 19), and recipe A's agree at `0.978` at both layers. Different loops point along nearly the same direction in the 1,024-dimensional stream.
+
+To measure how many answers are needed, the script draws $N \in \{1, 2, 4, 8, 16, 32, 64, 112\}$ answers from $H_1$ and measures the cosine of their direction with the full held-out half $H_2$:
+
+```mermaid
+xychart-beta
+  title "Layer 15: recipe B (upper line) vs recipe A (lower line)"
+  x-axis "Answers used" ["1", "2", "4", "8", "16", "32", "64", "112"]
+  y-axis "Cosine with held-out half" 0.5 --> 1.0
+  line [0.600, 0.628, 0.712, 0.841, 0.919, 0.952, 0.970, 0.978]
+  line [0.676, 0.781, 0.853, 0.912, 0.958, 0.970, 0.984, 0.988]
+```
+
+At $N = 1$ (one looping answer), recipe B already has a `0.68` cosine with the held-out half. By $N = 16$, it reaches `0.958` (`0.919` for recipe A), and past `32` answers the curve is nearly flat (`0.970` to `0.988`). In `lab8_data_needed.png`, the dashed curve (cosine with the released $u$) levels off near `0.78` for recipe B and `0.72` for recipe A. Averaging more answers reduces sampling noise around the natural-loop direction, while the gap between `0.78` and `1.00` is a systematic difference between windows of long natural answers and the agent's 4 hand-written pairs.
+
+Because each `build128` problem has two answers, splitting by answer can put two answers to the same problem in opposite halves. To check whether that inflates agreement, we reran with `--split-by problem`, keeping both answers of each problem in the same half:
+
+```bash
+CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --split-by problem   # writes results/lab8/try-split-by-problem/
+```
+
+The 225 loops come from 120 problems (105 with two loops), giving halves of 108 to 117 loops. The split-half cosines barely change: `0.977 / 0.978` for recipe A at layers 15 / 19 (vs `0.978 / 0.978` by answer) and `0.988 / 0.987` for recipe B (vs `0.988 / 0.989`). Both data-needed curves shift by at most `0.068` at $N = 1$. From $N = 16$ on, held-out agreement shifts by under `0.006` and released-$u$ agreement by under `0.009`. When the half that the $N$ answers come from holds fewer than 112 loops, the final point uses all of them (`108` to `112` in `answers_used_min_max` in `capstone.json`). The edits themselves are unchanged because the split affects only these two checks.
+
+### 4. Why we keep `build128` separate from `dev256` and use paired differences
+
+Section 13.3 of the theory guide states a basic evaluation rule: test an edit on problems not used to build it. Here `build128` (`128` problems, `256` answers) is used only to extract $u$ and the 64 neurons, and `dev256` (`256` separate problems, `512` answers) is used only for testing.
+
+On `dev256`, `capB-a6` scores `58.2% +/- 2.9%` and `p50c` scores `61.5% +/- 2.8%`, where the standard error (SE) is the typical random error from testing a finite sample of problems (a gap of 2 SE or more is unlikely to be luck). Judging from the standalone SEs (`2.9%` and `2.8%`), the `3.3`-point gap looks like it could be chance variation. Because both models run on the same 256 problems, ranging from easy GSM8K arithmetic to hard competition algebra, problem-to-problem difficulty inflates both standalone SEs. Taking the paired difference on each problem $p$ (the fraction of sample answers solved by model 1 minus the fraction solved by model 2 on problem $p$) and computing the SE across those 256 differences cancels out shared problem difficulty:
+
+- `capB-a6` minus `p50c`: `-3.3 +/- 1.5` percentage points (more than 2 SE below `p50c`, so `capB-a6` is close to the released edit and measurably a little weaker).
+- `capB-a6` minus `capA-a6`: `+15.2 +/- 2.2` percentage points (nearly seven standard errors).
+
+## Experiments: sample count and edit strength
+
+We ran two experiment flags on `sampler` to test how the capstone edit behaves with fewer loops (`--max-samples 16`) or a stronger edit (`--alpha 8`).
+
+### 1. How the directions and weight changes compare
+
+- `--max-samples 16` (`results/lab8/try-max-samples-16/`) keeps the first `16` usable loops in file order (`used[:16]`), a file prefix instead of a random draw across all 128 problems. Because `build128` lists its 64 GSM8K problems before its 64 MATH problems, all 16 loops come from `9` GSM8K problems (`gsm8k_train_3030`, `3613`, `4033`, `1318`, `4077`, `6248`, `6350`, `6176`, `7193`). With 16 loops, the data-needed curve stops at `8` answers, and the split-half cosine compares two random halves of `8` loops each (vs `112` and `113` in the main run).
+- `--alpha 8` (`results/lab8/try-alpha-8/`) uses all `225` loops, so it keeps the main run's unit directions $u$ and 64 neurons and raises edit strength from $\alpha = 6$ to $\alpha = 8$. The relative weight change (Frobenius norm (the overall size) of $W' - W$ divided by Frobenius norm of $W$, layer 15 / layer 19) is `0.078 / 0.075` for `capA-max-samples-16`, `0.097 / 0.092` for `capB-max-samples-16`, `0.141 / 0.129` for `capA-alpha-8`, and `0.149 / 0.147` for `capB-alpha-8`.
+
+| Edit variant | Loops used | Alpha | Cosine with released $u$ (L15 / L19) | Split-half cosine (L15 / L19) | Released neurons picked (L15 / L19) |
+|---|---:|---:|---|---|---|
+| `capA-a6` (just before, main) | `225` | `6` | `0.722 / 0.724` | `0.978 / 0.978` | `38 / 28` |
+| `capA-max-samples-16` (just before, 16 loops) | `16` | `6` | `0.574 / 0.596` | `0.632 / 0.622` | `27 / 21` |
+| `capB-a6` (well before, main) | `225` | `6` | `0.782 / 0.773` | `0.988 / 0.989` | `42 / 28` |
+| `capB-max-samples-16` (well before, 16 loops) | `16` | `6` | `0.687 / 0.682` | `0.828 / 0.864` | `34 / 20` |
+
+With 16 loops from 9 GSM8K problems, recipes A and B still agree (`0.873` at layer 15 and `0.902` at layer 19, vs `0.951` and `0.959` on 225 loops), neuron 11 of layer 19 still ranks first in both, and both directions separate the agent's 8 texts in all 4 pairs (`+0.69, +0.83, +1.67, +1.10` for recipe B at layer 15).
+
+### 2. Evaluation on `dev256`, endings, and breakdown by source
+
+| Edit variant | Accuracy % | Cut % | Mean tokens | Correct when finished % | Gain vs unedited (pp), rescued / lost | Paired change vs same recipe at alpha 6 with all 225 loops | Perplexity change | KL: WikiText / productive / looping |
+|---|---:|---:|---:|---:|---|---|---:|---|
+| `capA-a6` (just before, main) | `43.0 +/- 2.6` | `52.5` | `22,699` | `89.3` | `+30.9 +/- 2.3`, `95 / 1` | baseline for recipe A | `+0.51%` | `0.0178 / 0.0103 / 0.0162` |
+| `capA-alpha-8` (just before, alpha 8) | `42.8 +/- 2.8` | `53.5` | `22,772` | `89.9` | `+30.7 +/- 2.5`, `87 / 4` | `-0.2 +/- 2.3` | `+1.61%` | `0.0298 / 0.0183 / 0.0265` |
+| `capA-max-samples-16` (just before, 16 loops) | `17.6 +/- 1.9` | `82.6` | `28,525` | `96.6` | `+5.5 +/- 1.9`, `34 / 17` | `-25.4 +/- 2.3` | `+0.10%` | `0.0081 / 0.0063 / 0.0086` |
+| `capB-a6` (well before, main) | `58.2 +/- 2.9` | `30.7` | `19,899` | `83.7` | `+46.1 +/- 2.6`, `113 / 0` | baseline for recipe B | `+0.92%` | `0.0271 / 0.0117 / 0.0208` |
+| `capB-alpha-8` (well before, alpha 8) | `54.5 +/- 2.9` | `34.0` | `20,870` | `81.7` | `+42.4 +/- 2.6`, `107 / 1` | `-3.7 +/- 1.6` | `+2.76%` | `0.0468 / 0.0206 / 0.0340` |
+| `capB-max-samples-16` (well before, 16 loops) | `53.7 +/- 2.8` | `39.6` | `20,725` | `88.7` | `+41.6 +/- 2.6`, `110 / 0` | `-4.5 +/- 1.8` | `+0.06%` | `0.0204 / 0.0102 / 0.0173` |
+| `p50c` (released edit, alpha 6) | `61.5 +/- 2.8` | `19.7` | `16,104` | `76.6` | `+49.4 +/- 2.6`, `125 / 0` | (alpha 8 vs 6: `-0.8 +/- 1.6`) | `+0.79%` | `0.0333 / 0.0097 / 0.0241` |
+
+Answer endings in `results/lab8/try-alpha-8/endings.json` and `results/lab8/try-max-samples-16/endings.json` (out of 512 answers, with correct counts in parentheses):
+
+- `capA-alpha-8`: `242 (5)` cut while thinking, `0` stopped while thinking, `130 (87)` escaped a loop, `140 (127)` closed normally (vs `capA-a6`: `237 (3)` cut, `1 (0)` stopped, `141 (94)` escaped a loop, `133 (123)` closed normally).
+- `capB-alpha-8`: `133 (1)` cut while thinking, `1 (0)` stopped while thinking, `96 (44)` escaped a loop, `282 (234)` closed normally (vs `capB-a6`: `108 (0)` cut, `0` stopped, `145 (67)` escaped a loop, `259 (231)` closed normally).
+- `capA-max-samples-16`: `422 (4)` cut while thinking (`96.0%` of its cut answers end in repetition, `94.8%` in a verbatim loop), `0` stopped, `38 (34)` escaped a loop, `52 (52)` closed normally.
+- `capB-max-samples-16`: `150 (1)` cut while thinking (`76.8%` of its cut answers end in repetition, `36.9%` in a verbatim loop), `0` stopped, `183 (107)` escaped a loop, `179 (167)` closed normally.
+
+Breakdown by problem source on `dev256` (`128` GSM8K and `128` MATH problems, `256` answers each):
+
+| Edit variant | GSM8K accuracy % / cut % | MATH accuracy % / cut % |
+|---|---|---|
+| `capA-a6` (just before, main) | `55.5 / 37.5` | `30.5 / 67.6` |
+| `capA-alpha-8` (just before, alpha 8) | `51.2 / 41.8` | `34.4 / 65.2` |
+| `capA-max-samples-16` (just before, 16 loops) | `23.8 / 76.2` | `11.3 / 89.1` |
+| `capB-a6` (well before, main) | `71.5 / 13.3` | `44.9 / 48.0` |
+| `capB-alpha-8` (well before, alpha 8) | `68.0 / 15.6` | `41.0 / 52.3` |
+| `capB-max-samples-16` (well before, 16 loops) | `69.1 / 20.3` | `38.3 / 59.0` |
+| `p50c` (released edit, alpha 6) | `74.6 / 4.7` | `48.4 / 34.8` |
+
+Using 16 loops instead of 225 costs recipe A `-31.6 +/- 3.4` points on GSM8K and `-19.1 +/- 2.9` on MATH (paired vs `capA-a6`), and costs recipe B `-2.3 +/- 2.7` on GSM8K and `-6.6 +/- 2.5` on MATH (paired vs `capB-a6`). All 16 loops came from GSM8K, which may explain why recipe B loses slightly more on MATH, though the difference between sources (about 4 points) is within noise.
+
+Two lessons from these runs:
+
+1. **Turning up $\alpha$ did not help either recipe.** Raising $\alpha$ from `6` to `8` leaves `capA` flat (`-0.2 +/- 2.3` points) and drops `capB` by `-3.7 +/- 1.6` points, while WikiText-2 perplexity damage grows from `+0.51%` and `+0.92%` at $\alpha = 6$ to `+1.61%` and `+2.76%` at $\alpha = 8$. Even the released edit gains nothing at $\alpha = 8$ (`-0.8 +/- 1.6` points, perplexity `+3.33%`). Raising $\alpha$ scales the whole update, including any part of $u$ that points the wrong way. With one stronger setting per recipe, we cannot tell whether a partly wrong direction or a strength already past its peak explains the drop.
+2. **A few loops are enough for recipe B ("well before the loop"), but not for recipe A ("just before the loop").** From 16 loops (9 problems), recipe B still reaches `53.7%` accuracy (`+41.6` points over the unedited model), `4.5 +/- 1.8` points below 225-loop `capB-a6` (`58.2%`), even though its direction is noisier (two halves of 8 loops agree at cosine `0.828` and `0.864`, against `0.988` and `0.989` with halves of 112 and 113 loops) and sits further from the released $u$. The direction table above lists its cosines with the released direction for both layers. Recipe A on the same 16 loops collapses to `17.6%` (`-25.4 +/- 2.3` points vs `capA-a6`), and `82.6%` of its answers are still cut. Its direction is the noisiest of all: two halves of 8 loops agree at only `0.632` and `0.622` at layers 15 and 19, and the direction table above lists its lower cosines with the released $u$. Its WikiText KL is `0.0081` (vs `0.0204` for `capB-max-samples-16` and `0.0178` for `capA-a6`), but [Lab 4](lab4_dose.md) and [Lab 7](lab7_side_effects.md) suggest that KL size does not explain the accuracy gap: random-direction control `randdir` (same 64 neurons, same size of weight change, random direction) has WikiText KL `0.0046` and gains `+2.3` points, while applying the agent's direction to all 3,584 neurons (`all2`) at $\alpha = 2$ has almost the same KL as `capA-max-samples-16` (`0.0078` vs `0.0081`) and gains `+37.3` points. This suggests that, in this setup, the gain depends on where the change points instead of how far it shifts WikiText predictions. Recipe B on 16 loops beats recipe A on 225 loops by `+10.7 +/- 2.1` points (paired), suggesting that window choice matters more than loop count here. One caveat: `--max-samples 16` takes the first 16 loops in file order (all from 9 GSM8K problems). In `data_needed` (`results/lab8/capstone.json`), random 16-loop draws agreed better with the released $u$ at layers `15 / 19` (`0.666 / 0.683` for A and `0.757 / 0.746` for B, vs `0.574 / 0.596` and `0.687 / 0.682` on the first 16), so that narrow sample may explain part of the drop. We did not build or test edits from random 16-loop sets. Data quality, window placement, and sample count trade off against one another.
+
+## Check yourself
+
+1. Why does `labs/lab8_capstone.py` take both the loop window and the baseline window (window A or window B) from the same answer, instead of averaging all looping answers and subtracting the average of the 26 non-looping answers in `build128`?
+<details><summary>Answer</summary>
+
+The 26 finished answers come from 18 problems, and 215 of the 225 loops come from other problems where neither answer finished. Subtracting the finished answers' average would mix problem differences (topic, numbers, problem type) into $u$ along with the loop signal. Taking both windows from the same answer cancels the prompt and topic within each example, as the shared opening did in Lab 3.
+</details>
+
+2. At layer 15, recipe A and recipe B have a cosine similarity of `0.951`, yet recipe B scores `58.2%` on `dev256` and recipe A scores `43.0%`. What happens in the residual stream during the last few hundred tokens before a loop that makes window A a worse baseline than window B?
+<details><summary>Answer</summary>
+
+The residual stream along $u$ and the 64 neurons' push $s(t)$ are already climbing hundreds of tokens before the verbatim loop starts (`stream15` rises from `1.14` well before the loop to `1.58` in the last 128 tokens before onset, and `2.20` inside). Window A sits 32 to 68 tokens before loop onset, so subtracting it removes much of the loop buildup, halves the gap norm $\|\overline{\Delta h}\|$ (`0.689` vs `1.418`), and likely yields an edit that reacts mainly after a loop starts instead of stopping the drift into a loop.
+</details>
+
+3. When you run `labs/lab8_capstone.py --max-samples 16`, recipe B keeps most of its accuracy (`53.7%` vs `58.2%`), while recipe A drops from `43.0%` to `17.6%`. Which diagnostic numbers from the 16-loop extraction and the main run's `data_needed` table warn you before running `dev256` that recipe A will suffer much more than recipe B, and why did the first 16 loops fall below the random-16 curve?
+<details><summary>Answer</summary>
+
+First, `--max-samples 16` splits the 16 loops into two halves of `8` loops. Recipe B's two halves still agree at cosine `0.828` (layer 15) and `0.864` (layer 19), with released-$u$ cosines `0.687` and `0.682`. In recipe A, subtracting window A cancels half the loop signal (`gap_size` is half as large), so sample noise dominates: its two halves agree at `0.632` and `0.622`, and its released-$u$ cosines drop to `0.574` and `0.596`. Second, `--max-samples 16` takes `used[:16]` in file order (all from 9 GSM8K problems) instead of 16 random loops across all 128 problems; in `data_needed`, random 16-loop sets had released-$u$ cosines of `0.666 / 0.683` (recipe A) and `0.757 / 0.746` (recipe B).
+</details>
+
+4. Why is the standard error on the paired difference between `capB-a6` and `p50c` (`-3.3 +/- 1.5` percentage points) nearly half as large as the standalone standard error on either run (`58.2 +/- 2.9%` and `61.5 +/- 2.8%`)?
+<details><summary>Answer</summary>
+
+The standalone standard error (`2.8%` or `2.9%`) includes the wide difficulty variation across the 256 problems in `dev256`. Because both edits are tested on the same 256 problems, pairing the scores problem by problem subtracts each problem's baseline difficulty and leaves only the variance of the difference between the two edits.
+</details>
+
+## Try this next
+
+1. **Restrict to loops that start late (`--min-before 2048`).** We did not run this experiment, so write down your prediction first. Out of the `225` verbatim loops in `build128`, `218` start at least `2,048` tokens into the answer (10th percentile `2,813` tokens). We expect `218` loops to be used and the directions $u$ and neuron overlaps to barely move, because Lab 6 Part 3 and the onset profile show the same gradual pre-loop drift regardless of onset position. Run the extraction (about 5 minutes on one GPU):
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --min-before 2048
+   ```
+   Compare `cos_with_released_u`, `overlap_with_released_64`, and `split_half_cos_mean_sd` in `results/lab8/try-min-before-2048/capstone.json` against the default `--min-before 512` run.
+2. **Move window A further back from the loop onset (`GAP = 32` to `GAP = 512`).** Near the top of `labs/lab8_capstone.py`, `WIN, AWIN, GAP = 36, 24, 32` sets how many tokens before loop onset window A ends (`GAP = 32`). Change `GAP` from `32` to `512` and pass `--try-name gap-512`:
+   ```bash
+   CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --try-name gap-512
+   ```
+   Check `results/lab8/try-gap-512/capstone.json` (and reset `GAP` to `32` afterward). Does moving window A 512 tokens before onset raise recipe A's `gap_size` (`0.689` at layer 15 when `GAP = 32`) and bring its cosine with the released $u$ closer to recipe B's `0.782`?
+3. **Predict what recipe A would do with 16 random loops instead of the first 16 (`used[:16]`).** We did not run this experiment. In `labs/lab8_capstone.py`, `used = used[:a.max_samples]` takes the first 16 loops in file order (9 GSM8K problems), giving recipe A a cosine of `0.574 / 0.596` with the released $u$ and `17.6%` accuracy on `dev256`. In `results/lab8/capstone.json`, the `answers = 16` entry of `data_needed` shows that across random 16-loop draws from all 225 loops, recipe A averages `0.666` (layer 15) and `0.683` (layer 19) with the released $u$, and `0.919` and `0.914` with the held-out half (`113` answers). Predict before editing: if you replace that line with `used = [used[i] for i in rng.choice(len(used), a.max_samples, replace=False)]` (`rng` is defined earlier) and run `--max-samples 16 --try-name rand16`, how much closer will `cos_with_released_u` and `overlap_with_released_64` move toward `capA-a6` (`0.722 / 0.724`), and will recipe A still trail recipe B at 16 random loops?
+4. **Read windows A and B of one loop yourself with `labs/peek.py --tokens`.** Under `"loops"` in `results/lab8/capstone.json`, each of the 225 loops records `id`, answer index `j`, `loop_starts_at`, `period`, and the token ranges of `A`, `B`, and `loop` from the start of the answer. In the median-onset answer `math_train_algebra_1625` (answer `1`), window B covers tokens `2,893` to `2,929`, window A covers `5,754` to `5,790`, and the 51-token loop starts at `5,822`. Print window B and the run-up to the loop on `sampler`:
+   ```bash
+   CUDA_VISIBLE_DEVICES="" python labs/peek.py build/base --id math_train_algebra_1625 --answer 1 --tokens 2893 2929
+   CUDA_VISIBLE_DEVICES="" python labs/peek.py build/base --id math_train_algebra_1625 --answer 1 --tokens 5200 5822
+   ```
+   Each command prints where the loop starts, one cycle, the last 1,500 characters, and the requested tokens. Window B is ordinary algebra (*"The equation is $b + n = 17.8$. So $b = 17.8 -$"*). Soon after token `5,200` the model checks $b = 8.8$ (the right answer), then circles through near-copies of one check (*"Wait, if $b = 8.99$, then $b \le 9$."* alternating with *"... then $b < 9$."*) for a few hundred tokens before the 51-token cycle locks in. Window A sits inside that circling. What does that show about what recipe A subtracts out when using window A as its baseline?

code/labs/lab8_capstone.py

@@ -0,0 +1,348 @@
+#!/usr/bin/env python
+"""Lab 8 (capstone): build your own edit from the model's own loops, then test it on problems it never saw.
+
+    . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+    CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py                          # about 5 minutes on one GPU
+    CUDA_VISIBLE_DEVICES="" python scripts/build_models.py --variants configs/capstone.json   # CPU, about a minute
+    python scripts/run_jobs.py plans/extra.json --gpus 6,7 --only 'dev/cap*'    # about 30 minutes, one run per GPU
+    python scripts/summarize.py /nvme/wsp-lab/runs                               # writes /nvme/wsp-lab/summary.md
+
+Experiments (each writes to its own folder, results/lab8/try-*, and leaves the main results, the figures and
+configs/capstone.json alone; the script prints the commands that build and test the try's two edits):
+    CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --max-samples 16    # u from only 16 loops
+    CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --min-before 2048   # only loops that start late
+    CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --alpha 8           # same u and neurons, stronger edit
+    CUDA_VISIBLE_DEVICES=7 python labs/lab8_capstone.py --split-by problem  # stability checks with no problem in both halves
+
+The paper's agent built its edit from 8 short texts that it wrote and labeled itself. Here the data is the
+unedited model's own answers to 128 practice problems (runs/build/base; none of them is used for testing).
+For every answer that ends in a verbatim loop, take three 36-token windows:
+  in the loop    starts two periods after the loop starts, when the model is surely repeating
+  just before    ends 32 tokens before the loop starts                          (recipe A)
+  well before    the middle of the text before the loop                         (recipe B)
+u = mean over answers of (stream in the loop - stream before), after blocks 15 and 19, scaled to length 1.
+Neuron score = (how much neuron j writes along u) x (how much more neuron j fires in the loop; last 24 tokens of
+each window). The 64 highest-scoring neurons get the edit with alpha 6: the agent's recipe, your data.
+
+Before you trust any direction you extract, measure:
+  agreement     cos with the released u, and how many of the released 64 neurons you picked too
+  stability     split the answers in half at random: do the two halves give the same u?
+  data needed   how many answers you need before u stops changing (the agent used 4 pairs)
+  transfer      does your u, built from the model's loops, separate the agent's 8 hand-written texts?
+and one question about the loops themselves: lined up at the token where each loop starts, when does the stream
+move along the released u, and when do the released 64 neurons start pushing (s(t), the edit's trigger)?
+"""
+import argparse
+import json
+import os
+import sys
+
+import numpy as np
+import torch
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+from wsp import hf, loops  # noqa: E402
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--run", default=os.path.join(hf.LAB, "runs", "build", "base"))
+ap.add_argument("--data", default="build128")
+ap.add_argument("--min-before", type=int, default=512, help="skip loops that start fewer answer tokens in")
+ap.add_argument("--max-samples", type=int, default=0, help="use at most this many loops (0 = all)")
+ap.add_argument("--alpha", type=float, default=6)
+ap.add_argument("--split-by", choices=("answer", "problem"), default="answer",
+                help="stability checks: split the loops at random by answer, or keep both answers to a problem "
+                     "in the same half (the edits themselves do not change)")
+ap.add_argument("--try-name", help="write to results/lab8/try-<name>/ (a name is picked for you when a flag changes)")
+a = ap.parse_args()
+
+# Experiments write to results/lab8/try-*/ and name their edits capA-<try>/capB-<try>, so the main results, the
+# figures, configs/capstone.json and the models/qwen35-cap*-a6 folders are never overwritten.
+changed = [f"{k.replace('_', '-')}-{getattr(a, k):g}" for k in ("min_before", "max_samples", "alpha")
+           if getattr(a, k) != ap.get_default(k)]
+if a.run != ap.get_default("run") or a.data != ap.get_default("data"):
+    changed.append(f"{os.path.basename(os.path.normpath(a.run))}-{a.data}")
+# --split-by on its own changes only the stability checks, so those edits are the main run's. Every other experiment,
+# including a --try-name run after you change the code (GAP, say), writes edits you can build.
+new_edits = bool(changed) or a.split_by == "answer"
+if a.split_by != "answer":
+    changed.append(f"split-by-{a.split_by}")
+TRY = a.try_name or "-".join(changed) or None
+OUT = os.path.join(ROOT, "results", "lab8", f"try-{TRY}") if TRY else os.path.join(ROOT, "results", "lab8")
+FIG = OUT if TRY else os.path.join(ROOT, "results", "figures")
+os.makedirs(OUT, exist_ok=True)
+os.makedirs(FIG, exist_ok=True)
+LAYERS = (15, 19)
+WIN, AWIN, GAP = 36, 24, 32          # window sizes as in the agent's recipe; GAP = distance of window A from onset
+PRE, POST = 2048, 256                # the onset profile covers this many tokens before and after each loop starts
+RECIPES = {"A": "just before the loop", "B": "well before the loop"}
+rng = np.random.default_rng(0)
+
+
+def unit(v):
+    return v / v.norm()
+
+
+rel = hf.released_edit()
+lab3 = json.load(open(os.path.join(ROOT, "results", "lab3", "rebuild.json")))
+# the released u, signed to point toward looping (Lab 3 found which way it points)
+u_rel = {L: unit(rel[L]["u"].float()) * float(np.sign(lab3["layers"][str(L)]["cos_with_released_u"]))
+         for L in LAYERS}
+
+# 1. find the loops
+rows = loops.read_rows(hf.LAB, a.data)
+m, tok = hf.load("base")
+n_total = n_cut = 0
+for rec, j, s, text in loops.iter_samples(a.run):
+    n_total += 1
+    n_cut += bool(s.get("cut"))
+found = loops.looping_samples(tok, a.run, rows)
+print(f"{n_total} answers in {a.run}: {n_cut} hit the token cap, {len(found)} of those end in a verbatim loop")
+
+
+def windows(f):
+    n0, on, p, T = f["n_prompt"], f["onset"], f["period"], len(f["ids"])
+    r0 = min(on + 2 * p, T - 1 - WIN)   # the last token is not part of the checked loop
+    if on - n0 < a.min_before or r0 < on:
+        return None
+    b0 = n0 + (on - n0) // 2 - WIN // 2
+    return {"loop": (r0, r0 + WIN), "A": (on - GAP - WIN, on - GAP), "B": (b0, b0 + WIN)}
+
+
+used = [(f, w) for f in found if (w := windows(f))]
+if a.max_samples:
+    used = used[:a.max_samples]
+N = len(used)
+print(f"using {N} loops (loop starts at least {a.min_before} answer tokens in)")
+assert N >= 8, "too few loops to build a direction"
+Wd = {L: hf.down_proj(m, L).weight.detach().float().cpu() for L in LAYERS}
+r_rel = {L: Wd[L][:, rel[L]["cols"]].T @ u_rel[L] for L in LAYERS}   # per released neuron: its push along u
+
+# 2. one forward pass per answer, cut where nothing later is needed (the model is causal)
+dh = {k: {L: [] for L in LAYERS} for k in RECIPES}
+da = {k: {L: [] for L in LAYERS} for k in RECIPES}
+prof = {q: {L: [] for L in LAYERS} for q in ("stream", "s")}
+meta, example = [], None
+order_by_onset = sorted(range(N), key=lambda i: used[i][0]["onset"] - used[i][0]["n_prompt"])
+example_i = order_by_onset[N // 2]
+for i, (f, w) in enumerate(used):
+    n0, on = f["n_prompt"], f["onset"]
+    end = max(w["loop"][1], min(on + POST, len(f["ids"])))
+    x = torch.tensor([f["ids"][:end]], device="cuda")
+    with torch.no_grad(), hf.record_outputs(m, LAYERS) as H, hf.record_mlp_acts(m, LAYERS) as A:
+        m.model(x, use_cache=False)
+    pos = np.arange(end) - on
+    keep = (pos >= -PRE) & (pos < POST) & (np.arange(end) >= n0)
+    ex = {} if i == example_i else None
+    for L in LAYERS:
+        h, act = H[L][0], A[L][0]
+        hw = {k: h[s:e].mean(0) for k, (s, e) in w.items()}
+        aw = {k: act[e - AWIN:e].mean(0) for k, (s, e) in w.items()}
+        for k in RECIPES:
+            dh[k][L].append(hw["loop"] - hw[k])
+            da[k][L].append(aw["loop"] - aw[k])
+        for q, v in (("stream", h @ u_rel[L]), ("s", act[:, rel[L]["cols"]] @ r_rel[L])):
+            row = np.full(PRE + POST, np.nan, dtype=np.float32)
+            row[pos[keep] + PRE] = v.numpy()[keep]
+            prof[q][L].append(row)
+        if ex is not None:
+            ex[f"stream{L}"] = (h[n0:] @ u_rel[L]).tolist()
+            ex[f"s{L}"] = (act[n0:, rel[L]["cols"]] @ r_rel[L]).tolist()
+    meta.append({"id": f["id"], "j": f["j"], "loop_starts_at": on - n0, "period": f["period"],
+                 "answer_tokens": len(f["ids"]) - n0, "windows": {k: [s - n0, e - n0] for k, (s, e) in w.items()}})
+    if ex is not None:
+        example = {"meta": meta[-1], **ex}
+    del H, A
+    if (i + 1) % 25 == 0:
+        print(f"  {i + 1}/{N} answers")
+
+# 3. directions, neurons, and the checks
+res = {"answers": n_total, "cut": n_cut, "verbatim_loops": len(found), "used": N, "min_before": a.min_before,
+       "periods": np.percentile([x["period"] for x in meta], [10, 25, 50, 75, 90]).tolist(),
+       "loop_starts_at": np.percentile([x["loop_starts_at"] for x in meta], [10, 25, 50, 75, 90]).tolist(),
+       "recipes": {}, "loops": meta}
+print(f"loop period, 10/25/50/75/90th percentile: {res['periods']} tokens")
+print(f"loop starts at answer token, same percentiles: {res['loop_starts_at']}")
+pairs = json.load(open(os.path.join(ROOT, "labs", "data", "astra_pairs.json")))
+with torch.no_grad(), hf.record_outputs(m, LAYERS) as H:
+    for t in pairs["texts"]:
+        m.model(hf.encode(tok, t), use_cache=False)
+astra_tail = {L: torch.stack([h[-WIN:].mean(0) for h in H[L]]) for L in LAYERS}   # [8 texts, 1024]
+SIZES = sorted({n for n in (1, 2, 4, 8, 16, 32, 64, 128) if n < N // 2} | {N // 2})
+problem_of = np.unique([f["id"] for f, _ in used], return_inverse=True)[1]   # which problem each loop came from
+n_problems = int(problem_of.max()) + 1
+half_sizes = []
+
+
+def split():
+    """Two halves of the loops for the stability checks."""
+    if a.split_by == "answer":   # two answers to the same problem can land in opposite halves
+        perm = rng.permutation(N)
+        h1, h2 = perm[:N // 2], perm[N // 2:]
+    else:                        # every problem's loops stay together, so the halves share no problem
+        first = rng.permutation(n_problems)[:n_problems // 2]
+        in1 = np.isin(problem_of, first)
+        h1, h2 = np.flatnonzero(in1), np.flatnonzero(~in1)
+    half_sizes.extend((len(h1), len(h2)))
+    return h1, h2
+
+
+capstone = {}
+for k, desc in RECIPES.items():
+    res["recipes"][k] = {"windows": desc, "layers": {}}
+    spec = {}
+    for L in LAYERS:
+        D, DA = torch.stack(dh[k][L]), torch.stack(da[k][L])
+        u = unit(D.mean(0))
+        score = (u @ Wd[L]) * DA.mean(0)
+        top_order = torch.argsort(score, descending=True)
+        top = sorted(top_order[:64].tolist())
+        halves, curve = [], {n: {"held_out_half": [], "released": [], "used": []} for n in SIZES}
+        for _ in range(20):
+            h1, h2 = split()
+            u2 = unit(D[h2].mean(0))
+            halves.append(float(unit(D[h1].mean(0)) @ u2))
+            for n in SIZES:   # a problem split can leave fewer than N // 2 loops in h1; then the last point uses all
+                used_n = min(n, len(h1))
+                un = unit(D[rng.choice(h1, used_n, replace=False)].mean(0))
+                curve[n]["held_out_half"].append(float(un @ u2))
+                curve[n]["released"].append(float(un @ u_rel[L]))
+                curve[n]["used"].append(used_n)
+        proj = astra_tail[L] @ u
+        gaps = (proj[1::2] - proj[0::2]).tolist()   # looping minus productive, one per pair
+        r = {"cos_with_released_u": float(u @ u_rel[L]),
+             "overlap_with_released_64": len(set(top) & set(rel[L]["cols"])),
+             "split_half_cos_mean_sd": [float(np.mean(halves)), float(np.std(halves))],
+             "data_needed": [{"answers": n, "cos_with_held_out_half": float(np.mean(c["held_out_half"])),
+                              "cos_with_released_u": float(np.mean(c["released"])),
+                              **({"answers_used_min_max": [min(c["used"]), max(c["used"])]} if min(c["used"]) < n
+                                 else {})} for n, c in curve.items()],
+             "astra_pairs_looping_minus_productive": gaps,
+             "gap_size": float(D.mean(0).norm()),
+             "top12": top_order[:12].tolist(), "cols": top,
+             "scores_top200": [round(float(x), 6) for x in score[top_order[:200]]]}
+        res["recipes"][k]["layers"][str(L)] = r
+        spec[str(L)] = {"alpha": a.alpha, "dir": [round(float(x), 8) for x in u], "cols": top}
+        print(f"recipe {k} ({desc}), layer {L}: cos with released u {r['cos_with_released_u']:+.3f}; "
+              f"released neurons picked {r['overlap_with_released_64']}/64; split-half cos "
+              f"{r['split_half_cos_mean_sd'][0]:.3f}; agent's 4 pairs, looping minus productive along u: "
+              f"{np.round(gaps, 2).tolist()}")
+    capstone[f"cap{k}-{TRY}" if TRY else f"cap{k}-a{a.alpha:g}"] = {
+        "about": f"Lab 8 recipe {k}: u and 64 neurons from the base model's own loops (windows {desc}), "
+                 f"alpha {a.alpha:g}",
+        "layers": spec}
+    print("   answers needed:", [(d["answers"], round(d["cos_with_held_out_half"], 3))
+                                 for d in res["recipes"][k]["layers"]["15"]["data_needed"]])
+res["cos_A_B"] = {}
+for L in LAYERS:
+    uA, uB = (torch.tensor(spec_k["layers"][str(L)]["dir"]) for spec_k in capstone.values())
+    res["cos_A_B"][str(L)] = float(unit(uA) @ unit(uB))
+print("cos between recipe A's u and recipe B's u:", {L: round(v, 3) for L, v in res["cos_A_B"].items()})
+res["split_by"] = a.split_by
+res["problems"] = n_problems
+res["problems_with_two_loops"] = int((np.bincount(problem_of) > 1).sum())
+res["half_sizes_min_max"] = [min(half_sizes), max(half_sizes)]
+print(f"{N} loops from {n_problems} problems ({res['problems_with_two_loops']} with both answers looping); "
+      f"split by {a.split_by}: each half held {min(half_sizes)} to {max(half_sizes)} loops")
+
+# 4. the onset profile: mean over loops, lined up at the loop's first token
+res["onset_profile"] = {}
+for q in prof:
+    for L in LAYERS:
+        P = np.stack(prof[q][L])
+        mean = np.nanmean(P, 0)
+
+        def avg(lo, hi):
+            return float(np.nanmean(mean[lo + PRE:hi + PRE]))
+        res["onset_profile"][f"{q}{L}"] = {"-2048..-1024": avg(-2048, -1024), "-512..-128": avg(-512, -128),
+                                          "-128..0": avg(-128, 0), "0..128": avg(0, 128), "128..256": avg(128, 256),
+                                          "mean": [round(float(x), 4) for x in mean]}
+        print(f"{'stream along u' if q == 'stream' else 's(t), the trigger'} at layer {L}, mean over loops: "
+              + ", ".join(f"{z} {v:.2f}" for z, v in res["onset_profile"][f"{q}{L}"].items() if z != "mean"))
+json.dump(res, open(os.path.join(OUT, "capstone.json"), "w"), indent=1)
+json.dump(example, open(os.path.join(OUT, "example_answer.json"), "w"))
+if TRY and not new_edits:
+    variants = plan = None
+elif TRY:
+    variants, plan = os.path.join(OUT, "variant.json"), os.path.join(OUT, "plan.json")
+    json.dump(capstone, open(variants, "w"), indent=1)
+    json.dump({"about": f"Lab 8 experiment {TRY}: both recipes' edits on the 256 unseen practice problems",
+               "runs": [{"name": f"dev/{v}", "model": f"qwen35-{v}", "data": "dev256", "n": 2, "shards": 1, "est": 8}
+                        for v in capstone]}, open(plan, "w"), indent=1)
+else:
+    variants, plan = os.path.join(ROOT, "configs", "capstone.json"), os.path.join(ROOT, "plans", "extra.json")
+    json.dump(capstone, open(variants, "w"), indent=1)
+
+# figures
+import matplotlib  # noqa: E402
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+
+
+def smooth(v, w=32):
+    v = np.asarray(v, dtype=float)
+    return np.convolve(v, np.ones(w) / w, mode="valid")
+
+
+fig, ax = plt.subplots(2, 2, figsize=(13, 6.5), sharex=True)
+xs = np.arange(-PRE, POST)
+for c, L in enumerate(LAYERS):
+    for r_, (q, lab) in enumerate((("stream", "stream along the released u"), ("s", "s(t): the 64 neurons' push"))):
+        P = np.stack(prof[q][L])
+        lo, mid, hi = np.nanpercentile(P, [25, 50, 75], axis=0)
+        ax[r_, c].fill_between(xs, lo, hi, color="#0969da", alpha=0.15, label="middle half of loops")
+        ax[r_, c].plot(xs, mid, color="#0969da", lw=1, label="median")
+        ax[r_, c].axvline(0, color="black", ls="--", lw=1)
+        ax[r_, c].set_title(f"layer {L}: {lab}")
+        ax[r_, c].set_xlabel("tokens from the start of the loop")
+ax[0, 0].legend(loc="upper left")
+fig.suptitle(f"{N} loops from the unedited model, lined up at the token where each loop starts")
+fig.tight_layout()
+fig.savefig(os.path.join(FIG, "lab8_loop_onset.png"), dpi=130)
+
+fig, ax = plt.subplots(1, 2, figsize=(13, 4))
+for c, L in enumerate(LAYERS):
+    for k, col in (("A", "#cf222e"), ("B", "#8250df")):
+        dn = res["recipes"][k]["layers"][str(L)]["data_needed"]
+        n = [d["answers"] for d in dn]
+        ax[c].plot(n, [d["cos_with_held_out_half"] for d in dn], "o-", color=col,
+                   label=f"recipe {k} vs the other half of the answers")
+        ax[c].plot(n, [d["cos_with_released_u"] for d in dn], "o--", color=col, alpha=0.6,
+                   label=f"recipe {k} vs the released u")
+    ax[c].axvline(4, color="#57606a", lw=0.8, ls=":")
+    ax[c].annotate("the agent used 4 pairs", (4, 0.05), xytext=(4, 0), textcoords="offset points", fontsize=8)
+    ax[c].set_xscale("log", base=2)
+    ax[c].set_ylim(0, 1)
+    ax[c].set_xlabel("answers used to build u")
+    ax[c].set_ylabel("cos")
+    ax[c].set_title(f"layer {L}: how many loops does a direction need?")
+ax[0].legend(fontsize=8, loc="lower right")
+fig.tight_layout()
+fig.savefig(os.path.join(FIG, "lab8_data_needed.png"), dpi=130)
+
+em = example["meta"]
+fig, ax = plt.subplots(2, 1, figsize=(13, 6), sharex=True)
+for r_, (q, lab) in enumerate((("stream", "stream along the released u"), ("s", "s(t): the 64 neurons' push"))):
+    for L, col in zip(LAYERS, ("#0969da", "#8250df")):
+        v = smooth(example[f"{q}{L}"])
+        ax[r_].plot(np.arange(len(v)) + 16, v, color=col, label=f"layer {L}")
+    for k, col in (("B", "#8250df"), ("A", "#cf222e"), ("loop", "#1a7f37")):
+        ax[r_].axvspan(*em["windows"][k], color=col, alpha=0.25)
+    ax[r_].axvline(em["loop_starts_at"], color="black", ls="--", lw=1)
+    ax[r_].set_ylabel(lab)
+ax[0].legend(loc="upper left")
+ax[1].set_xlabel("token position in the answer (32-token moving average)")
+ax[0].set_title(f"One answer ({em['id']}): the loop starts at token {em['loop_starts_at']} and repeats every "
+                f"{em['period']} tokens. Shaded: window B, window A, loop window")
+fig.tight_layout()
+fig.savefig(os.path.join(FIG, "lab8_one_answer.png"), dpi=130)
+rp = lambda p: os.path.relpath(p, ROOT)  # noqa: E731
+if variants is None:
+    print("wrote", rp(OUT), "and three figures in", rp(FIG))
+    print("--split-by changes only the stability checks, so the edits match the main run's (configs/capstone.json) and "
+          "there is nothing new to build or test (if you also changed the code, run again without --split-by)")
+    sys.exit(0)
+print("wrote", rp(OUT), rp(variants), "and three figures in", rp(FIG))
+print("next, build both edits and test them on the 256 unseen practice problems (one GPU per run):")
+print(f"  CUDA_VISIBLE_DEVICES=\"\" python scripts/build_models.py --variants {rp(variants)}")
+print(f"  python scripts/run_jobs.py {rp(plan)} --gpus 6,7 --only " + " ".join(f"dev/{v}" for v in capstone))

code/labs/peek.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python
+"""Read the model's answers yourself: how each one ended, and the text where it looped or stopped.
+
+    . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
+    export CUDA_VISIBLE_DEVICES=""                                  # CPU only, a few seconds
+    python labs/peek.py quick/base                                  # one line per answer
+    python labs/peek.py quick/base --id gsm8k_train_5628            # the end of answer 0 of that problem
+    python labs/peek.py dev/p50c --id gsm8k_train_5628 --answer 1 --chars 3000
+    python labs/peek.py build/base --id math_train_algebra_1625 --answer 1 --tokens 2893 2929   # one window
+
+Run it on the VM that made the run: the answer texts stay there (scripts/pull_runs.sh copies only the per-answer
+numbers to the laptop). For an answer that ends in a verbatim loop, it also finds the loop in tokens: where the
+repetition starts, how many tokens one cycle has, and the text of one cycle. --tokens prints any stretch of the
+answer by token position, counted from the start of the answer as in Lab 8's windows (results/lab8/capstone.json).
+"""
+import argparse
+import glob
+import json
+import os
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, ROOT)
+from wsp import hf, loops  # noqa: E402
+
+ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ap.add_argument("run", help="run name under runs/, e.g. quick/base or dev/p50c")
+ap.add_argument("--id", help="problem id to print (from the list); without it, list every answer")
+ap.add_argument("--answer", type=int, default=0, help="which of the problem's answers (0 = the first)")
+ap.add_argument("--chars", type=int, default=1500, help="how many characters to print from the end")
+ap.add_argument("--tokens", type=int, nargs=2, metavar=("START", "END"),
+                help="also print answer tokens START to END (e.g. a window from results/lab8/capstone.json)")
+a = ap.parse_args()
+
+run_dir = os.path.join(hf.LAB, "runs", a.run)
+samples = list(loops.iter_samples(run_dir))
+if not samples:
+    sys.exit(f"no answer texts in {run_dir}; is this the VM that ran {a.run}?")
+
+
+def ending(s):
+    if s["cut"]:
+        return "cut at the cap " + ("after thinking" if s["closed"] else "while thinking")
+    return "finished" if s["closed"] else "stopped while thinking"
+
+
+if not a.id:
+    print(f"{'problem':36s} {'answer':>6s} {'tokens':>7s}  {'how it ended':29s} {'loops at the end':16s} "
+          f"{'correct':7s}  model said / truth")
+    for rec, j, s, text in samples:
+        loop = "yes" if s["verbatim_loop_end"] else ""
+        print(f"{rec['id']:36s} {j:6d} {s['tokens']:7,d}  {ending(s):29s} {loop:16s} {'yes' if s['correct'] else '':7s}"
+              f"  {str(s['pred'])[:20]} / {str(rec.get('answer', '-'))[:20]}")
+    n, cut = len(samples), sum(s["cut"] for _, _, s, _ in samples)
+    print(f"\n{n} answers: {sum(s['correct'] for _, _, s, _ in samples)} correct, {cut} cut at the cap, "
+          f"{sum(s['cut'] and s['verbatim_loop_end'] for _, _, s, _ in samples)} of the cut ones end in a verbatim loop")
+    print(f"read one: python labs/peek.py {a.run} --id <problem> [--answer N]")
+    sys.exit()
+
+hit = [(rec, j, s, text) for rec, j, s, text in samples if rec["id"] == a.id and j == a.answer]
+if not hit:
+    sys.exit(f"no answer {a.answer} for problem {a.id} in {a.run}; list them with: python labs/peek.py {a.run}")
+rec, j, s, text = hit[0]
+done = json.load(open(sorted(glob.glob(os.path.join(run_dir, "shard*.done.json")))[0]))
+problem = loops.read_rows(hf.LAB, os.path.basename(done["data"]).removesuffix(".jsonl")).get(rec["id"], {})
+print(f"problem {rec['id']}: {problem.get('problem', problem.get('prompt', '?'))}")
+print(f"truth: {rec.get('answer', '-')}   model said: {s['pred']}   correct: {'yes' if s['correct'] else 'no'}")
+print(f"answer {j}: {s['tokens']:,} tokens, {ending(s)}")
+
+if s["verbatim_loop_end"] or a.tokens:
+    from transformers import AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(hf.model_dir("base"))
+    ids = tok(text, add_special_tokens=False).input_ids
+found = loops.find_loop(ids) if s["verbatim_loop_end"] else None
+if found:
+    onset, period = found
+    print(f"it loops: from token {onset:,} of {len(ids):,}, one block of {period} tokens repeats about "
+          f"{(len(ids) - 1 - onset) / period:.0f} times. One cycle:")
+    print("    " + repr(tok.decode(ids[onset:onset + period])))
+i = text.find("</think>")
+if i >= 0:
+    print(f"\n--- where it closed its thinking (character {i:,} of {len(text):,}) ---")
+    print(text[max(0, i - 400):i + 200])
+print(f"\n--- the last {a.chars:,} characters ---")
+print(text[-a.chars:])
+if a.tokens:
+    t0, t1 = max(0, a.tokens[0]), min(len(ids), a.tokens[1])
+    print(f"\n--- answer tokens {t0:,} to {t1:,} (of {len(ids):,}) ---")
+    print(tok.decode(ids[t0:t1]))

code/plans/extra.json

@@ -0,0 +1,8 @@
+{
+ "about": "Labs 3 and 8 on unseen problems: the edit rebuilt from the agent's 8 texts, and the capstone edits built from the unedited model's own loops. Same 256 practice problems and settings as plans/sampler.json.",
+ "runs": [
+  {"name": "dev/rebuilt", "model": "qwen35-rebuilt", "data": "dev256", "n": 2, "shards": 1, "est": 8},
+  {"name": "dev/capA-a6", "model": "qwen35-capA-a6", "data": "dev256", "n": 2, "shards": 1, "est": 8},
+  {"name": "dev/capB-a6", "model": "qwen35-capB-a6", "data": "dev256", "n": 2, "shards": 1, "est": 8}
+ ]
+}

code/plans/lab3-tries.json

@@ -0,0 +1,29 @@
+{
+ "about": "Lab 3's three experiments on unseen problems: the edit rebuilt from only the agent's first 2 pairs, with a 36-token neuron window, and with a third layer (21). Same 256 practice problems and settings as plans/sampler.json. Build the three models first: run labs/lab3_rebuild.py with --pairs 2, --windows 36 36 and --layers 15 19 21, then build each results/lab3/try-*/variant.json (labs/lab3_rebuild.md, Try this next).",
+ "runs": [
+  {
+   "name": "dev/rebuilt-pairs2",
+   "model": "qwen35-rebuilt-pairs2",
+   "data": "dev256",
+   "n": 2,
+   "shards": 2,
+   "est": 8
+  },
+  {
+   "name": "dev/rebuilt-windows36-36",
+   "model": "qwen35-rebuilt-windows36-36",
+   "data": "dev256",
+   "n": 2,
+   "shards": 2,
+   "est": 8
+  },
+  {
+   "name": "dev/rebuilt-layers15-19-21",
+   "model": "qwen35-rebuilt-layers15-19-21",
+   "data": "dev256",
+   "n": 2,
+   "shards": 2,
+   "est": 8
+  }
+ ]
+}

code/plans/quick.json

@@ -0,0 +1,23 @@
+{
+ "about": "Quick start: the unedited model and the released edit on the first 16 practice problems of dev256 (all from GSM8K), 2 answers each, the paper's settings. About 5 minutes on 2 GPUs. Then read the answers with labs/peek.py.",
+ "runs": [
+  {
+   "name": "quick/base",
+   "model": "base",
+   "data": "dev256",
+   "n": 2,
+   "shards": 1,
+   "est": 1,
+   "args": ["--limit", "16"]
+  },
+  {
+   "name": "quick/p50c",
+   "model": "qwen35-p50c",
+   "data": "dev256",
+   "n": 2,
+   "shards": 1,
+   "est": 1,
+   "args": ["--limit", "16"]
+  }
+ ]
+}

code/plans/sampler.json

@@ -0,0 +1,29 @@
+{
+ "about": "Labs 4, 5 and 8: every edit variant and every decoding fix on 256 unseen practice problems (2 answers each), plus the unedited model's answers on 128 more practice problems for the capstone.",
+ "runs": [
+  {"name": "build/base", "model": "base", "data": "build128", "n": 2, "shards": 1, "est": 8, "priority": 2},
+  {"name": "dev/base", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "priority": 1},
+  {"name": "dev/p50c", "model": "qwen35-p50c", "data": "dev256", "n": 2, "shards": 1, "est": 8, "priority": 1},
+
+  {"name": "dev/a-4", "model": "qwen35-a-4", "data": "dev256", "n": 2, "shards": 1, "est": 17},
+  {"name": "dev/a1", "model": "qwen35-a1", "data": "dev256", "n": 2, "shards": 1, "est": 13},
+  {"name": "dev/a2", "model": "qwen35-a2", "data": "dev256", "n": 2, "shards": 1, "est": 10},
+  {"name": "dev/a4", "model": "qwen35-a4", "data": "dev256", "n": 2, "shards": 1, "est": 8},
+  {"name": "dev/a8", "model": "qwen35-a8", "data": "dev256", "n": 2, "shards": 1, "est": 8},
+  {"name": "dev/a12", "model": "qwen35-a12", "data": "dev256", "n": 2, "shards": 1, "est": 8},
+  {"name": "dev/l15only", "model": "qwen35-l15only", "data": "dev256", "n": 2, "shards": 1, "est": 11},
+  {"name": "dev/l19only", "model": "qwen35-l19only", "data": "dev256", "n": 2, "shards": 1, "est": 11},
+  {"name": "dev/randdir", "model": "qwen35-randdir", "data": "dev256", "n": 2, "shards": 1, "est": 15},
+  {"name": "dev/randcols", "model": "qwen35-randcols", "data": "dev256", "n": 2, "shards": 1, "est": 15},
+  {"name": "dev/all1", "model": "qwen35-all1", "data": "dev256", "n": 2, "shards": 1, "est": 12},
+  {"name": "dev/all2", "model": "qwen35-all2", "data": "dev256", "n": 2, "shards": 1, "est": 10},
+
+  {"name": "dev/base+temp1.0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["temperature=1.0"]},
+  {"name": "dev/base+pp0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=0"]},
+  {"name": "dev/base+pp2", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=2.0"]},
+  {"name": "dev/base+rep1.1", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 14, "set": ["repetition_penalty=1.1"]},
+  {"name": "dev/base+max4k", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "set": ["max_tokens=4096"]},
+  {"name": "dev/base+nothink", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"},
+  {"name": "dev/p50c+nothink", "model": "qwen35-p50c", "data": "dev256", "n": 2, "shards": 1, "est": 1, "preset": "nothink35"}
+ ]
+}

code/plans/trainer.json

@@ -0,0 +1,11 @@
+{
+ "about": "Lab 2: the paper's three benchmarks, unedited model vs the released edit, same settings as the paper (thinking on, 32,768-token cap).",
+ "runs": [
+  {"name": "bench/base/math500", "model": "base", "data": "math500", "n": 4, "shards": 4, "est": 62},
+  {"name": "bench/base/gsm8k", "model": "base", "data": "gsm8k", "n": 2, "shards": 5, "est": 77},
+  {"name": "bench/base/ifeval", "model": "base", "data": "ifeval", "n": 2, "shards": 2, "est": 27},
+  {"name": "bench/p50c/math500", "model": "qwen35-p50c", "data": "math500", "n": 4, "shards": 3, "est": 44},
+  {"name": "bench/p50c/gsm8k", "model": "qwen35-p50c", "data": "gsm8k", "n": 2, "shards": 2, "est": 32},
+  {"name": "bench/p50c/ifeval", "model": "qwen35-p50c", "data": "ifeval", "n": 2, "shards": 1, "est": 5}
+ ]
+}

code/reference/theory-guide.md

@@ -0,0 +1,1791 @@
+# Weight-space programming: from the basics to the actual research
+
+The central idea is that a neural network is a numerical program, and a capable LLM may be able to inspect, build, or repair parts of that program by editing its weights directly.
+
+Each lab in this project tests a section of this guide.
+
+| Guide section | Lab that makes it hands-on |
+|---|---|
+| 1 to 5: weights, architecture, training vs editing, the language-model machinery | Background. Read before [Lab 0](../labs/lab0_toy.md). |
+| 6: each MLP column writes a vector | [Lab 0](../labs/lab0_toy.md) |
+| 7: what went wrong with Qwen | [Lab 2](../labs/lab2_reproduce.md) |
+| 8.1 to 8.4: directions, dot products, removing a component | [Lab 0](../labs/lab0_toy.md) |
+| 8.5: from removal to counteraction (the strength alpha) | [Lab 0](../labs/lab0_toy.md), then [Lab 4](../labs/lab4_dose.md) |
+| 9, steps 1 to 3: contexts, the direction, the neurons, the columns | [Lab 3](../labs/lab3_rebuild.md), then [Lab 8](../labs/lab8_capstone.md) |
+| 9, step 4: the matrix operation | [Lab 1](../labs/lab1_diff.md) |
+| 10: why believe this repaired repetition | [Lab 2](../labs/lab2_reproduce.md) and the controls in [Lab 4](../labs/lab4_dose.md) |
+| 12C: low rank versus LoRA | [Lab 1](../labs/lab1_diff.md) (the change is rank one) |
+| 12D: weight editing versus activation steering | [Lab 6](../labs/lab6_steering.md) |
+| 13.2: compare against a strong practical baseline | [Lab 5](../labs/lab5_decoding.md) |
+| 13.3: keep development data separate | [Lab 8](../labs/lab8_capstone.md) |
+| 13.4: measure more than accuracy | [Lab 7](../labs/lab7_side_effects.md) |
+| 11, 12A, 12B, 12E, 12F, 13.1, 13.5, 13.6, 14 | No lab. Read them in the guide. |
+
+We usually find weights by training. This study asks whether an LLM can instead read what weights compute, repair a broken network, or write a network from scratch. [rikiyat.github.io](https://rikiyat.github.io/blog/weight-space-programming/)
+
+In the main language-model experiment, a small Qwen model often got stuck in repetitive reasoning, and a direct weight edit helped it finish many more answers. Understanding that repair requires five pieces:
+
+1. How weights turn numbers into a computation.
+2. How direct editing differs from training.
+3. What attention heads and MLPs do inside a language model.
+4. How vectors, projections, and small matrix updates change behavior.
+5. What these experiments show and what stays open.
+
+Every exercise has its worked solution right below "Check after trying." Cover the solution and work each one on paper first.
+
+---
+
+## 1. Start here: what is a neural-network weight?
+
+A paddle moves left or right to stay under a ball. Its input has two coordinates:
+
+$$
+x=
+\begin{pmatrix}
+p\\
+b
+\end{pmatrix},
+$$
+
+where:
+
+- $p$ is the paddle's horizontal position.
+- $b$ is the ball's horizontal position.
+
+With the paddle at $3$ and the ball at $8$:
+
+$$
+x=
+\begin{pmatrix}
+3\\
+8
+\end{pmatrix}.
+$$
+
+The ball is to the right, so we subtract:
+
+$$
+b-p.
+$$
+
+$$
+8-3=5.
+$$
+
+A positive difference means the ball is to the right; a negative difference means it is to the left. Written with weights:
+
+$$
+(-1)p+(1)b.
+$$
+
+The weights are $-1$ and $1$, which set how much each input contributes. A weight is a numerical coefficient in a computation; when we know the rule we want, we can choose the weights ourselves instead of training them.
+
+### Build a tiny network yourself
+
+We want two hidden quantities that measure how far the ball is to the right or left:
+
+$$
+h_{\text{right}}=\max(0,b-p),
+$$
+
+$$
+h_{\text{left}}=\max(0,p-b).
+$$
+
+The operation
+
+$$
+\operatorname{ReLU}(t)=\max(0,t)
+$$
+
+is pronounced "ree-loo." It keeps positive values and replaces negative values with zero.
+
+**Try first:** What $2\times2$ matrix computes $b-p$ in its first row and $p-b$ in its second row?
+
+*Hint: each row specifies one weighted sum of the input coordinates.*
+
+### Check after trying
+
+$$
+W_1=
+\begin{pmatrix}
+-1&1\\
+1&-1
+\end{pmatrix}.
+$$
+
+Now multiply:
+
+$$
+W_1x
+=
+\begin{pmatrix}
+-1&1\\
+1&-1
+\end{pmatrix}
+\begin{pmatrix}
+3\\
+8
+\end{pmatrix}
+=
+\begin{pmatrix}
+(-1)(3)+(1)(8)\\
+(1)(3)+(-1)(8)
+\end{pmatrix}
+=
+\begin{pmatrix}
+5\\
+-5
+\end{pmatrix}.
+$$
+
+Apply ReLU to each entry:
+
+$$
+h=
+\operatorname{ReLU}(W_1x)
+=
+\begin{pmatrix}
+5\\
+0
+\end{pmatrix}.
+$$
+
+Interpretation:
+
+- Rightward distance: $5$.
+- Leftward distance: $0$.
+
+### Turn those hidden quantities into actions
+
+Our three output scores will be ordered:
+
+$$
+\text{left},\quad \text{stay},\quad \text{right}.
+$$
+
+Define:
+
+$$
+z=
+W_2h+c,
+$$
+
+with
+
+$$
+W_2=
+\begin{pmatrix}
+0&1\\
+0&0\\
+1&0
+\end{pmatrix},
+\qquad
+c=
+\begin{pmatrix}
+0\\
+1\\
+0
+\end{pmatrix}.
+$$
+
+The vector $c$ is a bias, an offset added after multiplication.
+
+Calculate:
+
+$$
+z=
+\begin{pmatrix}
+0&1\\
+0&0\\
+1&0
+\end{pmatrix}
+\begin{pmatrix}
+5\\
+0
+\end{pmatrix}
++
+\begin{pmatrix}
+0\\
+1\\
+0
+\end{pmatrix}
+=
+\begin{pmatrix}
+0\\
+1\\
+5
+\end{pmatrix}.
+$$
+
+The highest score is "right." Because we knew the rule in advance, we could program the network by choosing its weights without a dataset.
+
+### What did the bias accomplish?
+
+If the ball and paddle have the same position:
+
+$$
+h=
+\begin{pmatrix}
+0\\
+0
+\end{pmatrix},
+\qquad
+z=
+\begin{pmatrix}
+0\\
+1\\
+0
+\end{pmatrix}.
+$$
+
+The paddle stays still. "Stay" wins whenever the position difference is smaller than $1$; a tie needs a tie-breaking rule.
+
+### Exercise: run it in the other direction
+
+Now let:
+
+$$
+x=
+\begin{pmatrix}
+8\\
+3
+\end{pmatrix}.
+$$
+
+Compute the hidden vector and action scores.
+
+### Check after trying
+
+$$
+W_1x=
+\begin{pmatrix}
+-5\\
+5
+\end{pmatrix},
+\qquad
+h=
+\begin{pmatrix}
+0\\
+5
+\end{pmatrix}.
+$$
+
+Therefore:
+
+$$
+z=
+\begin{pmatrix}
+5\\
+1\\
+0
+\end{pmatrix}.
+$$
+
+The paddle moves left.
+
+**What you should be able to explain:** the first layer computes relationships between inputs, and the second layer turns those relationships into action scores.
+
+---
+
+## 2. Architecture, weights, activations, and weight space
+
+Keep these concepts separate:
+
+| Term | Meaning | In our paddle network |
+|---|---|---|
+| **Architecture** | Which operations happen and how they connect | Matrix multiplication, ReLU, another matrix multiplication |
+| **Weights** | Stored coefficients specifying the computation | Entries of $W_1$ and $W_2$ |
+| **Biases** | Stored offsets | The $1$ that favors staying still |
+| **Activations** | Intermediate values produced for a particular input | $h=(5,0)^T$ |
+| **Logits** | Raw output scores before conversion to probabilities | $z=(0,1,5)^T$ |
+| **Parameters** | Collective name for adjustable stored values | All weights and biases |
+
+During inference, inputs and activations change from example to example while the stored parameters stay fixed. Collecting every parameter into one vector gives
+
+$$
+\theta,
+$$
+
+pronounced "theta." The network is the function:
+
+$$
+f_\theta(x).
+$$
+
+Read this as "the function applied to input $x$ with parameters $\theta$."
+
+### What does "weight space" mean?
+
+If a model has two weights:
+
+$$
+\theta=(w_1,w_2).
+$$
+
+Every setting is a point on a two-dimensional plane. A model with a million parameters lives in a million-dimensional space, with one coordinate per stored parameter. Editing weights moves the model to another point in that space.
+
+Weight space differs from activation space. An activation vector holds the intermediate numbers produced on one input; a weight vector holds part of the stored program.
+
+### A useful limitation of the "program" analogy
+
+In ordinary code, a line might say:
+
+```python
+if ball_is_right:
+    move_right()
+```
+
+Trained networks rarely separate into clean lines like this. One concept can spread across many neurons and matrices, and one neuron can take part in several behaviors. Our paddle network is easy to read because we designed each coordinate by hand; networks shaped by training are much harder to read.
+
+---
+
+## 3. How ordinary training differs from direct weight editing
+
+Start with a one-weight model:
+
+$$
+\hat y=wx.
+$$
+
+Here:
+
+- $x$: input.
+- $w$: weight.
+- $\hat y$, pronounced "y-hat": predicted output.
+
+Suppose:
+
+$$
+x=2,\qquad y=6,\qquad w=1.
+$$
+
+The target output is $6$, and the model predicts:
+
+$$
+\hat y=1\cdot2=2.
+$$
+
+### Forward pass
+
+A forward pass runs the input through the current weights to produce an output, here $2$.
+
+### Loss
+
+A loss function scores how far the prediction is from the target:
+
+$$
+L(w)=\frac12(wx-y)^2.
+$$
+
+At $w=1$:
+
+$$
+L(1)=\frac12(2-6)^2=8.
+$$
+
+### Backward pass
+
+A backward pass computes how the loss changes when each weight changes (the derivative of the loss with respect to that weight):
+
+$$
+\frac{dL}{dw}=(wx-y)x.
+$$
+
+Substituting:
+
+$$
+\frac{dL}{dw}=(2-6)\cdot2=-8.
+$$
+
+Because the derivative is negative, nudging $w$ upward lowers the loss.
+
+### Gradient-descent update
+
+With learning rate $\eta=0.1$:
+
+$$
+w_{\text{new}}
+=
+w-\eta\frac{dL}{dw}
+=
+1-0.1(-8)
+=
+1.8.
+$$
+
+The new prediction is closer to $6$:
+
+$$
+1.8\cdot2=3.6.
+$$
+
+### Direct programming
+
+When we know the exact relationship, we can solve directly:
+
+$$
+2w=6,
+$$
+
+so:
+
+$$
+w=3.
+$$
+
+**Try first:** Why does this example not prove that gradient descent is unnecessary for large networks?
+
+### Check after trying
+
+Here the specification gave the equation directly. In a large nonlinear network:
+
+- We may not know the exact internal formula the task needs.
+- Many weights interact across layers.
+- Many parameter settings produce the same output on a given input.
+- Fixing one behavior can break another.
+
+Direct editing works when we can isolate the mechanism we want to change; training works when the mechanism has to be discovered from examples.
+
+### Where weight-space programming fits
+
+| Approach | How the target model is changed |
+|---|---|
+| Prompting | Change the input; keep its weights fixed |
+| Ordinary fine-tuning | Compute gradients and update weights |
+| An LLM writing a training script | The LLM designs the procedure; a training algorithm updates weights |
+| Random or evolutionary search | Try parameter changes and retain promising ones |
+| Weight-space programming in this work | The LLM reasons about the target and directly constructs parameter changes |
+
+Keep two distinctions in mind:
+
+$$
+\boxed{
+\text{No gradients does not imply no data, no feedback, or no computation.}
+}
+$$
+
+Some experiments in this study are one-shot constructions, while others test and revise edits over many trials. Every experiment also involves two separate models:
+
+- The editor: a frontier pretrained LLM.
+- The target: the network being inspected or changed.
+
+"No training" means no gradient updates on the target during the edit. Both the editor and any pretrained target were already trained beforehand.
+
+---
+
+## 4. How could anyone infer behavior from weights?
+
+Return to a single weighted sum:
+
+$$
+z=-p+b.
+$$
+
+When we know what $p$ and $b$ mean, we can see that $z$ measures relative position. From the weight row alone,
+
+$$
+(-1,1),
+$$
+
+we cannot conclude "paddle controller." It could compute:
+
+- Revenue minus cost.
+- Target position minus robot position.
+- Tomorrow's temperature minus today's temperature.
+
+Weights get their meaning from the architecture and from the input and output coordinates they connect. In the game-identification experiments, the authors gave the editor the architecture and the input and output descriptions while withholding the game names, so the task was to infer the game from how the learned weights used those known inputs. [rikiyat.github.io](https://rikiyat.github.io/blog/weight-space-programming/)
+
+### Why boundary layers can be revealing
+
+Suppose the input is an image grid. First-layer weights that concentrate on the bottom row of the "agent" channel suggest an agent constrained near the bottom of the screen. Output weights that mainly separate left from right add another clue. Together, those clues can suggest a paddle game. That is meaningful inference, but it is a different achievement from explaining every internal computation.
+
+### Exercise: the same numbers, a changed interface
+
+A robot controller uses:
+
+$$
+z=x_1-x_2,
+$$
+
+where $x_1$ is the target position and $x_2$ is the robot position.
+
+A new sensor interface reverses the order:
+
+$$
+x'=
+\begin{pmatrix}
+\text{robot position}\\
+\text{target position}
+\end{pmatrix}.
+$$
+
+What should the new weight row be?
+
+### Check after trying
+
+The original weight row was:
+
+$$
+(1,-1).
+$$
+
+The reversed interface needs:
+
+$$
+(-1,1).
+$$
+
+The behavior is unchanged, while the weights flip to match the coordinate order. This is why reading weights requires knowing what the coordinates mean.
+
+### Another complication: different weights can implement the same function
+
+Suppose:
+
+$$
+h_1=\operatorname{ReLU}(x_1),\qquad
+h_2=\operatorname{ReLU}(x_2),
+$$
+
+and:
+
+$$
+y=2h_1+3h_2.
+$$
+
+Now swap the hidden neurons:
+
+$$
+h'_1=\operatorname{ReLU}(x_2),\qquad
+h'_2=\operatorname{ReLU}(x_1).
+$$
+
+Use:
+
+$$
+y=3h'_1+2h'_2.
+$$
+
+Both versions compute the same function. For $x=(4,5)$, both produce:
+
+$$
+2(4)+3(5)=23.
+$$
+
+Because hidden neurons can be swapped without changing the output, "neuron 1" has no fixed meaning across networks; what matters is how a neuron connects to the layers around it.
+
+---
+
+## 5. The language-model machinery you need
+
+### Tokens become vectors
+
+A tokenizer splits text into tokens (words, subwords, or punctuation), and an embedding maps each token to a vector. At each token position, the model updates an internal vector:
+
+$$
+h.
+$$
+
+### The residual stream
+
+As a useful approximation, each component adds its update to the running vector, and the existing vector continues forward:
+
+$$
+h_{\text{next}}=h+\text{update}.
+$$
+
+This running vector is the residual stream. Attention heads and MLPs read from it and add their outputs back into it, like a shared workspace made of numerical features rather than readable text.
+
+### Attention moves information between positions
+
+For a simplified attention head:
+
+$$
+q_i=W_Qh_i,
+\qquad
+k_j=W_Kh_j,
+\qquad
+v_j=W_Vh_j.
+$$
+
+Here:
+
+- $i$ is the token position receiving information.
+- $j$ is a candidate position to read from.
+- $q_i$, the query, specifies what position $i$ looks for.
+- $k_j$, the key, specifies what position $j$ offers.
+- $v_j$, the value, holds the information that can be copied.
+
+The matching score includes:
+
+$$
+q_i^Tk_j.
+$$
+
+The superscript $T$ means transpose, which turns a column vector into a row so the product is one number. Expanding the query and key shows why multiplying weight matrices together is informative:
+
+$$
+q_i^Tk_j
+=
+h_i^TW_Q^TW_Kh_j.
+$$
+
+The product
+
+$$
+W_Q^TW_K
+$$
+
+describes which token features match which, and the output path uses the product
+
+$$
+W_OW_V.
+$$
+
+In the transformer-circuits framework (shown here without positional encodings or normalization): [transformer-circuits.pub](https://transformer-circuits.pub/2021/framework/index.html)
+
+- **QK:** selects where to read information from.
+- **OV:** determines what is written back into the residual stream.
+
+### What is an induction head?
+
+When a sequence contains
+
+> A B ... A
+
+an induction circuit predicts
+
+> B
+
+by matching the current token to its earlier occurrence and copying what followed it.
+
+**Try first:** In the pattern "red apple ... red," should a copying circuit copy the previous word "red"?
+
+### Check after trying
+
+No, the continuation is "apple." The circuit must link the earlier "red" to the token after it, which is why the researchers inspected products of matrices across heads rather than looking only for large single weights.
+
+### MLPs compute and write features
+
+An MLP (multilayer perceptron) transforms the representation at one position:
+
+$$
+a=\phi(W_{\text{up}}h),
+$$
+
+$$
+m=W_{\text{down}}a,
+$$
+
+$$
+h_{\text{next}}=h+m.
+$$
+
+Here $a$ holds the intermediate feature activations, and $W_{\text{down}}$ turns those activations into an update for the residual stream (real models may also use gating and normalization). For the Qwen target, the dimensions are:
+
+$$
+h\in\mathbb R^{1024},
+\qquad
+a\in\mathbb R^{3584},
+\qquad
+W_{\text{down}}\in\mathbb R^{1024\times3584}.
+$$
+
+The notation $\mathbb R^{1024}$ means "a vector of 1,024 real numbers." This Qwen model mixes Gated DeltaNet and gated-attention blocks, so standard attention does not describe every layer. [Hugging Face](https://huggingface.co/Qwen/Qwen3.5-0.8B)
+
+---
+
+## 6. The key insight: each MLP column writes a vector
+
+Let the columns of $W_{\text{down}}$ be:
+
+$$
+w_1,w_2,\ldots,w_{3584}.
+$$
+
+Then matrix multiplication can be rewritten:
+
+$$
+\boxed{
+W_{\text{down}}a
+=
+a_1w_1+a_2w_2+\cdots+a_{3584}w_{3584}.
+}
+$$
+
+This is the most useful equation for understanding the edit. Each intermediate activation $a_j$ controls how much of column $w_j$ gets added:
+
+- If $a_j=0$, that feature contributes nothing.
+- If $a_j=2$, it contributes $2w_j$.
+- Editing $w_j$ changes the vector that feature writes whenever it activates.
+
+### Calculate it
+
+Suppose:
+
+$$
+W=
+\begin{pmatrix}
+1&2\\
+3&4
+\end{pmatrix},
+\qquad
+a=
+\begin{pmatrix}
+2\\
+0
+\end{pmatrix}.
+$$
+
+**Try first:** Compute $Wa$ by thinking in columns.
+
+### Check after trying
+
+The columns are:
+
+$$
+w_1=
+\begin{pmatrix}
+1\\
+3
+\end{pmatrix},
+\qquad
+w_2=
+\begin{pmatrix}
+2\\
+4
+\end{pmatrix}.
+$$
+
+Therefore:
+
+$$
+Wa
+=
+2w_1+0w_2
+=
+\begin{pmatrix}
+2\\
+6
+\end{pmatrix}.
+$$
+
+Even if the second column held huge numbers, it would contribute nothing on this input because its activation is zero. Looking only for the largest weights is not enough to find important behavior: a neuron's effect depends on both its stored vector and when it activates.
+
+---
+
+## 7. What went wrong with Qwen?
+
+Language models generate text autoregressively:
+
+1. Read the current context.
+2. Predict the next token.
+3. Append it.
+4. Repeat.
+
+During chain-of-thought reasoning, each forward pass is finite, yet the generated tokens can steer the context into repeating the same material until generation hits the token cap without a final answer.
+
+### Why fixing repetition can improve "math accuracy"
+
+Consider an invented illustration, not a decomposition measured in this study:
+
+- A model finishes 10% of problems.
+- Among the answers it finishes, 80% are correct.
+
+The fraction of problems with a correct finished answer is:
+
+$$
+0.10\cdot0.80=0.08.
+$$
+
+If a repair raises completion to 60% while accuracy on finished answers stays 80%:
+
+$$
+0.60\cdot0.80=0.48.
+$$
+
+Overall accuracy rises from 8% to 48% without adding mathematical knowledge, because:
+
+$$
+P(\text{correct and finished})
+=
+P(\text{finished})
+P(\text{correct}\mid\text{finished}).
+$$
+
+### The reported results
+
+Under the authors' evaluation settings:
+
+| Measurement | Original Qwen | Edited Qwen |
+|---|---:|---:|
+| MATH-500 accuracy | 7.9% | **48.1%** |
+| GSM8K accuracy | 13.8% | **67.1%** |
+| IFEval accuracy | 21.0% | **62.3%** |
+| MATH-500 responses reaching the token cap | 93.0% | **33.1%** |
+
+MATH-500 and GSM8K test mathematical problem solving; IFEval tests instruction compliance. The gain on IFEval fits a repair that reduces a shared generation failure across tasks. [Hugging Face](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming)
+
+Reported scores average correctness across sampled responses (four responses per MATH-500 problem), rather than counting a problem solved if any of the four attempts succeeds. Evaluation used thinking mode, a 32,768-token limit, temperature $0.6$, top-p $0.95$, top-k $20$, and presence penalty $1.5$. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md)
+
+### Exercise: distinguish two evaluation methods
+
+A model answers one problem four times:
+
+$$
+\text{correct},\ \text{wrong},\ \text{wrong},\ \text{correct}.
+$$
+
+What is its average sample accuracy on that problem? What is its "at least one success among four" result?
+
+### Check after trying
+
+Average sample accuracy:
+
+$$
+\frac24=50\%.
+$$
+
+At least one success:
+
+$$
+100\%.
+$$
+
+Confusing these two metrics can make a model look much stronger than the experiment shows.
+
+---
+
+## 8. The linear algebra behind the repair
+
+Changing a vector's contribution along one direction uses three operations: dot product, projection, and scaled subtraction.
+
+### 8.1 A direction is a pattern across coordinates
+
+Suppose:
+
+$$
+u=
+\begin{pmatrix}
+1\\
+0
+\end{pmatrix}.
+$$
+
+This points along the first coordinate.
+
+Let:
+
+$$
+w=
+\begin{pmatrix}
+1\\
+3
+\end{pmatrix}.
+$$
+
+It contains:
+
+- $1$ unit along $u$.
+- $3$ units perpendicular to $u$.
+
+In a real model, a direction typically spans many coordinates and need not align with a single neuron or coordinate axis.
+
+### 8.2 The dot product measures the component
+
+For a unit-length vector $u$, the scalar
+
+$$
+u^Tw
+$$
+
+gives the signed component of $w$ along $u$.
+
+Here:
+
+$$
+u^Tw
+=
+\begin{pmatrix}
+1&0
+\end{pmatrix}
+\begin{pmatrix}
+1\\
+3
+\end{pmatrix}
+=
+1.
+$$
+
+Multiply that scalar by $u$:
+
+$$
+u(u^Tw)
+=
+\begin{pmatrix}
+1\\
+0
+\end{pmatrix}.
+$$
+
+This is the vector component along $u$.
+
+### 8.3 Remove that component
+
+Subtract it:
+
+$$
+w'
+=
+w-u(u^Tw).
+$$
+
+Therefore:
+
+$$
+w'
+=
+\begin{pmatrix}
+1\\
+3
+\end{pmatrix}
+-
+\begin{pmatrix}
+1\\
+0
+\end{pmatrix}
+=
+\begin{pmatrix}
+0\\
+3
+\end{pmatrix}.
+$$
+
+The component along $u$ is now zero while the perpendicular component remains. Geometrically, this is orthogonal projection onto the space perpendicular to $u$.
+
+### 8.4 What if the direction is not unit length?
+
+For any nonzero direction $d$, the denominator accounts for its length:
+
+$$
+\operatorname{proj}_d(w)
+=
+d\frac{d^Tw}{d^Td}.
+$$
+
+Or normalize $d$ to unit length first, where $\|d\|$ denotes its length:
+
+$$
+u=\frac{d}{\|d\|}.
+$$
+
+For:
+
+$$
+d=
+\begin{pmatrix}
+3\\
+4
+\end{pmatrix},
+$$
+
+we have:
+
+$$
+\|d\|=\sqrt{3^2+4^2}=5,
+\qquad
+u=
+\begin{pmatrix}
+3/5\\
+4/5
+\end{pmatrix}.
+$$
+
+### Exercise: remove a diagonal component
+
+Let:
+
+$$
+d=
+\begin{pmatrix}
+1\\
+1
+\end{pmatrix},
+\qquad
+w=
+\begin{pmatrix}
+3\\
+1
+\end{pmatrix}.
+$$
+
+Find the part of $w$ along $d$, then subtract it.
+
+*Hint: compute $d^Tw$ and $d^Td$ separately.*
+
+### Check after trying
+
+$$
+d^Tw=3+1=4,
+$$
+
+$$
+d^Td=1+1=2.
+$$
+
+Thus:
+
+$$
+\operatorname{proj}_d(w)
+=
+\begin{pmatrix}
+1\\
+1
+\end{pmatrix}
+\frac42
+=
+\begin{pmatrix}
+2\\
+2
+\end{pmatrix}.
+$$
+
+Subtract:
+
+$$
+w'
+=
+\begin{pmatrix}
+3\\
+1
+\end{pmatrix}
+-
+\begin{pmatrix}
+2\\
+2
+\end{pmatrix}
+=
+\begin{pmatrix}
+1\\
+-1
+\end{pmatrix}.
+$$
+
+Check perpendicularity:
+
+$$
+d^Tw'=1-1=0.
+$$
+
+Omitting the denominator and subtracting $(4,4)^T$ would overcorrect because $d$ is not normalized.
+
+### 8.5 Generalize from removal to counteraction
+
+Introduce an edit strength $\alpha$, pronounced "alpha":
+
+$$
+\boxed{
+w'=w-\alpha u(u^Tw).
+}
+$$
+
+For unit $u$:
+
+$$
+u^Tw'=(1-\alpha)u^Tw.
+$$
+
+Therefore:
+
+| Strength | Effect on the component along $u$ |
+|---|---|
+| $\alpha=0$ | Leave it unchanged |
+| $\alpha=0.5$ | Halve it |
+| $\alpha=1$ | Remove it |
+| $\alpha=2$ | Reverse it with equal magnitude |
+| $\alpha=6$ | Reverse it with five times the original magnitude |
+
+On the toy vector $w=(1,3)^T$ with the horizontal direction selected, raising the strength from $0$ to $1$ and then to $6$ changes only the horizontal component. The Qwen patch used strength $6$, reversing the selected component more strongly than projection removal.
+
+---
+
+## 9. What the successful Qwen edit actually did
+
+### Step 1: compare productive and repetitive contexts
+
+The editor built four matched pairs of math traces where one continuation made progress and the other repeated. It ran these fixed traces through the original model to record internal activations (teacher forcing, where the model reads a supplied sequence rather than sampling each next token).
+
+At each relevant layer, it averaged the representations of the last 36 tokens of each trace and took a difference:
+
+$$
+d
+=
+\text{average representation during repetition}
+-
+\text{average representation during progress}.
+$$
+
+Then it normalized that difference:
+
+$$
+u=\frac{d}{\|d\|}.
+$$
+
+Each edited layer used its own direction vector rather than sharing one vector across the model. [Source: executed experiment code](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/tool_calls.jsonl)
+
+**Interpretation:** $u$ is an activation pattern correlated with repetition. Correlation alone does not prove causation; weight interventions test whether changing it alters generation.
+
+### Step 2: identify which features write that pattern
+
+Recall:
+
+$$
+Wa=\sum_j a_jw_j.
+$$
+
+For feature $j$, two things matter:
+
+1. Does it activate more strongly during repetition?
+2. Does its output vector point along the repetition direction?
+
+The selection score multiplied both terms:
+
+$$
+s_j
+=
+\underbrace{u^Tw_j}_{\text{writing along the direction}}
+\underbrace{
+\left(
+\overline{a_j^{\text{repeat}}}
+-
+\overline{a_j^{\text{productive}}}
+\right)
+}_{\text{extra activation during repetition}}.
+$$
+
+The code selected the 64 highest-scoring neurons in each of two layers using the activation averages from the matched traces. [Source: neuron-selection code](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/tool_calls.jsonl)
+
+### Exercise: choose the stronger candidate
+
+Suppose:
+
+| Neuron | Writing component $u^Tw_j$ | Extra activation during repetition |
+|---|---:|---:|
+| A | 3 | 2 |
+| B | 100 | 0 |
+| C | $-2$ | 4 |
+
+Which receives the largest score?
+
+### Check after trying
+
+$$
+s_A=3\cdot2=6,
+$$
+
+$$
+s_B=100\cdot0=0,
+$$
+
+$$
+s_C=(-2)\cdot4=-8.
+$$
+
+Neuron A gets the highest score. Neuron B aligns with the direction but shows no extra activation during repetition, and Neuron C pushes in the opposite direction.
+
+### Step 3: edit selected output columns
+
+The final patch changed 64 columns in each of:
+
+```text
+layers.15.mlp.down_proj.weight
+layers.19.mlp.down_proj.weight
+```
+
+For each selected column:
+
+$$
+\boxed{
+w_j'=w_j-6u(u^Tw_j).
+}
+$$
+
+The successful run's [final report](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/work_notes/final_report.md) specifies that strength and those two matrices. Each column has 1,024 entries, so the edit touches:
+
+$$
+2\cdot64\cdot1024=131{,}072.
+$$
+
+Out of roughly 800 million parameters, that is:
+
+$$
+\frac{131{,}072}{800{,}000{,}000}\times100
+\approx0.0164\%.
+$$
+
+That is the origin of the headline percentage. [Hugging Face](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming)
+
+### Step 4: understand the matrix operation
+
+Let $m$ be a mask: $1$ for a selected column, $0$ otherwise.
+
+The implementation has the form:
+
+```python
+projection = u @ W
+correction = np.outer(u, mask * projection)
+W_edited = W - 6 * correction
+```
+
+An outer product multiplies a column vector by a row vector to build a matrix:
+
+$$
+\begin{pmatrix}
+1\\
+2
+\end{pmatrix}
+\begin{pmatrix}
+3&4
+\end{pmatrix}
+=
+\begin{pmatrix}
+3&4\\
+6&8
+\end{pmatrix}.
+$$
+
+Every column of this matrix is a multiple of $(1,2)^T$. That makes the correction matrix rank at most one before rounding: it modifies only one output direction per edited matrix. Only the correction has this structure, not the entire model.
+
+### A subtle but essential distinction
+
+The edit does not subtract a fixed constant vector from every token. For an intermediate activation vector $a$:
+
+$$
+W'a-Wa
+=
+-6u
+\sum_{j\in S}a_j(u^Tw_j),
+$$
+
+where $S$ is the selected neuron set. The push scales with how strongly those selected neurons fire on the current token. Preserving each selected column's perpendicular component in the edited matrix also does not freeze perpendicular activations in later layers, because downstream nonlinearities and future tokens respond to the updated stream.
+
+---
+
+## 10. Why believe this repaired repetition?
+
+To check whether the edit worked by cutting repetition, the authors tested several controls:
+
+- Sweeping edit strengths.
+- Editing each layer on its own.
+- Comparing against a random direction with matched edit norm.
+- Evaluating on held-out practice problems after freezing the edit.
+
+On a 1,152-problem held-out confirmation set, 579 problems flipped from wrong to right, and 576 of them had originally been truncated at the token cap. Aggregate accuracy on outputs that already finished before the edit did not change. [Source: successful run's final assessment](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/final_answer.md)
+
+That strongly supports the interpretation that the intervention rescued many answers by reducing a generation failure. It still does not prove that every form of repetition shares one mechanism, or that this direction represents "looping" in every context.
+
+### The unsuccessful follow-up is informative
+
+The authors also tried Gemma 3 270M, which rarely hit its token cap. Its MATH-500 score was 6.8% before editing and 7.0 to 7.7% across editing runs, showing no clear gain. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/gemma_followup/README.md)
+
+That contrast separates two failure modes:
+
+| Failure type | What may be needed |
+|---|---|
+| The model often knows enough but gets trapped in repetitive generation | A targeted execution repair may help |
+| The model finishes but makes mathematical mistakes | It may need additional learning or a different intervention |
+
+The Gemma follow-up does not show that direct editing can never add capabilities, only that these experiments did not demonstrate it.
+
+---
+
+## 11. What the other experiments demonstrate
+
+Each experiment gave the editor different inputs and tools. Keep their conditions separate to avoid overgeneralizing:
+
+| Experiment | What the editor could inspect or use | Reported outcome and its scope |
+|---|---|---|
+| **1. Identify a game** | Small policy weights, architecture, input/action descriptions; no tools | Correct game identification for four selected networks. [rikiyat.github.io](https://rikiyat.github.io/blog/weight-space-programming/) |
+| **2. Identify generated shapes** | Small diffusion-network weights; no target forward passes | Correct coarse shape categories, but it described outlines when the models generated filled shapes. [Experiment 2](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp2_diffusion/README.md) |
+| **3. Identify training text** | Middle transformer blocks and Python linear algebra; no target forward passes | Correct choice among Python, romanized Japanese, and English in two attempts per model. [Experiment 3](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp3_language/README.md) |
+| **4. Repair a car controller** | Weights, scene, rules, baseline failures, one trajectory | A single edit succeeded in 1,024 tested episodes after the original policy failed. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp4_fix_car/README.md) |
+| **5. Write game policies** | Game rules, source code, network specification; no trial gameplay | Written networks beat the reported 100-million-step PPO baselines in three games. [GitHub](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp5_write_from_scratch/README.md) |
+| **6. Write a Go policy** | Rules, Python, and varying feedback budgets | Zero-feedback policies were useful; beating the tuned equal-sized PPO baselines required the feedback-assisted run. [Experiment 6](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp6_go_from_scratch/README.md) |
+| **7. Identify sycophancy** | Initially weights alone; later attention patterns on supplied texts | Weights-only investigations failed to distinguish reliably; attention probes enabled three correct pair classifications. [Experiment 7](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp7_sycophancy/README.md) |
+
+### Why the car example is interesting
+
+To clear a ramp, the car had to back up before accelerating. Where the original policy kept repeating an unsuccessful approach, the editor wired a control pathway through selected units that used position and velocity to create a run-up before handing control back to the trained network. That edit changed about 3.5% of the actor's parameters. The result applies to the tested task distribution, without guaranteeing success in arbitrary driving situations. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp4_fix_car/README.md) This resembles our paddle example at a more sophisticated level: devise a controller, then encode it into a network.
+
+### Why the game results need their full comparison
+
+The reported scores were:
+
+| Game | PPO: 100M steps | PPO: 500M steps | Directly written network |
+|---|---:|---:|---:|
+| Breakout | 15.8 | 25.3 | **517.4** |
+| Seaquest | 5.6 | **88.9** | 53.1 |
+| Asterix | 25.8 | **55.2** | 53.2 |
+
+The written networks beat all three 100M-step baselines, but only on Breakout did they also beat the 500M-step baseline. [GitHub](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp5_write_from_scratch/README.md) Because the editor received the game source code and knew the games, it could design heuristics and compile them into weights.
+
+### What is the minimum diffusion background?
+
+A diffusion generator repeatedly applies a learned denoiser to turn random noise into structured output. In the shape experiment, the editor identified the coarse shape category from the denoiser's weights, yet mistook filled shapes for outlines: spotting a structural signature in weights is weaker than reconstructing a generator's full output behavior.
+
+### What is the minimum PPO background?
+
+A policy maps a state to action probabilities. PPO (proximal policy optimization) is a reinforcement-learning algorithm that updates a policy from interaction rewards while limiting how sharply its action probabilities shift on each step. One environment step is a single interaction with the simulator, so it is not directly comparable to an LLM token, a full game, or a batched evaluation call.
+
+### Why the sycophancy result matters
+
+Sycophancy means following the user's suggested answer even when that suggestion is wrong. Seeing attention shift toward the suggestion shows where a head reads from, not whether that head causes the behavior: a head can attend strongly to a token while writing almost nothing through its value-output transformation. Causal interventions are stronger evidence than attention maps alone.
+
+---
+
+## 12. The distinctions you need for mastery
+
+### A. Gradient-free versus feedback-free
+
+The Qwen editor inspected practice problems, activations, and candidate outcomes. Its budget of "50 forward passes" meant 50 forward-tool requests, where one request could generate up to 512 multi-token responses or inspect up to 1,024 supplied texts. The winning run used 158 turns and 80 Python executions over 20.3 hours of elapsed time, including 4.7 hours of service unavailability. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md) This is gradient-free editing, yet it still involved a substantial experimental process.
+
+### B. Sparse versus small
+
+Sparse counts how many coordinates change; small measures how much the transformation or output changes. Changing one branch condition in a program can redirect nearly every run. Likewise, editing a tiny fraction of parameters can produce a large behavioral shift and does not guarantee that all other capabilities stay intact:
+
+$$
+\boxed{
+\text{0.016\% of parameters changed}
+\not\Rightarrow
+\text{0.016\% of behavior changed}.
+}
+$$
+
+### C. Low rank versus LoRA
+
+Both the Qwen patch and LoRA write an update using low-rank factors:
+
+$$
+W'=W+BA.
+$$
+
+Ordinary LoRA learns those factors by gradient descent, whereas the Qwen patch constructed its rank-one update directly from activation contrasts. Low rank describes the shape of an update, not how it was found. [arxiv.org](https://arxiv.org/abs/2106.09685)
+
+### D. Weight editing versus activation steering
+
+Activation steering modifies temporary internal activations during an inference run; weight editing modifies the stored matrices that produce future activations. They can be related mathematically, while intervening at different places.
+
+### E. Direct construction versus reverse engineering
+
+Writing our paddle policy was straightforward because we chose the algorithm and coordinate meanings up front. Reading a trained network is harder because its computation can be distributed and redundant across many units. Building a network from scratch does not imply an ability to reverse engineer arbitrary trained networks.
+
+### F. Novel combination versus entirely new ingredients
+
+Each ingredient has prior work behind it:
+
+- Mechanistic interpretability analyzes internal circuits.
+- Model editing patches targeted behaviors.
+- Tracr compiles human-readable programs into transformer weights, and it predates this work. [arxiv.org](https://arxiv.org/abs/2301.05062)
+- Low-rank updates represent structured parameter deltas.
+
+The interesting claim here is that a general-purpose LLM can perform several parts of this work itself, without specialized training as a weight generator.
+
+---
+
+## 13. How to evaluate the claims like a researcher
+
+### 13.1 Separate three levels of understanding
+
+Ask which level of claim the evidence supports:
+
+1. **Identification:** "This network probably plays a paddle game."
+2. **Prediction:** "On this input, it will choose right."
+3. **Causal explanation:** "This pathway computes the position error, and changing it produces the predicted behavioral shift."
+
+A correct label alone does not establish a causal explanation.
+
+### 13.2 Compare against a strong practical baseline
+
+Original and edited Qwen were tested under matched decoding settings, which supports a comparison under those conditions. However, the official model card recommends temperature $1.0$ for thinking text tasks, whereas the experiment used $0.6$, and notes that this small model can still enter thinking loops under recommended settings. [Hugging Face](https://huggingface.co/Qwen/Qwen3.5-0.8B)
+
+To judge whether weight editing is the best practical fix, also compare against:
+
+- Tuned decoding parameters.
+- Thinking versus non-thinking modes.
+- Repetition detection or other simple generation controls.
+- Gradient-based fine-tuning under a similar resource budget.
+
+This does not invalidate the measured edit effect. It determines whether editing is the best available intervention.
+
+### 13.3 Keep development data separate from final evaluation
+
+A sound workflow is:
+
+```mermaid
+flowchart TD
+    A["Inspect failures and internals"] --> B["Propose structured edit"]
+    B --> C["Test on development cases"]
+    C -->|"Revise"| A
+    C -->|"Select"| D["Freeze the edit"]
+    D --> E["Evaluate on held-out tasks"]
+```
+
+Repeatedly checking the evaluation set while choosing layers, directions, or alpha turns that test set into development data. Final evaluation must run on held-out tasks after the edit is frozen.
+
+### 13.4 Measure more than accuracy
+
+For a repetition repair, track metrics that test the proposed mechanism alongside accuracy:
+
+- Correct-answer rate and fraction hitting the token cap.
+- Repetition rate, output length, and latency.
+- Regressions on problems the unedited model solved.
+- Performance on unrelated tasks.
+
+### 13.5 Count the whole cost
+
+A full cost comparison includes:
+
+$$
+\text{editor inference}
++
+\text{target inference}
++
+\text{analysis}
++
+\text{development attempts}
++
+\text{verification}.
+$$
+
+Skipping backpropagation does not automatically mean lower total cost.
+
+### 13.6 Understand the current evidence boundary
+
+The repository notes that earlier prompt trials are not all included in the released logs, making these runs exploratory demonstrations rather than a measure of success rate across models and tasks. [GitHub](https://github.com/RikiyaT/lm_understand-logs)
+
+Our assessment is that the clearest near-term uses are:
+
+- Targeted repairs of localized failure modes.
+- Constructing small controllers from known rules.
+- Proposing interpretable weight initializations.
+- Automating parts of mechanistic investigation.
+
+Installing broad new capabilities, auditing models reliably from weights alone, and replacing gradient-based training remain open research problems.
+
+---
+
+## 14. Use the demo as an active-learning experiment
+
+Before opening the demo, solve its example yourself:
+
+> Positive integers $x,y$ satisfy $x+y=20$ and $xy=91$. Find $x^2+y^2$.
+
+**Try first:** Can you solve it without finding $x$ and $y$ separately?
+
+*Hint: expand $(x+y)^2$.*
+
+### Check after trying
+
+$$
+(x+y)^2=x^2+2xy+y^2.
+$$
+
+Rearrange:
+
+$$
+x^2+y^2=(x+y)^2-2xy.
+$$
+
+Substitute:
+
+$$
+x^2+y^2=20^2-2(91)=400-182=\boxed{218}.
+$$
+
+Now test both checkpoints in the [live comparison](https://huggingface.co/spaces/ML-Intern-lab/weight-space-programming-demo) and check:
+
+1. Does each model reach a final answer?
+2. Does it repeat a sentence or reasoning pattern?
+3. Is the final answer correct?
+4. Does a repeated run behave differently?
+
+The edited model need not win on every individual prompt. The live app defaults to 4,096 tokens (the benchmark used 32,768) and omits the benchmark's presence penalty of $1.5$; the recorded-runs tab shows full-protocol examples. [Source: demo implementation](https://huggingface.co/spaces/ML-Intern-lab/weight-space-programming-demo/raw/main/app.py)
+
+The model card also reports an independent full MATH-500 rerun of the edited checkpoint at 51.0%, paired with a 25-problem smoke test on the original model rather than a second full baseline evaluation. [Hugging Face](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming)
+
+### How to read the paper's log repository
+
+For the Qwen experiment, read in this order:
+
+| File or folder | What it helps you understand |
+|---|---|
+| `exp8_math/README.md` | Rules, tools, evaluation protocol, and run mapping |
+| `runs/p50c/final_answer.md` | Final choice and the main supporting evidence |
+| `runs/p50c/work_notes/final_report.md` | Exact intervention and confirmation results |
+| `runs/p50c/tool_calls.jsonl` | Executed computations behind the method |
+| `runs/p50c/edit/` | The released parameter difference |
+| `exp8_math/code/apply_edit.py` | Reconstructing and verifying the edited tensors |
+| `exp8_math/eval/` | Benchmark scores and individual outcomes |
+
+The reconstruction script verifies tensor hashes so you can confirm that the rebuilt weights match the released checkpoint. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md) For understanding, the final report plus the edit script are more useful at first than the full conversation log.
+
+---
+
+## 15. Transfer exercise: predict collateral effects
+
+Start with:
+
+$$
+W=
+\begin{pmatrix}
+1&0\\
+0&1
+\end{pmatrix}.
+$$
+
+This is the identity matrix:
+
+$$
+Wx=x.
+$$
+
+We want the input
+
+$$
+v=
+\begin{pmatrix}
+1\\
+1
+\end{pmatrix}
+$$
+
+to produce:
+
+$$
+\begin{pmatrix}
+3\\
+1
+\end{pmatrix}
+$$
+
+instead of $(1,1)^T$.
+
+One possible edit is:
+
+$$
+\Delta W=
+\begin{pmatrix}
+1&1\\
+0&0
+\end{pmatrix}.
+$$
+
+Then:
+
+$$
+W'=W+\Delta W
+=
+\begin{pmatrix}
+2&1\\
+0&1
+\end{pmatrix}.
+$$
+
+Indeed:
+
+$$
+W'v=
+\begin{pmatrix}
+3\\
+1
+\end{pmatrix}.
+$$
+
+**Try first:** Does the edit also change the outputs for:
+
+$$
+x_A=
+\begin{pmatrix}
+1\\
+-1
+\end{pmatrix},
+\qquad
+x_B=
+\begin{pmatrix}
+1\\
+0
+\end{pmatrix}?
+$$
+
+### Check after trying
+
+For $x_A$:
+
+$$
+\Delta Wx_A
+=
+\begin{pmatrix}
+1&1\\
+0&0
+\end{pmatrix}
+\begin{pmatrix}
+1\\
+-1
+\end{pmatrix}
+=
+\begin{pmatrix}
+0\\
+0
+\end{pmatrix}.
+$$
+
+Its output is unchanged.
+
+For $x_B$:
+
+$$
+\Delta Wx_B
+=
+\begin{pmatrix}
+1\\
+0
+\end{pmatrix}.
+$$
+
+Its output changes.
+
+An edit designed for one target behavior also changes other inputs. That shared reach is the source of both useful generalization and possible collateral effects, and you need evaluation to determine which one you obtained.
+
+---
+
+# Compact mastery check
+
+Test yourself on these questions before reading the answers.
+
+## 1. Recall: explain weight-space programming to a beginner
+
+What does the editor do, and what does "no training" mean in this study?
+
+### Check after trying
+
+> A neural network computes a function defined by its architecture and stored numerical weights. Weight-space programming asks a general-purpose LLM to inspect those weights and directly construct targeted updates, without running gradient-based training on the target model. Several experiments still used practice inputs, internal activation probes, and iterative testing, and both the editor and any pretrained targets were trained beforehand.
+
+Your answer should separate the editor from the target and distinguish "no gradients" from "no feedback."
+
+## 2. Intuition: why could a math repair improve instruction following?
+
+### Check after trying
+
+Repetitive looping can cut off generation on any task before the model emits its final answer. Suppressing that shared generation failure lets instruction-following responses finish too, without teaching each instruction separately. This is a mechanism-based prediction: improvement should transfer to other tasks affected by the same failure.
+
+## 3. Calculation: compute the directional edit
+
+Let:
+
+$$
+u=
+\begin{pmatrix}
+1\\
+0
+\end{pmatrix},
+\qquad
+w=
+\begin{pmatrix}
+2\\
+5
+\end{pmatrix}.
+$$
+
+Calculate:
+
+$$
+w'=w-\alpha u(u^Tw)
+$$
+
+for $\alpha=1$ and $\alpha=6$.
+
+### Check after trying
+
+First:
+
+$$
+u^Tw=2.
+$$
+
+For $\alpha=1$:
+
+$$
+w'=
+\begin{pmatrix}
+2\\
+5
+\end{pmatrix}
+-
+\begin{pmatrix}
+2\\
+0
+\end{pmatrix}
+=
+\boxed{
+\begin{pmatrix}
+0\\
+5
+\end{pmatrix}
+}.
+$$
+
+For $\alpha=6$:
+
+$$
+w'=
+\begin{pmatrix}
+2\\
+5
+\end{pmatrix}
+-
+6
+\begin{pmatrix}
+2\\
+0
+\end{pmatrix}
+=
+\boxed{
+\begin{pmatrix}
+-10\\
+5
+\end{pmatrix}
+}.
+$$
+
+The original component $2$ becomes $-10$, which is $-5$ times the original. The perpendicular component remains $5$.
+
+## 4. Discrimination: which claim is justified?
+
+Someone says:
+
+> "Only 0.016% of the weights changed, so all other capabilities must be preserved."
+
+What is wrong with the argument?
+
+### Check after trying
+
+The percentage counts how many parameter coordinates changed, not how many behaviors depend on them. A small set of edited columns can sit on a pathway used across many tasks, so capability preservation must be tested on benchmarks rather than inferred from parameter count. A small edit count is useful evidence that the intervention is localized, but it is not a preservation guarantee.
+
+## 5. Transfer: design a convincing experiment
+
+You have a small model that often loops during tool-use planning. You propose editing a repetition-associated direction. What would make the result convincing?
+
+### Check after trying
+
+1. Measure the baseline failure under fixed decoding settings.
+2. Use a development split to find candidate directions and choose the edit.
+3. Check whether the proposed mechanism predicts the change in repetition rate.
+4. Compare against controls, including a norm-matched random direction and simple decoding adjustments.
+5. Freeze the edit before evaluating on held-out tasks.
+6. Report task completion, accuracy, token usage, and regressions on previously solved cases.
+7. Evaluate unrelated tasks for collateral side effects.
+8. Include both editor and target compute when comparing total cost.
+
+The explanation you should now be able to defend is:
+
+> The promising result is that a frontier LLM can sometimes inspect a network's weights and activations, isolate a structured failure mode, construct a targeted matrix edit, and validate it. In the Qwen experiment, a rank-one update to two MLP matrices substantially reduced repetitive looping under the tested decoding protocol. Whether direct weight editing can install broad new capabilities or replace gradient training remains an open question.

code/results/edit_info.json

@@ -0,0 +1,2223 @@
+{
+ "layers": {
+  "15": {
+   "u": [
+    -0.24895589,
+    0.02042487,
+    0.04845341,
+    0.04011886,
+    0.01447522,
+    -0.22025995,
+    -0.03074205,
+    -0.13840941,
+    -0.06866153,
+    0.09642404,
+    0.0353046,
+    0.0226175,
+    -0.0256597,
+    0.14359753,
+    0.02338508,
+    0.1498878,
+    0.08304029,
+    0.08268246,
+    -0.19874601,
+    -0.09147045,
+    0.07413888,
+    0.06162012,
+    0.10411097,
+    0.10747174,
+    -0.03906542,
+    -0.00092564,
+    0.05868161,
+    0.03840519,
+    -0.01002132,
+    -0.01528517,
+    0.02351971,
+    -0.10380503,
+    0.04659568,
+    -0.05271685,
+    -0.11021212,
+    -0.02139628,
+    0.05885201,
+    0.27632716,
+    -0.07798066,
+    -0.00321422,
+    -0.0532624,
+    0.02187534,
+    -0.10555955,
+    0.03803927,
+    -0.02107798,
+    -0.01953006,
+    0.11633734,
+    -0.02078464,
+    -0.01075826,
+    -0.03035684,
+    -0.0609631,
+    0.04406511,
+    -0.03191375,
+    0.01976214,
+    -0.0230045,
+    0.10779605,
+    0.02044537,
+    0.03443716,
+    -0.00222996,
+    0.04737126,
+    0.02329854,
+    -0.06319241,
+    -0.03822986,
+    0.02501477,
+    0.03194892,
+    -0.01586943,
+    0.11077951,
+    -0.01513172,
+    0.01495523,
+    -0.02717557,
+    0.00209524,
+    0.02339578,
+    -0.05470583,
+    -0.01732864,
+    -0.13191094,
+    0.00083009,
+    -0.03853427,
+    0.00820126,
+    -0.08445728,
+    -0.01159446,
+    0.04487658,
+    0.09761336,
+    0.01805925,
+    -0.0599457,
+    -0.00016578,
+    0.12068176,
+    0.07060459,
+    -0.01015618,
+    0.00422971,
+    0.00676035,
+    -0.03283456,
+    -0.04117064,
+    -0.03248039,
+    0.01022426,
+    0.04163227,
+    0.02309724,
+    -0.06292485,
+    0.04230608,
+    -0.03557596,
+    -0.00750133,
+    -0.01376741,
+    -0.0576286,
+    -0.01923029,
+    0.01580647,
+    -0.03461713,
+    -0.02074857,
+    -0.00220167,
+    0.02275244,
+    -0.05322073,
+    -0.02142559,
+    0.00997804,
+    0.06086341,
+    0.01686513,
+    -0.01301738,
+    -0.0158516,
+    0.08372771,
+    -2.635e-05,
+    -0.03974802,
+    -0.08011906,
+    0.05203765,
+    0.02601849,
+    -0.00085876,
+    -0.01701879,
+    -0.01993006,
+    -0.03135775,
+    0.02044019,
+    -0.00596576,
+    -0.04512589,
+    0.02383405,
+    0.0689159,
+    0.00962629,
+    -0.02303838,
+    0.02315754,
+    0.01817906,
+    0.00915363,
+    -0.02060907,
+    -0.00802063,
+    -0.06477001,
+    0.0903992,
+    0.00209243,
+    0.02819165,
+    0.04408683,
+    -0.00083157,
+    0.00915311,
+    -0.02369876,
+    0.00748562,
+    0.0616214,
+    0.02288214,
+    -0.01104823,
+    -0.01828848,
+    0.00216472,
+    0.00094484,
+    0.01034287,
+    0.01963162,
+    -0.02591415,
+    0.0176215,
+    -0.04005237,
+    0.03766043,
+    -0.00072997,
+    0.008385,
+    0.04705599,
+    -0.02323771,
+    0.03808962,
+    0.01465922,
+    -0.02919623,
+    0.01405872,
+    0.09836871,
+    0.03246114,
+    -0.03860031,
+    0.02028195,
+    -0.03454779,
+    -0.04695908,
+    -0.01388437,
+    0.01351815,
+    -0.00994505,
+    -0.00924062,
+    0.01713493,
+    -0.02566713,
+    0.01801864,
+    -0.00727941,
+    -0.03246095,
+    -0.00945351,
+    0.02356717,
+    -0.01617906,
+    0.0002673,
+    0.0152512,
+    -0.05930668,
+    -0.02066182,
+    0.03876992,
+    -0.01206058,
+    -0.0229992,
+    -0.01304931,
+    0.03253681,
+    0.01845163,
+    -0.02612325,
+    0.03437841,
+    0.00318938,
+    -0.02345755,
+    0.0103406,
+    -0.0547941,
+    -0.01196721,
+    0.03946429,
+    -0.01932701,
+    -0.02480044,
+    -0.03023697,
+    0.01206645,
+    0.01655773,
+    -0.03254522,
+    -0.06123636,
+    -9.718e-05,
+    0.04361208,
+    0.01765037,
+    0.00690927,
+    -0.01908739,
+    0.00710973,
+    -0.01635341,
+    -0.01807397,
+    0.03195731,
+    -0.00062558,
+    -0.07525031,
+    -0.0073614,
+    0.02113813,
+    0.01171023,
+    -0.02401062,
+    0.00238248,
+    -0.07541478,
+    0.01359196,
+    0.00859964,
+    -0.03433002,
+    -0.0139362,
+    -0.0203502,
+    0.04897985,
+    0.0189726,
+    0.03786045,
+    0.00315449,
+    0.02159249,
+    0.00791335,
+    -0.0041802,
+    -0.04222211,
+    -0.00197757,
+    -0.01045728,
+    0.01888887,
+    0.00323757,
+    0.0024898,
+    0.01738279,
+    0.01731246,
+    -0.00074183,
+    -0.04513367,
+    0.01699476,
+    0.00029007,
+    0.0313411,
+    -0.00078065,
+    -0.04166787,
+    0.00046865,
+    0.01072607,
+    0.04978328,
+    -0.00921632,
+    -0.0318539,
+    -0.01703876,
+    -0.00962165,
+    0.0084379,
+    0.01526213,
+    -0.00175763,
+    -0.00346911,
+    -0.01330116,
+    0.0110462,
+    0.00858184,
+    0.00793702,
+    -0.02276208,
+    0.00776517,
+    0.01489741,
+    -0.03487925,
+    -0.00485283,
+    -0.01072808,
+    0.01804194,
+    0.00069952,
+    -0.0350399,
+    -0.00891272,
+    -0.05034607,
+    -0.02013264,
+    0.02382344,
+    0.00452687,
+    -0.03417827,
+    0.01371371,
+    0.01013853,
+    0.02402873,
+    0.00687579,
+    -0.00293787,
+    -0.04200112,
+    0.0155979,
+    0.02103346,
+    0.00102457,
+    -0.01553039,
+    0.0048782,
+    0.01342523,
+    -0.00700563,
+    -0.0008517,
+    -0.00815003,
+    0.022058,
+    -0.01499639,
+    0.08941843,
+    0.00295805,
+    0.01554664,
+    0.04637756,
+    -0.01786274,
+    0.00893744,
+    0.00528128,
+    0.01970699,
+    0.03162822,
+    0.01654248,
+    -0.00447435,
+    0.02589581,
+    0.0091261,
+    0.02967603,
+    -0.05372572,
+    0.0408803,
+    -0.06336509,
+    -0.06271089,
+    -0.06071565,
+    0.0155491,
+    0.00989458,
+    0.00055541,
+    0.06561002,
+    0.02666396,
+    -0.00034945,
+    -0.00383269,
+    0.06482472,
+    -0.01433675,
+    0.0087758,
+    0.0072566,
+    -0.01086841,
+    -0.01884942,
+    0.00414664,
+    0.03651181,
+    0.00429372,
+    0.00615434,
+    0.02905573,
+    0.00150529,
+    -0.01203159,
+    -0.00446294,
+    -0.00850888,
+    0.02013037,
+    -0.01540696,
+    0.00676713,
+    -0.03000972,
+    0.02548919,
+    0.0193899,
+    -0.00388361,
+    -0.0120254,
+    0.03926307,
+    -0.01654343,
+    -0.0079602,
+    -0.03867238,
+    -0.00353636,
+    -0.06303667,
+    -0.01678802,
+    0.00676866,
+    -0.02049406,
+    0.01812037,
+    -0.01209067,
+    0.01619302,
+    -0.00095294,
+    0.0351405,
+    -0.01859815,
+    -0.00705459,
+    0.00921238,
+    0.00619499,
+    -0.01505822,
+    0.00492557,
+    -0.02152072,
+    -0.00978784,
+    -0.01185161,
+    -0.01378092,
+    0.01576162,
+    0.00231337,
+    0.00048437,
+    -0.00383284,
+    0.01836542,
+    0.00245513,
+    -0.00395904,
+    -0.02486439,
+    -0.00779246,
+    -0.03026413,
+    0.00707892,
+    -0.00066427,
+    0.02269584,
+    0.00932425,
+    0.00880448,
+    -0.01355996,
+    -0.0213621,
+    -0.02409151,
+    0.00435197,
+    -0.00831101,
+    -0.00868758,
+    -0.00613131,
+    0.0235176,
+    0.00874971,
+    -0.00766997,
+    -0.00574366,
+    0.00216369,
+    -0.02224782,
+    0.03144755,
+    0.02889329,
+    0.02490773,
+    -0.02212773,
+    -0.03703013,
+    -0.004282,
+    -0.0073097,
+    0.04372522,
+    -0.00034812,
+    0.02630935,
+    -0.01894524,
+    0.02313965,
+    -0.02358787,
+    0.02153209,
+    -0.01355532,
+    -0.02822855,
+    0.00959931,
+    -0.02350557,
+    0.00530226,
+    -0.03639463,
+    -0.00146062,
+    0.01056713,
+    -0.00041511,
+    -0.06991735,
+    0.01254784,
+    -0.00036259,
+    0.00870194,
+    -0.00938099,
+    0.00348201,
+    -0.02274709,
+    0.00077291,
+    -0.00150898,
+    0.02153555,
+    0.00189581,
+    0.04270721,
+    -0.02096639,
+    -0.00875471,
+    0.01324256,
+    0.03579133,
+    -0.00585621,
+    -0.00067577,
+    0.02695031,
+    0.01527341,
+    0.01530647,
+    0.01833474,
+    0.03444716,
+    0.01472223,
+    0.01547363,
+    -0.02332466,
+    -0.01141694,
+    0.0147395,
+    -0.0099107,
+    0.00653573,
+    0.01798838,
+    -0.00588708,
+    -0.00307174,
+    -0.01062604,
+    0.01664033,
+    0.0183456,
+    0.02847986,
+    -0.01004799,
+    0.01069096,
+    -0.01371973,
+    -0.02210832,
+    0.03123316,
+    0.02425844,
+    0.00473939,
+    0.01202199,
+    0.01749886,
+    -0.02504295,
+    0.03103473,
+    -0.01667802,
+    -0.00476564,
+    0.00896814,
+    0.01514297,
+    -0.00828517,
+    0.0351658,
+    0.00495074,
+    -0.0195084,
+    -0.03341911,
+    -0.00168051,
+    -0.02753034,
+    0.01903882,
+    -0.0075416,
+    -0.0049103,
+    0.01581883,
+    0.0321242,
+    -0.00483631,
+    -0.00991875,
+    -0.00546838,
+    -0.01638026,
+    -0.02765951,
+    0.00011725,
+    0.00141833,
+    0.00129717,
+    0.02686252,
+    0.0125603,
+    0.04140906,
+    -0.03283318,
+    -0.00511294,
+    -0.01852764,
+    0.0173699,
+    0.02363337,
+    0.01911985,
+    0.02551599,
+    -0.00080893,
+    -0.00387747,
+    -0.01336184,
+    -0.00042963,
+    0.01777469,
+    -0.01526318,
+    -0.02027697,
+    -0.00069285,
+    -0.0191701,
+    0.02119613,
+    -0.01990354,
+    0.01281992,
+    -0.00026741,
+    -0.00895518,
+    0.04394981,
+    0.0006412,
+    0.00957361,
+    0.01212802,
+    0.00837842,
+    0.01858075,
+    -0.0227863,
+    -0.00424773,
+    0.00491972,
+    -0.01974545,
+    0.01062466,
+    0.00607882,
+    0.0155143,
+    -0.00043194,
+    -0.03623306,
+    -0.00583369,
+    0.01177452,
+    0.03419334,
+    0.02956153,
+    0.0052018,
+    0.01841628,
+    -0.01298493,
+    0.00708795,
+    -0.00819775,
+    0.01528713,
+    0.01948304,
+    -0.02305606,
+    0.02498595,
+    0.02640811,
+    0.01192026,
+    -0.01046528,
+    -0.05339457,
+    0.01436175,
+    -0.02520583,
+    0.02034942,
+    -0.00458668,
+    0.02091778,
+    -0.00933316,
+    0.00395203,
+    0.03259778,
+    -0.00922393,
+    0.02764433,
+    0.00442787,
+    0.03087782,
+    -0.0116332,
+    -0.00695784,
+    0.01182763,
+    -0.00538845,
+    0.00021123,
+    0.01527191,
+    0.00304669,
+    -0.03362828,
+    0.00619225,
+    -0.01377127,
+    0.00983315,
+    -0.00093796,
+    -0.03062665,
+    -0.02714591,
+    0.03252864,
+    0.0116705,
+    0.0046911,
+    0.02099744,
+    0.00110952,
+    -0.01673126,
+    -0.00478365,
+    -0.0029851,
+    -0.01008971,
+    0.01333155,
+    0.01805684,
+    -0.02784364,
+    -0.00718621,
+    0.00725112,
+    0.01517552,
+    -0.00338144,
+    -0.00584107,
+    -0.02987582,
+    0.00158235,
+    -0.00127262,
+    0.00509386,
+    0.03241724,
+    0.0053058,
+    -0.0192148,
+    0.03831203,
+    0.0062505,
+    -0.01712746,
+    0.00881974,
+    -0.00132802,
+    0.01592885,
+    -0.02010931,
+    -0.01624735,
+    0.01509205,
+    0.00294987,
+    0.00105078,
+    -0.00484741,
+    0.00536887,
+    -0.00379959,
+    0.01152481,
+    0.00751099,
+    0.02359054,
+    0.03136535,
+    -0.0063684,
+    -0.0170561,
+    -0.0185177,
+    -0.00116783,
+    0.02391142,
+    -0.00098122,
+    0.0231038,
+    -0.00058802,
+    -0.01934259,
+    0.01925419,
+    -0.00160873,
+    -0.00343772,
+    0.00204482,
+    -0.00612924,
+    0.00664135,
+    -0.00410643,
+    0.02248072,
+    -0.01219042,
+    -0.005031,
+    0.01700245,
+    0.04766306,
+    -0.00623854,
+    0.00205189,
+    -0.01746178,
+    0.01765306,
+    -0.00321955,
+    0.01712283,
+    0.03226326,
+    -0.01496384,
+    -0.02296406,
+    0.01896325,
+    -0.00347675,
+    -0.00667849,
+    -0.00348551,
+    0.00290251,
+    0.00737752,
+    0.00164614,
+    0.02763323,
+    0.00416791,
+    0.04870329,
+    -0.00788404,
+    0.02007534,
+    0.03416662,
+    -0.01308093,
+    0.02324471,
+    -0.00844425,
+    -0.01111874,
+    -0.00479347,
+    0.0400974,
+    -0.02121527,
+    -0.00885901,
+    -0.02882532,
+    -0.01977198,
+    0.00859949,
+    0.01935566,
+    -0.03194553,
+    -0.03194808,
+    -0.00403009,
+    -0.02349184,
+    -0.03737707,
+    -0.03581585,
+    -0.01526899,
+    -0.01812003,
+    -0.03087662,
+    -0.01981069,
+    0.00603969,
+    0.0297529,
+    0.00296423,
+    -0.02692392,
+    -0.01962237,
+    0.01595629,
+    0.0133781,
+    -0.00838766,
+    0.00202173,
+    -0.00669928,
+    0.01277655,
+    0.00910131,
+    -0.01616032,
+    0.00190905,
+    -0.00290908,
+    -0.00844915,
+    0.02247931,
+    0.00278589,
+    0.0074408,
+    0.01545245,
+    -0.00589412,
+    0.00073301,
+    0.0002124,
+    -0.02050213,
+    0.00781624,
+    -0.02782231,
+    -0.02541454,
+    -0.02580804,
+    0.00299428,
+    -0.0064333,
+    -0.01027469,
+    -0.00159922,
+    -0.0115659,
+    0.01340889,
+    0.00064927,
+    -0.01072332,
+    0.02129249,
+    0.00946642,
+    -0.01493452,
+    0.02495065,
+    0.01947329,
+    0.02000644,
+    0.02259371,
+    0.0192214,
+    0.03618272,
+    0.0010455,
+    -0.02290383,
+    0.00989413,
+    0.02508756,
+    0.01118085,
+    -0.00993386,
+    -0.01107015,
+    0.00862738,
+    -0.02019879,
+    -0.01283852,
+    -0.0022925,
+    0.00202893,
+    -0.02117793,
+    -0.00277756,
+    0.01195571,
+    -0.01994682,
+    -0.0079788,
+    0.01083683,
+    0.0089177,
+    -0.02156102,
+    0.01252366,
+    0.01663527,
+    -0.00813946,
+    0.01951447,
+    0.00973816,
+    -0.02092359,
+    -0.0287288,
+    -0.00458775,
+    -0.01974544,
+    -0.03533125,
+    -0.00473243,
+    0.00108676,
+    -0.00633175,
+    0.0253578,
+    0.00719606,
+    -0.00228621,
+    -0.00385442,
+    -0.01694687,
+    -0.00414391,
+    0.02107211,
+    0.00458527,
+    -0.02927144,
+    -0.00728856,
+    0.0214509,
+    -0.01268997,
+    -0.01285534,
+    -0.01931156,
+    -0.0241033,
+    -0.00974043,
+    0.0114543,
+    0.02551264,
+    -0.0184916,
+    -0.02875029,
+    -0.03214873,
+    0.01575397,
+    0.02314444,
+    -0.01654227,
+    -0.00960274,
+    -0.01798794,
+    -0.01836254,
+    0.02789288,
+    0.01121621,
+    0.02051398,
+    0.01267839,
+    0.03018006,
+    -0.00046684,
+    -0.0016793,
+    0.00511237,
+    -0.0284223,
+    -0.00453048,
+    0.00387164,
+    0.00151989,
+    0.00119731,
+    -0.01721905,
+    0.04307334,
+    0.00410935,
+    -0.02953692,
+    -0.01089871,
+    0.02228069,
+    0.00585313,
+    -0.00188712,
+    0.00644261,
+    0.00592251,
+    0.00384603,
+    0.03889028,
+    -0.02183412,
+    0.01400458,
+    -0.00418775,
+    0.00452603,
+    -0.00569817,
+    0.00766993,
+    0.02411799,
+    -0.01340204,
+    0.02915989,
+    0.02999854,
+    -0.00952892,
+    0.03975626,
+    0.00725433,
+    0.00375535,
+    -0.01060731,
+    0.01101369,
+    -0.00209802,
+    -0.00799715,
+    0.00572794,
+    0.00746325,
+    0.0195495,
+    0.03768567,
+    0.02324522,
+    -0.00865699,
+    0.01903305,
+    -0.0080013,
+    0.00185655,
+    -0.02940235,
+    0.00213212,
+    0.01456472,
+    0.00119804,
+    -0.01697334,
+    -0.02465601,
+    0.02958135,
+    -0.03395347,
+    0.02159254,
+    -0.02046661,
+    -0.02458588,
+    0.00049309,
+    -0.00332874,
+    -0.00102587,
+    -0.00399199,
+    -0.02822484,
+    0.00208327,
+    0.00223905,
+    -0.01686649,
+    0.01564872,
+    0.00826747,
+    -0.03714826,
+    0.02550041,
+    -0.00482738,
+    -0.00063024,
+    0.02527108,
+    -0.00863101,
+    0.0155461,
+    0.00435747,
+    -0.02645369,
+    0.00606709,
+    0.01549895,
+    -0.00731538,
+    0.00661131,
+    -0.02080153,
+    -0.01456788,
+    -0.01408948,
+    0.04520504,
+    0.00120645,
+    -0.03185837,
+    -0.00196832,
+    -0.03367321,
+    -0.00950671,
+    0.00269519,
+    -0.01255799,
+    0.01798061,
+    -0.0206966,
+    0.0053775,
+    -0.01866919,
+    -0.0157447,
+    0.02510054,
+    -0.00183053,
+    -0.02721807,
+    -0.00336808,
+    0.00617041,
+    -0.0037063,
+    -0.00847081,
+    0.01475876,
+    0.00360406,
+    -0.00919089,
+    0.00846259,
+    -0.00986784,
+    0.03616112,
+    -0.01348828,
+    -0.03773517,
+    0.01366153,
+    0.01557916,
+    0.00253959,
+    0.01761871,
+    0.02746272,
+    -2.889e-05,
+    -0.02247486,
+    -0.00598504,
+    0.00305881,
+    -0.0189243,
+    0.00307659,
+    -0.00270951,
+    -0.01400889,
+    0.00048907,
+    -0.0141398,
+    0.01238399,
+    -0.01219304,
+    -0.01345699,
+    -0.02739621,
+    0.00530776,
+    -0.00869406,
+    0.01128332,
+    -0.00857811,
+    0.00681696,
+    -0.02160553,
+    0.00656677,
+    -0.00244236,
+    -0.01392634,
+    0.02499205,
+    -0.01085414,
+    -0.0210641,
+    -0.01952154,
+    -0.00241756,
+    0.02450087,
+    0.00031882,
+    0.01047522,
+    -0.00239012,
+    -0.01098445,
+    0.00337774,
+    0.02112963,
+    0.02175324,
+    -0.00375673,
+    -0.0055648,
+    -0.04740867,
+    0.00146061,
+    0.00249884,
+    -0.00492785,
+    0.0217794,
+    0.0009986,
+    0.00053267,
+    0.02192291,
+    -0.04000758,
+    -0.02102595,
+    -0.01386886,
+    0.00854084,
+    -0.00585625,
+    -0.01580242,
+    0.00548004,
+    0.01337633,
+    -0.01481773,
+    -0.00783593,
+    0.01119454,
+    -0.02044921,
+    0.00733242,
+    -0.03132246,
+    0.01873356,
+    -0.00689313,
+    0.01189136,
+    0.00341645,
+    0.00868533,
+    0.01298623,
+    -0.02415536,
+    -0.00280589,
+    0.00276073,
+    -0.00506605,
+    0.00613886,
+    0.00331544,
+    0.01548592,
+    0.00503579,
+    -0.00113607,
+    0.0292991,
+    0.00820742,
+    0.03085042,
+    0.00137113,
+    -0.00751863,
+    -0.00376906,
+    -0.0054176,
+    -0.02161988,
+    -0.02108021,
+    0.01517477,
+    -0.01694685,
+    -0.0144569,
+    -0.0112995,
+    -0.00250447,
+    -0.01745691,
+    -0.03337117,
+    -0.00301234,
+    -0.02143885,
+    0.00391685,
+    0.02673941,
+    0.00387426,
+    0.00849906,
+    -0.03743575,
+    -0.00248575,
+    0.01569809,
+    0.02312513,
+    -0.01871841,
+    -0.02665746,
+    -0.02359086,
+    -0.00458622,
+    -0.02799702,
+    -0.01016588,
+    -0.00041483,
+    -0.01132115,
+    0.01678916,
+    0.02570533,
+    -0.00215934
+   ],
+   "alpha": 6.000321105305686,
+   "cols": [
+    0,
+    2,
+    11,
+    25,
+    34,
+    35,
+    38,
+    39,
+    46,
+    50,
+    53,
+    56,
+    61,
+    62,
+    63,
+    64,
+    67,
+    73,
+    100,
+    108,
+    129,
+    135,
+    156,
+    169,
+    209,
+    215,
+    264,
+    266,
+    294,
+    327,
+    328,
+    357,
+    371,
+    376,
+    412,
+    414,
+    442,
+    504,
+    505,
+    509,
+    516,
+    580,
+    590,
+    601,
+    638,
+    682,
+    738,
+    882,
+    926,
+    930,
+    1047,
+    1060,
+    1087,
+    1130,
+    1155,
+    1184,
+    1460,
+    1481,
+    1787,
+    1943,
+    2005,
+    2066,
+    2738,
+    3003
+   ],
+   "singular_values_top5": [
+    1.7488828897476196,
+    0.0009874117095023394,
+    0.0009485556511208415,
+    0.0008693150011822581,
+    0.0008093058713711798
+   ],
+   "rank1_share": 0.9999944567680359,
+   "relative_change": 0.10635697096586227,
+   "changed_entries": 65260,
+   "rebuild_alpha_6.0003_entries_differing": 5875,
+   "rebuild_alpha_6.0003_max_abs_diff": 0.00048828125,
+   "rebuild_alpha_6.0000_entries_differing": 5924,
+   "rebuild_alpha_6.0000_max_abs_diff": 0.00048828125
+  },
+  "19": {
+   "u": [
+    0.33559576,
+    -0.03464305,
+    -0.06492217,
+    -0.0758407,
+    -0.04330086,
+    0.20503135,
+    -0.01100612,
+    0.07837953,
+    0.02964944,
+    -0.07836166,
+    -0.02681363,
+    0.00668682,
+    -0.02350798,
+    -0.10140933,
+    -0.03288105,
+    -0.14028722,
+    -0.09824702,
+    -0.11770061,
+    0.20775542,
+    0.10649866,
+    -0.09821087,
+    -0.0250454,
+    -0.11596692,
+    -0.12496462,
+    0.02373243,
+    0.01544948,
+    -0.05152019,
+    -0.07280387,
+    0.0386979,
+    0.04272014,
+    -0.0081986,
+    0.10454679,
+    -0.06094285,
+    0.01446683,
+    0.10608368,
+    -0.00541141,
+    -0.08207095,
+    -0.21935181,
+    0.13905452,
+    0.0108749,
+    0.05618587,
+    -0.02780561,
+    0.06047707,
+    -0.01481501,
+    0.03299749,
+    0.06129911,
+    -0.07319687,
+    0.01751008,
+    -0.0128193,
+    -0.02267084,
+    0.04299525,
+    -0.06126266,
+    0.03324774,
+    -0.01272442,
+    0.02234195,
+    -0.10648298,
+    -0.02969377,
+    -0.02647679,
+    -0.01312392,
+    -0.03706729,
+    -0.02219274,
+    0.08834642,
+    0.04794782,
+    -0.02925058,
+    -0.04679199,
+    0.02374988,
+    -0.06003199,
+    0.02554107,
+    -0.00404207,
+    0.04579034,
+    0.01502077,
+    -0.0341815,
+    0.05213764,
+    -0.00307369,
+    0.11000572,
+    0.01039833,
+    0.05742714,
+    0.01005035,
+    0.09028663,
+    -0.00666727,
+    -0.03628933,
+    -0.07142809,
+    -0.00386145,
+    0.06643536,
+    -0.02378962,
+    -0.11098496,
+    -0.08980843,
+    0.01918232,
+    -0.01326295,
+    0.00460524,
+    0.03124605,
+    0.02860291,
+    0.04878333,
+    0.00629162,
+    -0.0320451,
+    -0.01327783,
+    0.04441041,
+    -0.03540138,
+    0.02303142,
+    -0.00700999,
+    0.02519903,
+    0.07456323,
+    0.01754859,
+    -0.03160535,
+    0.02420698,
+    0.01294643,
+    0.00773049,
+    -0.00283266,
+    0.02386064,
+    0.03280374,
+    -0.02897603,
+    -0.06339945,
+    -0.01799321,
+    -0.00071628,
+    0.01139896,
+    -0.07300469,
+    0.00444034,
+    0.06423035,
+    0.06576908,
+    -0.04132114,
+    -0.01963111,
+    -0.03308957,
+    0.02335624,
+    -0.00031846,
+    0.01106018,
+    -0.04028105,
+    -0.00064796,
+    0.0194139,
+    -0.00128262,
+    -0.06438185,
+    -0.0089235,
+    0.04541202,
+    -0.01789016,
+    -0.01187054,
+    -0.00504768,
+    0.00589341,
+    0.01914849,
+    0.06267631,
+    -0.05639358,
+    0.02688798,
+    -0.02004607,
+    -0.02166805,
+    -0.01570235,
+    -0.01208901,
+    0.03124614,
+    -0.0246713,
+    -0.0409742,
+    -0.01053298,
+    0.02573625,
+    0.00536171,
+    -0.01311869,
+    -0.0021853,
+    -0.00545144,
+    -0.02813631,
+    0.00454908,
+    -0.02800255,
+    0.05073616,
+    -0.02977879,
+    -0.00736543,
+    -0.00806641,
+    -0.039063,
+    0.01411793,
+    -0.03575376,
+    -0.02516236,
+    0.01975346,
+    0.00354834,
+    -0.04622909,
+    0.01877856,
+    0.03596357,
+    -0.02185726,
+    0.01092272,
+    0.03444305,
+    0.00924869,
+    0.01236634,
+    0.01875189,
+    0.00123886,
+    0.00015365,
+    0.02428256,
+    -0.03664358,
+    -0.03418206,
+    0.02913373,
+    0.01644926,
+    -0.00766343,
+    0.01605212,
+    0.00822296,
+    0.01068919,
+    0.04207898,
+    0.04286939,
+    -0.02778637,
+    0.01277927,
+    0.03045678,
+    0.00630897,
+    -0.02207148,
+    -0.03986497,
+    0.0315614,
+    -0.03530436,
+    0.01030708,
+    0.02566668,
+    0.00233784,
+    0.0426428,
+    0.02157422,
+    -0.04566223,
+    0.00993582,
+    0.01377577,
+    0.05212355,
+    -0.03241672,
+    0.00040995,
+    0.02884942,
+    0.05572666,
+    -0.01136362,
+    -0.02396954,
+    -0.0074198,
+    -0.01548682,
+    0.02242211,
+    0.00626146,
+    0.02337368,
+    0.00290542,
+    -0.03092874,
+    -0.0145256,
+    0.0860513,
+    0.01966276,
+    -0.04058313,
+    0.01011636,
+    0.02531578,
+    -0.02338178,
+    0.06847191,
+    -0.00949347,
+    0.00520009,
+    0.01857924,
+    0.00234352,
+    0.04116894,
+    -0.03657442,
+    -0.01991575,
+    -0.0411019,
+    -0.01821601,
+    -0.0151482,
+    -0.00906486,
+    -0.0120324,
+    0.04747478,
+    0.00392497,
+    0.00323421,
+    -0.01088273,
+    -0.00134013,
+    0.00170012,
+    -0.02111938,
+    -0.02722598,
+    -0.01383084,
+    0.00617009,
+    0.00610261,
+    -0.01101309,
+    -0.02752119,
+    0.02427548,
+    0.02947209,
+    0.01694234,
+    -0.00122762,
+    -0.05191525,
+    0.01423117,
+    0.04006915,
+    0.0091782,
+    0.02908517,
+    0.00436178,
+    0.00128976,
+    -0.02698803,
+    -0.01225181,
+    -0.00978529,
+    -0.0362439,
+    -0.02339351,
+    -0.02572778,
+    0.02842867,
+    -0.0092746,
+    -0.01647593,
+    0.04589776,
+    -0.01857447,
+    0.02361508,
+    -0.00518944,
+    -0.001908,
+    0.01267422,
+    -0.02436761,
+    0.02689295,
+    0.0360607,
+    -0.01325329,
+    0.00222294,
+    0.02398918,
+    -0.02143449,
+    -0.00039261,
+    -0.0127418,
+    -0.00827601,
+    0.02336513,
+    0.0375852,
+    0.01680375,
+    -0.00802757,
+    -0.01239895,
+    0.01229273,
+    -0.02017267,
+    -0.0171816,
+    0.0323827,
+    -0.00800342,
+    0.00295632,
+    0.0116492,
+    0.01432486,
+    -0.03085296,
+    0.0109554,
+    0.02080563,
+    -0.03709523,
+    0.03610288,
+    0.00124742,
+    0.00432386,
+    -0.01376396,
+    -0.03764868,
+    -0.02849224,
+    0.01595173,
+    0.01317301,
+    -0.01462442,
+    -0.02009147,
+    0.03663041,
+    -0.02031766,
+    0.07009073,
+    0.05813979,
+    0.03254977,
+    -0.00911946,
+    -0.03866731,
+    0.00168065,
+    -0.02967888,
+    -0.00748741,
+    0.00997796,
+    0.006374,
+    -0.06434561,
+    0.00840214,
+    0.00192905,
+    0.01429112,
+    -0.02104565,
+    0.01245597,
+    -0.03688834,
+    -0.05348125,
+    -0.01398059,
+    -0.01013292,
+    -0.01436885,
+    -0.00795575,
+    0.01192914,
+    -0.00317343,
+    0.00442111,
+    0.01588613,
+    0.01969983,
+    0.00358299,
+    0.00843091,
+    0.01197292,
+    -0.00808269,
+    -0.0120325,
+    0.01489386,
+    -0.0229412,
+    0.03391672,
+    0.00041582,
+    0.0444411,
+    0.00166488,
+    0.04418352,
+    -0.0045545,
+    0.0105409,
+    0.0129758,
+    -0.01058341,
+    0.01379588,
+    -0.05057723,
+    0.01058852,
+    -0.01741165,
+    0.03224251,
+    -0.00865988,
+    -0.00366896,
+    -0.01064463,
+    0.01051221,
+    -0.00587606,
+    -0.00057805,
+    -0.00040366,
+    0.00980105,
+    -0.01242636,
+    0.00816267,
+    -0.00530881,
+    0.0107462,
+    0.00933116,
+    -0.01976871,
+    0.00267897,
+    0.01366714,
+    5.26e-05,
+    -0.02362413,
+    0.0266993,
+    0.00520968,
+    -0.00263139,
+    -0.01041333,
+    -0.00524971,
+    -0.01508682,
+    0.0208301,
+    -0.00231445,
+    0.01323305,
+    0.0148322,
+    0.0298044,
+    0.00122306,
+    -0.0082885,
+    -0.0325029,
+    -0.01031271,
+    0.00835443,
+    0.01340494,
+    -0.02087971,
+    0.024467,
+    -0.01740892,
+    -0.00783929,
+    -0.02822377,
+    -0.00948129,
+    0.03420782,
+    -0.0123644,
+    -0.00093461,
+    -0.00221112,
+    -0.00794372,
+    -0.02958393,
+    0.01561154,
+    -0.02206564,
+    0.02932473,
+    -0.01317207,
+    0.01216475,
+    0.05258676,
+    -0.04038979,
+    0.03514089,
+    0.01140578,
+    0.04206885,
+    0.00339565,
+    -0.01667423,
+    0.01111635,
+    0.1091687,
+    -0.02007846,
+    8.865e-05,
+    -0.01171773,
+    -0.01453419,
+    -0.00614797,
+    0.00426437,
+    -0.01244039,
+    0.00979596,
+    -0.01847589,
+    -0.01246628,
+    -0.06025706,
+    0.00985852,
+    0.0059323,
+    -0.01498602,
+    -0.02150767,
+    -0.00260376,
+    0.00529287,
+    0.02987313,
+    -0.01029432,
+    -0.01696572,
+    -0.02138881,
+    -0.01986272,
+    -0.04967582,
+    -0.01098283,
+    0.01132601,
+    -0.00342707,
+    -0.02904738,
+    0.01663383,
+    0.00752067,
+    0.00172083,
+    0.02616538,
+    0.014528,
+    0.02282346,
+    -0.01403882,
+    -0.03293377,
+    -0.03202629,
+    0.00279875,
+    -0.02190205,
+    0.01519247,
+    0.01755691,
+    -0.01623092,
+    -0.01870362,
+    -0.02548813,
+    -0.0095768,
+    -0.01465815,
+    0.02063545,
+    -0.02311257,
+    0.01527115,
+    -0.00958146,
+    -0.01391149,
+    0.00278968,
+    -0.00244607,
+    -0.01574637,
+    -0.02303971,
+    0.05559954,
+    0.04239292,
+    -0.00063367,
+    0.02805366,
+    -0.01336559,
+    0.00476259,
+    0.02709932,
+    0.0059806,
+    -0.00172392,
+    -0.01144029,
+    0.00935203,
+    0.0347795,
+    0.03206396,
+    0.00100924,
+    0.00740879,
+    0.01276771,
+    -0.02085232,
+    -0.0040931,
+    0.00163903,
+    -0.01503349,
+    0.02570495,
+    0.00230727,
+    0.00422022,
+    -0.00537709,
+    -0.00793678,
+    -0.01116326,
+    -0.00440048,
+    0.02106057,
+    0.00996581,
+    -0.01476111,
+    -0.03709668,
+    -0.0181785,
+    0.01074154,
+    0.02489132,
+    0.01092589,
+    0.02339229,
+    -0.00799304,
+    0.0194321,
+    -0.01218181,
+    -0.00301016,
+    0.00131753,
+    -0.04913622,
+    -0.00534636,
+    0.00865552,
+    -0.00328565,
+    -0.01685773,
+    -0.01348882,
+    0.01474129,
+    0.03932326,
+    -0.0140139,
+    0.00828108,
+    0.00203812,
+    -0.0016696,
+    -0.02406231,
+    0.01888967,
+    0.02036756,
+    -0.00563243,
+    -0.00151437,
+    -0.01412627,
+    0.0138975,
+    0.0254507,
+    -0.00512592,
+    0.02180942,
+    0.00169177,
+    -0.00658894,
+    -0.01424513,
+    -0.01952751,
+    0.00987476,
+    -0.04215077,
+    -0.03103609,
+    -0.03450365,
+    -0.00404743,
+    0.02796279,
+    -0.01409377,
+    0.02256705,
+    0.00011281,
+    0.00330633,
+    0.00917419,
+    0.02237586,
+    -0.00486028,
+    -0.02149612,
+    0.00931693,
+    -0.02292099,
+    0.00794967,
+    -0.03169209,
+    -0.00961952,
+    0.01466525,
+    -0.01301782,
+    -0.02313516,
+    -0.01651174,
+    -0.01917735,
+    -0.00666742,
+    0.01564164,
+    -0.01308908,
+    0.01639422,
+    -0.02955538,
+    0.0093139,
+    0.01910305,
+    -0.04276742,
+    -0.02866714,
+    -0.0257399,
+    0.00713598,
+    -0.02067339,
+    -0.01301082,
+    0.00712061,
+    -0.00678727,
+    0.0025609,
+    0.0082221,
+    0.0205679,
+    0.00185209,
+    0.00875445,
+    0.03590834,
+    -0.00312236,
+    -0.05863045,
+    -0.02253641,
+    0.03322073,
+    0.03223033,
+    0.00666336,
+    -0.02435645,
+    0.00948125,
+    -0.01510644,
+    -0.01706491,
+    0.00094003,
+    -0.02911649,
+    -0.02057935,
+    0.00863724,
+    0.01242663,
+    -0.00044811,
+    -0.0096365,
+    0.0145297,
+    -0.00646989,
+    -0.01096002,
+    -0.02550274,
+    0.01688749,
+    0.01177412,
+    -0.0141668,
+    0.01863014,
+    0.00935381,
+    0.03003733,
+    -0.02493563,
+    -0.02605457,
+    -0.00891686,
+    0.0150456,
+    0.01764304,
+    -0.00095473,
+    -0.02690967,
+    0.01516299,
+    -0.01976084,
+    0.0176523,
+    6.914e-05,
+    -0.01932689,
+    -0.00354382,
+    -0.00172482,
+    -0.00987696,
+    -0.00907846,
+    0.01484565,
+    -0.014577,
+    -0.01390697,
+    0.04038126,
+    -0.0048265,
+    -0.00247458,
+    -0.01462614,
+    0.00847769,
+    0.0153567,
+    0.00848636,
+    -0.00389406,
+    -0.00768076,
+    -0.01452379,
+    -0.02445431,
+    0.01986887,
+    0.01502052,
+    -0.0102555,
+    0.03740228,
+    0.00843638,
+    0.0082063,
+    -0.00148864,
+    -0.00990654,
+    -0.0164611,
+    -0.03292083,
+    0.0217579,
+    -0.05537551,
+    -0.00516081,
+    -0.00284331,
+    -0.00884788,
+    -0.00121732,
+    -0.00610297,
+    0.00762023,
+    0.0217163,
+    0.00473982,
+    -0.01743882,
+    -0.00036,
+    -0.01559782,
+    -0.00327546,
+    0.03106499,
+    -0.00088182,
+    -0.01079302,
+    0.02392566,
+    0.03034966,
+    -0.01150686,
+    0.00968921,
+    0.03213762,
+    0.04562483,
+    0.01939051,
+    0.00593721,
+    0.03199087,
+    0.00499116,
+    -0.0,
+    -0.00065295,
+    -0.0139709,
+    0.04694964,
+    0.01491039,
+    -0.011949,
+    -0.01931378,
+    0.00424174,
+    -0.00083657,
+    0.01842102,
+    -0.03574473,
+    0.01963529,
+    0.00709899,
+    0.02524782,
+    -0.01975095,
+    -0.00502984,
+    -0.02291168,
+    -0.0025806,
+    0.0093552,
+    -0.00500408,
+    0.01239402,
+    -0.00599165,
+    -0.01745293,
+    0.0153375,
+    -0.00978259,
+    -0.00413711,
+    -0.00162416,
+    0.02107732,
+    0.00189593,
+    -0.01225992,
+    -0.00017994,
+    0.0193435,
+    -0.01469106,
+    0.00198856,
+    0.0195028,
+    0.0110589,
+    -0.0039253,
+    -0.03639412,
+    0.02734545,
+    -0.00599648,
+    0.00229398,
+    -0.00796998,
+    -0.00644312,
+    -0.02087018,
+    -0.02181302,
+    -0.00862427,
+    -0.00391108,
+    0.02068237,
+    0.026373,
+    0.00668544,
+    0.01560128,
+    -0.00049076,
+    0.0115634,
+    0.00455035,
+    0.0356213,
+    -0.00194795,
+    8.843e-05,
+    -0.00874462,
+    -0.00446164,
+    0.00585578,
+    0.02350405,
+    -0.00209728,
+    -0.02649821,
+    -0.02575775,
+    -0.00397179,
+    -0.01277832,
+    -0.01572774,
+    0.01737997,
+    0.01299238,
+    0.01312802,
+    0.01565149,
+    0.0065699,
+    0.00885125,
+    0.03822977,
+    0.01536098,
+    0.02513214,
+    -0.03427662,
+    0.00189558,
+    -0.01710075,
+    -0.0016079,
+    0.03475442,
+    0.00088984,
+    0.01690382,
+    0.00949159,
+    -0.04899343,
+    -0.00457814,
+    0.00797072,
+    0.01603921,
+    -0.00151148,
+    0.00165931,
+    0.02781364,
+    0.00976236,
+    0.00831935,
+    -0.00147864,
+    -0.01809784,
+    -0.0110757,
+    0.01264199,
+    0.01910636,
+    0.0324349,
+    -0.0100819,
+    0.00105819,
+    -0.00291676,
+    0.03201768,
+    -0.00393973,
+    0.03793667,
+    -0.02129277,
+    -0.00139829,
+    -0.03023741,
+    0.01313352,
+    -0.02107666,
+    -0.01446228,
+    0.03014615,
+    0.00998374,
+    0.02206057,
+    0.00608394,
+    0.01368592,
+    0.02053929,
+    0.00695174,
+    0.00442718,
+    -0.03535806,
+    -0.03074319,
+    0.00144798,
+    0.01089423,
+    -0.01115825,
+    0.01397229,
+    0.01757357,
+    -0.01300421,
+    -0.02727049,
+    0.00731468,
+    -0.0159276,
+    0.00104553,
+    0.00665206,
+    0.01975397,
+    -0.02238113,
+    0.01083192,
+    -0.0046235,
+    0.00074646,
+    0.00504114,
+    -0.01104437,
+    -0.00610299,
+    0.01871506,
+    -0.02166009,
+    0.0016235,
+    -0.01377717,
+    0.01032949,
+    -0.01005503,
+    -0.0014602,
+    -0.01458238,
+    0.00213626,
+    -0.00465989,
+    -0.03355782,
+    -0.01548542,
+    -0.02482919,
+    0.00147339,
+    -0.01707004,
+    0.01612629,
+    -0.00786161,
+    0.00012219,
+    -0.00965089,
+    0.00940183,
+    0.00342041,
+    0.01054248,
+    0.0242572,
+    -0.03094187,
+    0.01119379,
+    -0.01056487,
+    0.00866525,
+    0.02372767,
+    -0.00327202,
+    0.00686602,
+    -0.02390694,
+    0.02468651,
+    0.01500709,
+    0.00445921,
+    0.01888491,
+    0.01410272,
+    -0.02463877,
+    0.00823074,
+    0.03660055,
+    -0.00133192,
+    0.01779861,
+    -0.01875987,
+    0.01344969,
+    -0.01102422,
+    -0.01044747,
+    -0.01006741,
+    0.0323382,
+    -0.01484594,
+    0.00146564,
+    0.00804006,
+    0.00426122,
+    0.01966095,
+    -0.00127307,
+    -0.00580236,
+    -0.01033067,
+    0.01667609,
+    0.02310249,
+    0.01413857,
+    0.02454613,
+    0.01059382,
+    0.0026291,
+    0.00943861,
+    -3.41e-06,
+    0.0033934,
+    -0.01704926,
+    0.01335935,
+    0.00020378,
+    0.00195949,
+    -0.00776831,
+    0.01629458,
+    -0.00759811,
+    -0.00645715,
+    0.01794815,
+    -0.00121991,
+    0.01632476,
+    0.01075443,
+    -0.00039715,
+    -0.01542044,
+    -0.00214471,
+    -0.03306837,
+    -0.03733351,
+    0.02991378,
+    -0.01045495,
+    -0.02377043,
+    -0.01897853,
+    -0.00408702,
+    -0.00167294,
+    -0.00295758,
+    0.02123102,
+    -0.01130867,
+    -0.00295642,
+    -0.01521876,
+    0.02204927,
+    0.00140542,
+    0.04342286,
+    0.02072281,
+    0.03388473,
+    -0.00877712,
+    -0.00512505,
+    0.0151763,
+    0.02042603,
+    0.00843577,
+    0.01931063,
+    0.00979518,
+    0.00592844,
+    -0.02599867,
+    0.01489202,
+    -0.00266025,
+    -0.00813517,
+    -0.01534928,
+    -0.01736461,
+    0.01320925,
+    0.0355656,
+    0.02181039,
+    0.00512136,
+    -0.02044921,
+    -0.02641958,
+    0.01625668,
+    0.00261996,
+    0.01905341,
+    -0.00689309,
+    -0.02259504,
+    -0.01227041,
+    0.02103539,
+    0.00072978,
+    0.05127988,
+    -0.0009185,
+    0.02334338,
+    0.0002852,
+    -0.01754911,
+    -0.02641877,
+    -0.01193665,
+    -0.02183538,
+    -0.00192791,
+    0.01890938,
+    0.02368859,
+    0.0052784,
+    0.00044876,
+    -0.00013281,
+    0.0180367,
+    -0.01584567,
+    0.03249414,
+    0.00945242,
+    -0.01653406,
+    0.01701443,
+    -0.00781736,
+    0.01829203,
+    -0.01258584,
+    -0.00042628,
+    -0.00774671,
+    -0.00815235,
+    0.00224324,
+    0.0114774,
+    0.02551686,
+    0.01460277,
+    -0.00243856,
+    -0.00373503,
+    0.02294976,
+    -0.00273832,
+    -0.01014692,
+    -0.00593784,
+    -0.00318341,
+    -0.01724492,
+    -0.00022391,
+    -0.01840212,
+    -0.01111254,
+    0.01020701,
+    0.01218268,
+    -0.0004537,
+    0.02419603,
+    0.046826,
+    -0.01415514,
+    -0.00445342,
+    0.0266875,
+    0.02994117,
+    -0.00343018,
+    0.02188344,
+    -0.0069619,
+    0.00566721,
+    0.02015941,
+    0.02477168,
+    -0.03093816,
+    0.01301537,
+    0.02665879,
+    0.02318588,
+    -2.81e-06,
+    -0.01443277,
+    -0.02519409,
+    0.00244334,
+    0.00380622,
+    -0.00973464,
+    0.00215472,
+    0.00762649,
+    -0.01932207,
+    -0.00900411,
+    0.03000734,
+    -0.01070951,
+    -0.00942091,
+    -0.00686134
+   ],
+   "alpha": 6.0001732486833035,
+   "cols": [
+    1,
+    5,
+    8,
+    11,
+    13,
+    15,
+    17,
+    18,
+    34,
+    36,
+    44,
+    55,
+    63,
+    66,
+    69,
+    92,
+    93,
+    95,
+    98,
+    114,
+    119,
+    127,
+    145,
+    152,
+    161,
+    168,
+    171,
+    195,
+    246,
+    259,
+    265,
+    343,
+    364,
+    445,
+    506,
+    544,
+    621,
+    625,
+    636,
+    744,
+    766,
+    770,
+    799,
+    877,
+    965,
+    1003,
+    1346,
+    1410,
+    1505,
+    1704,
+    1799,
+    2072,
+    2135,
+    2175,
+    2248,
+    2314,
+    2376,
+    2437,
+    2514,
+    2595,
+    2664,
+    2684,
+    2816,
+    3343
+   ],
+   "singular_values_top5": [
+    1.3987572193145752,
+    0.0009012575028464198,
+    0.0008135843672789633,
+    0.0007443478680215776,
+    0.000705311365891248
+   ],
+   "rank1_share": 0.9999914765357971,
+   "relative_change": 0.08105870336294174,
+   "changed_entries": 64928,
+   "rebuild_alpha_6.0002_entries_differing": 6158,
+   "rebuild_alpha_6.0002_max_abs_diff": 0.00048828125,
+   "rebuild_alpha_6.0000_entries_differing": 6166,
+   "rebuild_alpha_6.0000_max_abs_diff": 0.00048828125
+  }
+ },
+ "cos_u15_u19": -0.8490049839019775
+}