# Lab 6: The weight edit is steering in disguise **Big thing:** Changing the 64 chosen columns of `down_proj` in the weight file gives the same outputs, up to rounding, as leaving the weights alone and subtracting $\alpha \cdot s(t) \cdot u$ from that layer's output during the forward pass. The single number $s(t)$ stays small while the model reasons well (about `0.1` at layer 19 on answers that finished correctly) and grows several times larger once it repeats itself (about `0.76` inside real loops). A fixed weight edit therefore acts like steering whose strength the text sets on each token. **Why it matters:** Weight editing changes the numbers stored in a model, while activation steering adds a vector to internal activations as the model runs. This edit is both at once: a steering vector $u$, scaled on every token by how hard 64 neurons push along it. That explains how changing 131,072 stored numbers pushes hard against loops and much less on ordinary reasoning, though it still pushes a little. **Time and hardware:** On one A100 GPU, parts 1 and 2 take about 1 minute, part 3 takes about 2 minutes, and part 4 takes under a minute. Every part runs on `sampler` or `trainer`. Parts 3 and 4 read the base model's practice answers in `runs/build/base`. If the machine lacks that run, make it first with `python scripts/run_jobs.py plans/sampler.json --only build/base --gpus 7` (about 24 minutes on one GPU). Redrawing the part 3 figures (`--plot-only`) runs on CPU in a few seconds. **Theory guide:** Section 12, part D (weight editing versus activation steering). ## Before you run: predict Write down a short guess for each question before running the script. 1. Load Qwen3.5-0.8B in 32-bit floats (`fp32`) and run the same texts two ways: by editing the 64 columns of `down_proj` in memory, and by leaving the weights untouched while a hook subtracts $\alpha \cdot s(t) \cdot u$ from `down_proj`'s output. Will the final next-token scores (logits) match to 16 decimal places, to about 2 decimal places, or only roughly? 2. On one MLP matrix (`layer 15`) on CPU in 64-bit floats (`fp64`), how big is the maximum difference between the edited matrix output and the hook output? 3. On the agent's 4 productive math texts versus its 4 looping math texts, is $s(t)$ (how hard the 64 neurons push along the loop direction $u$) large on both and canceled by later layers, or near zero on productive text and positive on looping text? ## Run it Connect to `sampler` or `trainer`, since every part runs on either one. Then use `nvidia-smi` to pick a free GPU index and pass it in `CUDA_VISIBLE_DEVICES`. The script reads three files from the repository: `results/edit_info.json` (the released edit's $u$, $\alpha$ and 64 columns, from `scripts/build_models.py`), `results/lab3/rebuild.json` (from Lab 3, giving the sign of $u$ that points toward looping), and `labs/data/astra_pairs.json` (the agent's 8 texts). ```bash ssh sampler . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code nvidia-smi CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py ``` Parts 1 and 2 (the default, `--parts 1,2`) finish in about 1 minute and write two files: - `results/lab6/equivalence.json` (comparing the in-memory weight edit, the steering hook, and the released `bf16` weight file) - `results/lab6/s_on_pairs.json` ($s(t)$ on every token of the 8 texts in `labs/data/astra_pairs.json`) Part 3 traces real loops token by token from the unedited model's answers in `/nvme/wsp-lab/runs/build/base`: ```bash CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3 ``` Part 3 finishes in about 2 minutes and writes four files: - `results/lab6/real_loops.json` (summary statistics across 48 looping answers) - `results/lab6/trace_one_loop.json` (per-token arrays for one answer whose loop starts near the middle) - `results/figures/lab6_real_loop.png` and `results/figures/lab6_loops_summary.png` To measure the same signals on answers that finished without looping, run part 4 on the same VM as part 3 (or pass `--parts 3,4` to run both): ```bash CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 4 ``` Part 4 takes under a minute and reads part 3's `results/lab6/real_loops.json` on the same machine to compare the two groups and draw the figure; without that file, it writes only the good-answer numbers. With `--try-name` or `--max-samples`, part 4 uses the same `results/lab6/try-/` folder as part 3 and compares against the loops saved there, or against the main `real_loops.json` if part 3 saved none. It writes two files: - `results/lab6/finished_answers.json` (summary statistics and per-answer numbers for 26 finished answers) - `results/figures/lab6_finished_vs_loops.png` (three-panel box-plot comparison of good answers and loops) To redraw the part 3 figures from the saved JSON files on CPU (`--plot-only` does not redraw part 4's figure), run: ```bash python labs/lab6_steering.py --plot-only ``` ## What you should see Parts 1 and 2 print: ```text fp32_hook_vs_edit_max_abs_logit_diff 0.00774 fp32_base_vs_edit_max_abs_logit_diff 11.4 fp32_hook_vs_edit_mean_kl 1.41e-08 fp32_base_vs_edit_mean_kl 0.0178 bf16_hook_vs_released_file_max_abs_logit_diff 0.438 bf16_hook_vs_released_file_mean_kl 0.000539 bf16_base_vs_released_file_mean_kl 0.0184 bf16_same_top_token_hook_vs_file 0.994 fp64_one_layer_max_abs_diff 3.33e-16 fp64_one_layer_output_scale 0.65 layer 15: s over the last 36 tokens, productive [0.0, 0.04, 0.04, 0.01] vs looping [0.19, 0.19, 0.42, 0.25] layer 19: s over the last 36 tokens, productive [-0.03, -0.04, 0.02, -0.01] vs looping [0.09, 0.11, 0.44, 0.3] ``` Part 1 (`results/lab6/equivalence.json`) compares the gap between the hook and the weight edit against the edit's effect on the base model: | Comparison | Max absolute difference | Mean KL divergence | What it checks | |---|---:|---:|---| | `fp64`, layer 15 `down_proj`, hook vs edit | `3.33e-16` (on outputs of scale `0.65`) | not applicable (single layer) | Exact match up to 64-bit rounding | | `fp32`, full model, hook vs in-memory edit | `0.00774` (`0.0077` in logits) | `1.41e-08` (`1.4e-8`) | Full-model rounding across 24 layers (largest logit gap about 1,500 times smaller than the edit's effect, mean KL about 1.3 million times smaller; the gap drops to `0.00134` with full IEEE `fp32`) | | `fp32`, full model, base vs in-memory edit | `11.4` (logits) | `0.0178` | Actual size of the edit's effect in `fp32` | | `bf16`, hook on base vs released `qwen35-p50c` file | `0.438` (`0.4375` in logits) | `0.000539` (`5.4e-4`) | Same top token on `99.4%` (`0.994`) of positions | | `bf16`, unedited base vs released `qwen35-p50c` file | not measured by script | `0.0184` | Actual size of the released `bf16` edit's effect (34 times larger than the hook-vs-file KL) | Part 2 (`results/lab6/s_on_pairs.json`) measures mean $s(t)$ over the last 36 tokens of each of the 8 texts, with $u$ oriented so positive values point toward looping: | Layer | Productive texts (4 texts) | Looping texts (4 texts) | |---|---|---| | Layer 15 | `0.00, 0.04, 0.04, 0.01` | `0.19, 0.19, 0.42, 0.25` | | Layer 19 | `-0.03, -0.04, 0.02, -0.01` | `0.09, 0.11, 0.44, 0.30` | On the four short productive texts, the 64 neurons barely push along $u$, or push slightly the other way. On the four looping texts, their push along $u$ reaches `0.09` to `0.44`. Because the edit subtracts $6 \cdot s(t) \cdot u$, the text sets the push strength, unlike a constant nudge. Part 4 shows that on long real answers that finish correctly, $s(t)$ is small but not zero. Each pair shares an opening (`75`, `85`, `70` and `69` tokens). On that shared prefix, $s(t)$ matches in the first three pairs (`0.0000` max difference) and differs by at most `0.0018` at layer 15 and `0.0042` at layer 19 in the fourth pair, from `bf16` rounding on the GPU when the two texts have different total lengths. After the split, $s(t)$ climbs on the looping text to peaks of `0.46, 0.47, 0.59, 0.49` at layer 15 (`0.4558, 0.4713, 0.5897, 0.4873`) and `0.36, 0.43, 0.74, 0.63` at layer 19 (`0.3620, 0.4300, 0.7376, 0.6322`). Part 3 (`results/lab6/real_loops.json` and `results/lab6/trace_one_loop.json`) takes `48` base-model answers from `runs/build/base` whose verbatim loop starts at least `1,024` tokens in. It feeds each fixed token sequence through the base and edited models (teacher forcing) and records at every token: - $s_{15}$ and $s_{19}$: how hard the 64 neurons push along $u$ at layers 15 and 19. - $\text{KL}(\text{base} \parallel \text{edited})$: how far the edited next-token probabilities moved from the base model's. - `p_next`: the probability of the actual next token under plain softmax at temperature 1. - `p_pick`: the chance that vLLM's sampler picks that next token under the paper's decoding settings (presence penalty `1.5` on tokens already used in the answer, temperature `0.6`, top-k `20`, top-p `0.95`, applied in vLLM's order by `wsp/protocol.py` `pick_prob`). Means across the `48` looping answers by stretch: | Stretch of the answer | `s15` | `s19` | `KL` | `p_next` base | `p_next` edit | `p_pick` base | `p_pick` edit | |---|---:|---:|---:|---:|---:|---:|---:| | Early (first half before the loop) | `0.096` | `0.116` | `0.021` | `0.768` (`0.7675`) | `0.750` | `0.834` | `0.821` | | Late (last 512 tokens before the loop) | `0.178` | `0.241` | `0.038` | `0.862` | `0.825` | `0.904` | `0.881` | | Inside the loop | `0.424` | `0.758` | `0.069` | `0.99975` | `0.946` | `0.99995` | `0.979` | Multiplying per-token probabilities across one repeating block of length `period` gives the cycle odds, the chance of writing one more identical cycle: | Scoring rule | Unedited, mean | Unedited, median | Edited, mean | Edited, median | |---|---:|---:|---:|---:| | Temperature 1, plain softmax | `0.992` | `0.995` | `0.205` | `0.097` | | The paper's sampler | `0.999` | `0.999` | `0.487` | `0.625` | - Under the paper's sampler and the edit, `23 of the 48` loops have odds below `0.5` of completing one more cycle. - Loop periods range from `10 to 224` tokens (median `43.5`). Longer loops break more easily: the correlation between $\log(\text{period})$ and edited sampler cycle odds is `-0.66` (`-0.657`), because a small per-token cut (`0.99995` to `0.979`) compounds across a long cycle. - The trigger rises before the verbatim loop starts. Across the early, late, and in-loop stretches, $s_{15}$ climbs `0.10 -> 0.18 -> 0.42` (`0.096 -> 0.178 -> 0.424`), $s_{19}$ climbs `0.12 -> 0.24 -> 0.76` (`0.116 -> 0.241 -> 0.758`), and $\text{KL}(\text{base} \parallel \text{edited})$ climbs `0.021 -> 0.038 -> 0.069`. Even in the early stretch, $s_{19}$ averages `0.12` and KL averages `0.021`, so the edit is never fully off; part 4 checks answers that never looped. - In the traced loop (`trace_one_loop.json`, plotted in `lab6_real_loop.png`), problem `math_train_intermediate_algebra_1137`, answer `0`, the loop starts at token `14,213` of `32,768` with period `15`. At temperature 1, one-cycle odds drop from `0.995` (unedited) to `0.64` (`0.6415`, edited), and ten-cycle odds drop from `0.988` to `0.013`. Under the paper's sampler, one-cycle odds are `0.997` (unedited) versus `0.991` (edited), and ten-cycle odds are `0.996` versus `0.970`. The ten-cycle numbers average the chance of writing each 150-token window inside the loop unchanged. Because probabilities vary along the loop, that differs from raising the one-cycle odds to the tenth power (`0.954`, `0.012`, `0.974` and `0.914`). This 15-token loop survives the edit under the paper's sampler because low temperature (`0.6`) and top-p (`0.95`) keep the repeated token far ahead of the runner-up. ![One real loop from the base model, token by token: the push s(t) at layers 15 and 19, the KL change in next-token predictions, and the probability of the token that came next for the base and edited models. All of them jump where the loop starts at token 14,213.](../results/figures/lab6_real_loop.png) ![Four panels for 48 real loops from the base model: s(t) at layer 15, the KL change and the next-token probability, each early, late and inside the loop, and the chance of one more identical cycle against loop length for the base and edited models.](../results/figures/lab6_loops_summary.png) Part 3 only examined answers that eventually looped, so it could not tell us how $s(t)$ behaves when reasoning goes well from start to finish. Part 4 (`results/lab6/finished_answers.json`) checks that, using all `26` answers in `runs/build/base` that closed `` on their own and gave the right final number. These answers think for `750` to `9,698` tokens (median `3,770`), followed by a written answer of `190` to `740` tokens after `` (median about `320`). Teacher forcing the `26` good answers through the unedited and edited models and comparing them with the `48` loops from part 3 gives these stretch means: | Group and stretch | `s15` | `s19` | `KL` | |---|---:|---:|---:| | Good answers (`26`), early (first half of thinking) | `0.062` | `0.091` | `0.011` | | Good answers (`26`), late (last 512 tokens of thinking) | `0.079` | `0.116` | `0.017` | | Good answers (`26`), written answer after `` | `0.066` | `0.108` | `0.007` | | Looping answers (`48`), early (first half before loop) | `0.096` | `0.116` | `0.021` | | Looping answers (`48`), late (last 512 tokens before loop) | `0.178` | `0.241` | `0.038` | | Looping answers (`48`), inside the loop | `0.424` | `0.758` | `0.069` | - **Small on good answers, but not zero.** Across the three stretches of the `26` good answers, $s_{19}$ averages `0.091` to `0.116`, about one seventh of the in-loop mean (`0.758`). Since $u$ has length `1` and $\alpha = 6$, the push per token has length $6 \cdot |s(t)|$. At these average levels, that is about `0.5` to `0.7` at layer 19 on good answers versus `4.5` inside loops. Mean $\text{KL}(\text{base} \parallel \text{edited})$ on good answers is `0.007` to `0.017` per token, about a tenth to a quarter of the in-loop KL (`0.069`). - **Never at loop level.** For each good answer, the script finds the hardest 64-token stretch of thinking (the highest 64-token moving average of $s$). Across the `26` answers, this peak is `0.082` to `0.316` at layer 15 (median `0.159`) and `0.147` to `0.465` at layer 19 (median `0.223`). By comparison, the median loop averages `0.427` (layer 15) and `0.754` (layer 19) inside the loop, and even the weakest of the `48` loops averages `0.374` and `0.671`. Thus `0 of the 26` good answers have any 64-token stretch reaching even the weakest loop's in-loop mean. - **Early overlap, late separation.** The 25th / 50th / 75th percentiles of $s_{19}$ are `0.076 / 0.086 / 0.109` (good early), `0.078 / 0.105 / 0.123` (good late), and `0.082 / 0.099 / 0.122` (good written), versus `0.096 / 0.111 / 0.127` (loop early), `0.166 / 0.219 / 0.267` (loop late), and `0.719 / 0.754 / 0.785` (loop inside). The two groups overlap early in thinking; in the late stretch, most loops sit above most good answers; inside the loop, there is a wide gap. The signal $s(t)$ is a graded level: low on good reasoning, higher as an answer circles, and highest inside a loop. - **What this comparison does not control for.** The `26` good answers come from `18` problems (`8` where both practice answers finished, and `10` where the other answer ran to the token limit). Because they are not matched to the problems behind the `48` loops, and problems that finish are likely easier than those that loop, problem difficulty can explain part of the gap. ![Box plots of s(t) at layers 15 and 19 and of the KL change, for 26 answers that finished with the right number (green) and 48 that ended in a loop (red). The loop boxes climb from early to inside the loop, while the good answers stay low.](../results/figures/lab6_finished_vs_loops.png) The good answer closest to loop level is `gsm8k_train_6504`, answer `1` ("Bob bought 2 show dogs for \$250.00 each to breed as a side business. The female just had a litter of 6 puppies. If he sells each puppy for \$350.00, what is his total profit?", truth `1600`). It thinks for `6,585` tokens (`6,795` total). Its late stretch averages $s_{19}$ `0.299`, above the loops' late median (`0.219`), and its peak 64-token window reaches `0.465` at layer 19 and `0.316` at layer 15. In its last 512 thinking tokens, the model circles through near-copies of the same check ("Wait, is it possible the "2 show dogs" are the *items*? Yes." / "Is it possible the "2 show dogs" are the *number* of puppies? No.") between restatements of `250 x 2 = 500`, `350 x 6 = 2100`, and `2100 - 500 = 1600`. It then writes "Okay, I'll proceed.", closes ``, and gives the right answer, `1600.00`. Here $s(t)$ rose on circling text even though the answer recovered. ## Why it works ### 1. What `down_proj` computes on each token A language model reads and writes text one token (word piece) at a time. Inside Qwen3.5-0.8B, each token carries a residual stream: a running list of 1,024 numbers that passes through 24 repeated blocks (or layers). Each block has a sequence mixer (standard attention at layers 3, 7, 11, 15, 19, and 23, and a linear DeltaNet mixer at the other 18 layers) followed by an MLP (multi-layer perceptron). The MLP maps those 1,024 numbers to 3,584 intermediate neurons. Qwen's MLP is gated: it makes two different mixes of the stream, `gate_proj` and `up_proj`, passes the first through SiLU (a smooth cousin of $\max(0, x)$ that dips slightly below zero), and multiplies the two together, number by number. Because the `up_proj` number can have either sign, a neuron's activation can be positive or negative. An output matrix called `down_proj`, of shape $1,024 \times 3,584$, then writes a 1,024-number update back into the residual stream. Let $a_j(t)$ be the neuron activation of neuron $j$ on token $t$ (how strongly it fires), and let $W[:, j]$ be column $j$ of `down_proj`. The product $W a(t)$ sums the columns scaled by their activations: $$W a(t) = \sum_{j=0}^{3583} a_j(t) \, W[:, j]$$ Column $W[:, j]$ is the 1,024-number vector that neuron $j$ adds to the residual stream per unit of activation. ### 2. Why editing columns equals subtracting $\alpha \cdot s(t) \cdot u$ Let $u$ be the unit direction in the 1,024-number residual stream (a vector of length 1 pointing toward looping), $S$ be the 64 chosen neuron indices ($|S| = 64$), and $\alpha = 6$ be the edit strength. A rank-one matrix change modifies every selected column along the single direction $u$. For each neuron $j$ in $S$, the dot product $u \cdot W[:, j]$ measures how much column $j$ writes along $u$. The weight edit subtracts $\alpha$ times that projection from each column in $S$ and leaves the other 3,520 columns unchanged: $$W'[:, j] = \begin{cases} W[:, j] - \alpha \, u \, (u \cdot W[:, j]) & \text{if } j \in S \\ W[:, j] & \text{if } j \notin S \end{cases}$$ Multiply the edited matrix $W'$ by the activations $a(t)$ on token $t$. Because summation is linear, we can pull $\alpha$ and $u$ outside the sum over the 64 chosen neurons: $$W' a(t) = \sum_{j=0}^{3583} a_j(t) \, W[:, j] - \alpha \, u \sum_{j \in S} a_j(t) \, (u \cdot W[:, j]) = W a(t) - \alpha \, s(t) \, u$$ Here $s(t)$ is one number per token: $$s(t) = \sum_{j \in S} a_j(t) \, (u \cdot W[:, j]) = u \cdot (W_S \, a_S(t))$$ In words, $W_S a_S(t)$ is the vector the 64 chosen neurons alone add to the residual stream on token $t$, and $s(t)$ is how much of that vector points along the loop direction $u$. In `wsp/hf.py`, `steer_down` attaches a PyTorch hook (a function that intercepts a layer's output during the forward pass) to `down_proj`. It precomputes $r_j = u \cdot W[:, j]$ for the 64 neurons, computes $s(t) = a_S(t) \cdot r$ on each token, and returns `output - alpha * s(t) * u`. The script gives the hook the released file's own $u$, $\alpha$ and 64 columns (from `results/edit_info.json`), so the hook and the weight edit are the same edit written two ways. ```mermaid flowchart LR A["Neuron activations a(t)
3,584 numbers per token"] --> B["Path A: Edited matrix W'
W' = W - alpha u u^T W_S"] A --> C["Path B: Unedited matrix W
y = W a(t)"] A --> D["64 chosen neurons a_S(t)
s(t) = u . (W_S a_S(t))"] C --> E["Hook subtracts
alpha * s(t) * u"] D --> E B --> F["Same MLP output
to floating-point rounding"] E --> F ``` ### 3. Why float64 is exact while float32 and bf16 show small rounding gaps The four number formats in part 1 keep different numbers of bits of precision: `fp64` has 53 bits, `fp32` has 24, `TF32` has 11, and `bf16` has 8. When part 1 runs layer 15 alone on CPU in `fp64`, `y_edit` and `y_hook` agree to `3.33e-16` on outputs of scale `0.65`. That is 3 times `2^-53` (about `1.1e-16`), where `2^-53` is one 64-bit rounding step for numbers between `0.5` and `1`. The two paths differ only because they sum terms in a different order. Across all 24 layers on GPU in `fp32`, the final logits (raw next-token scores before softmax turns them into probabilities) differ by at most `0.00774` (`0.0077`), with mean KL divergence (how far one probability distribution moved from another, where `0` is identical) of `1.41e-8` (`1.4e-8`). This gap comes mainly from the linear DeltaNet mixers, whose Triton GPU kernels default to `TF32` (11-bit precision) for `fp32` matrix multiplies. Small differences after layer 15 then pass through layers 16 to 23. Forcing full 24-bit IEEE `fp32` in Triton (`try-ieee` below) cuts the max logit gap `5.8`-fold, to `0.00134` (`0.0013`). Even with default `TF32`, the `0.00774` gap is about `1,500` times smaller than the edit's effect on the base model (`11.4`), and the `1.4e-8` mean KL is about 1.3 million times smaller than the edit's (`0.0178`). The released `qwen35-p50c` file stores weights in bf16 (16-bit brain floating point, with 8 bits of precision, or about 3 significant decimal digits). Rounding $W'$ to `bf16` on disk introduces one rounding step of up to `0.00049` per weight entry (Lab 1). Against the released `bf16` file, the `bf16` hook on the base model has a max logit gap of `0.438` (`0.4375`), mean KL `0.000539` (`5.4e-4`), and the same top token on `99.4%` of positions; the mean KL between the base model and the released file in `bf16` (`0.0184`) is `34` times larger. ### 4. How static weights act like a token-dependent trigger Adding a constant steering vector $-c \cdot u$ at layers 15 and 19 on every token would push the residual stream equally hard during clean algebra and during a loop. Because the weight edit multiplies $u$ by $-\alpha \cdot s(t)$, the token itself sets the steering dose through the activations $a_S(t)$ of the 64 chosen neurons. ```mermaid flowchart LR T["Current token t"] --> N["64 chosen MLP neurons
fire with activations a_S(t)"] N --> P["Finished real answers
s(t) small (layer 19: 0.09 to 0.12)
Edit push -6 s(t) u is small"] N --> L["Inside real loops
s(t) large (0.42 L15, 0.76 L19)
Edit push -6 s(t) u opposes u"] ``` Part 2 shows this on the 8 paired texts in `labs/data/astra_pairs.json`. Each pair shares its problem prompt and opening calculation before splitting into a productive check or a repeated sentence. In `results/lab6/s_on_pairs.json`, the `per_token` arrays match on the shared opening, apart from tiny `bf16` rounding differences in the fourth pair, then $s(t)$ stays near zero on the productive branch and climbs on the looping branch. Over the last 36 tokens of these short hand-picked texts, $s(t)$ is near zero on the productive continuations and large on the loops; on long real answers in part 4, productive $s(t)$ is higher, though still far below loop level. Part 3 tests this trigger on long base-model answers (`runs/build/base` on `build128`). It selects 48 answers that were cut at the 32,768-token limit and end in a verbatim loop (the same token block repeated at least 3 times across at least 64 tokens, where one block's length is the period) starting after at least 1,024 tokens, to focus on answers that reasoned normally before getting stuck. Using teacher forcing, which feeds a fixed token sequence through the model instead of generating token by token, it runs each answer once with the edit hooks off and once with them on. That lets us line up every token position $t$ and compare $s(t)$, $\text{KL}(\text{base} \parallel \text{edited})$, `p_next`, and `p_pick` across the early answer, the last 512 tokens before the loop starts, and the tokens inside the loop. Part 4 runs the same check on the `26` base-model answers that closed `` and got the right answer, where $s_{19}$ averages `0.09` to `0.12` per stretch and `0 of 26` reach even the weakest loop's in-loop mean (`0.671`) on any 64-token window. **Lesson:** The weight edit and subtracting $\alpha \cdot s(t) \cdot u$ at layers 15 and 19 are the same computation. The text sets the dose through $s(t)$: small on ordinary reasoning (about one seventh of the in-loop mean at layer 19), rising as the model starts to circle, and largest inside a loop, where a small per-token cut adds up to a large chance of breaking the cycle. Because $s(t)$ stays above zero on good answers, the edit still nudges them a little; [Lab 7](lab7_side_effects.md) measures that cost. ## Check yourself 1. In Lab 3, the released file's singular vector $u$ at layer 19 had the opposite sign from the mean-difference vector $u_{\text{loop}}$. Why does replacing $u$ with $-u$ flip the sign of $s(t)$ without changing the steering push $-\alpha s(t) u$ or the weight edit $W'$?
Answer Both $s(t) = u \cdot (W_S a_S(t))$ and $u$ appear once in $s(t) u$. Replacing $u$ with $-u$ turns $s(t)$ into $-s(t)$, and the two minus signs cancel in $(-s(t))(-u) = s(t) u$. In $W' = W - \alpha u u^T W_S$, $u$ multiplies its transpose $u^T$, and $(-u)(-u)^T = u u^T$.
2. In part 1, the maximum `fp32` logit difference between the hook and the in-memory edit is `0.00774` (`0.0077`). Which numbers in `equivalence.json` and `try-ieee/equivalence.json` show this comes from floating-point kernel rounding and not from a mistake in the hook formula?
Answer Running layer 15's `down_proj` alone in `fp64` on CPU, where no GPU kernel rounds intermediate sums, gives `fp64_one_layer_max_abs_diff` of `3.33e-16` on outputs of scale `0.65`. Forcing full 24-bit IEEE `fp32` in the DeltaNet Triton kernels (`results/lab6/try-ieee/equivalence.json`) cuts `fp32_hook_vs_edit_max_abs_logit_diff` `5.8`-fold from `0.00774` to `0.00134`. Across the full `fp32` model, the edit itself shifts base logits by `11.4` (`fp32_base_vs_edit_max_abs_logit_diff`) with mean KL `0.0178`, while the hook and in-memory edit differ by `0.00774` (`1,500` times smaller) with mean KL `1.41e-8` (1.3 million times smaller).
3. On the four productive texts at layer 15, mean $s(t)$ over the last 36 tokens stays between `0.00` and `0.04`, versus `0.19`, `0.19`, `0.42`, and `0.25` on the four looping texts. How does $-\alpha \cdot s(t) \cdot u$ explain why the edit pushes much less on productive steps?
Answer Since $u$ has length `1` and $\alpha = 6$, the vector subtracted on token $t$ has length $|6 \cdot s(t)|$: at most about `0.24` (`6` times `0.04`) on the productive texts versus about `1.1` to `2.5` (`6` times `0.19` to `6` times `0.42`) on the looping texts. As the 64 neurons fire together in a loop, the push grows with $s(t)$ against $u$. Part 4 finds a smaller gap in the same direction on long real answers, so the push on good answers is small but not zero.
4. Why use teacher forcing in part 3 instead of letting the edited model generate freely from the prompt?
Answer Free generation diverges as soon as the edit changes one sampled token, leaving the base and edited models on different sentences. Teacher forcing feeds the same base-model loop into both models so we can compare $s(t)$, KL divergence, and loop-continuation odds at identical token positions.
5. Part 4 finds that `0 of 26` good answers have any 64-token stretch where $s(t)$ reaches loop level. Why does that not show that the edit leaves good answers unchanged?
Answer The push is proportional to $s(t)$, which on good answers is about one seventh of the in-loop mean (`0.091` to `0.116` at layer 19 versus `0.758`), not zero. That shifts next-token probabilities by a mean KL of `0.007` to `0.017` per token on good answers, about a tenth to a quarter of the in-loop KL (`0.069`). Across thousands of thinking tokens, those small shifts add up and can change which tokens get sampled; [Lab 7](lab7_side_effects.md) measures those side effects directly.
## Try this next Non-default flags write to `results/lab6/try-/` and leave the main result files and figures untouched. 1. **Inspect where $s(t)$ splits inside the 4 pairs.** Run this one-liner on the VM or locally to check the shared openings (`75`, `85`, `70` and `69` tokens) and post-split peaks in `results/lab6/s_on_pairs.json`: ```bash python3 -c 'import json; d=json.load(open("results/lab6/s_on_pairs.json")); k="per_token"; [print(f"L{L} pair {p}: shared diff={max(abs(a-b) for a,b in zip(d[L][k][2*p][:sh], d[L][k][2*p+1][:sh])):.4f}, loop peak={max(d[L][k][2*p+1]):.4f}") for L in ("15","19") for p,sh in enumerate((75,85,70,69))]' ``` This prints `0.0000` on the first three shared openings, `0.0018` (layer 15) and `0.0042` (layer 19) on the fourth from `bf16` rounding, and loop peaks of `0.4558, 0.4713, 0.5897, 0.4873` at layer 15 and `0.3620, 0.4300, 0.7376, 0.6322` at layer 19. 2. **Double the edit strength to $\alpha = 12$ with `--alpha-scale 2`.** The `--alpha-scale` flag scales $\alpha$ in both the hook and the in-memory edit (skipping the `bf16` file comparison because the file on disk was saved at $\alpha = 6$) and writes `results/lab6/try-alpha-scale-2/equivalence.json`: ```bash CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 1 --alpha-scale 2 ``` In `results/lab6/try-alpha-scale-2/equivalence.json`, the hook and in-memory edit still agree (`0.00916` max logit diff, mean KL `2.71e-8`). Against the base model, the max logit diff rises to `15.9` (`15.93`) and mean KL rises to `0.0745`, or `4.2` times the $\alpha = 6$ KL (`0.0178`). Doubling a small change roughly quadruples the KL divergence, because KL grows like the square of a small change. 3. **Force full IEEE `fp32` in the Triton DeltaNet kernels (`try-ieee`).** Triton defaults to 11-bit `TF32` for `fp32` matrix multiplies on A100 Tensor Cores. Pass `TRITON_F32_DEFAULT=ieee` with a fresh cache directory to recompile in 24-bit `fp32` and write `results/lab6/try-ieee/equivalence.json`: ```bash TRITON_F32_DEFAULT=ieee TRITON_CACHE_DIR=/nvme/wsp-lab/tmp/triton-ieee \ CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 1 --try-name ieee ``` The max `fp32` logit gap drops from `0.00774` to `0.00134` (`0.0013`), `5.8` times smaller, showing that `TF32` rounding in the DeltaNet mixers caused most of the `fp32` gap. 4. **Run part 3 on 10 loops (`--max-samples 10`) and redraw with less smoothing (`--smooth 16`).** Passing `--max-samples 10` runs part 3 in about a minute on 10 loops and writes `real_loops.json`, `trace_one_loop.json`, and both figures to `results/lab6/try-max-samples-10/`: ```bash CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3 --max-samples 10 ``` Across those 10 loops, edited `p_pick` inside the loop averages `0.984`, mean cycle odds at temperature 1 drop from `0.991` to `0.150`, and under the paper's sampler cycle odds drop from `0.999` to `0.511`. To redraw the 48-loop figure with a 16-token moving average instead of 64 (CPU only, writing to `results/lab6/try-smooth-16/`): ```bash python labs/lab6_steering.py --plot-only --smooth 16 ``` Comparing `results/lab6/try-smooth-16/lab6_real_loop.png` with `results/figures/lab6_real_loop.png` shows the token-by-token oscillations inside each cycle that the 64-token window smooths out. 5. **Read the good answer that came closest to loop level.** Sort `samples` in `results/lab6/finished_answers.json` by `max_s19_64_tokens`: ```bash python3 -c 'import json; d=json.load(open("results/lab6/finished_answers.json")); [print(s["id"], s["j"], round(s["max_s19_64_tokens"], 3)) for s in sorted(d["samples"], key=lambda s: -s["max_s19_64_tokens"])[:3]]' ``` This prints `gsm8k_train_6504 1 0.465`, `math_train_intermediate_algebra_605 0 0.353`, and `math_train_prealgebra_141 1 0.31`. Inspect the last 512 thinking tokens of `gsm8k_train_6504` answer `1` (tokens `6073` to `6585` from the start of the answer, on `sampler`, CPU only): ```bash CUDA_VISIBLE_DEVICES="" python labs/peek.py build/base --id gsm8k_train_6504 --answer 1 --tokens 6073 6585 ``` The header `answer 1: 6,796 tokens, finished` shows the generator's token count (`6,795` when re-tokenized from saved text). On this stretch, where $s_{19}$ averages `0.299` (this answer's hardest 64-token window of thinking reaches `0.465` at layer 19 and `0.316` at layer 15), the model circles through the repeated check and arithmetic described in part 4 before closing `` and giving the right answer `1600.00`. Does $s(t)$ respond only to fatal infinite loops, or to circling text even when the answer recovers?