Blame for code/reference/theory-guide.md

# Weight-space programming: from the basics to the actual research

The central idea is that a neural network is a numerical program, and a capable LLM may be able to inspect, build, or repair parts of that program by editing its weights directly.

Each lab in this project tests a section of this guide.

| Guide section | Lab that makes it hands-on |
|---|---|
| 1 to 5: weights, architecture, training vs editing, the language-model machinery | Background. Read before [Lab 0](../labs/lab0_toy.md). |
| 6: each MLP column writes a vector | [Lab 0](../labs/lab0_toy.md) |
| 7: what went wrong with Qwen | [Lab 2](../labs/lab2_reproduce.md) |
| 8.1 to 8.4: directions, dot products, removing a component | [Lab 0](../labs/lab0_toy.md) |
| 8.5: from removal to counteraction (the strength alpha) | [Lab 0](../labs/lab0_toy.md), then [Lab 4](../labs/lab4_dose.md) |
| 9, steps 1 to 3: contexts, the direction, the neurons, the columns | [Lab 3](../labs/lab3_rebuild.md), then [Lab 8](../labs/lab8_capstone.md) |
| 9, step 4: the matrix operation | [Lab 1](../labs/lab1_diff.md) |
| 10: why believe this repaired repetition | [Lab 2](../labs/lab2_reproduce.md) and the controls in [Lab 4](../labs/lab4_dose.md) |
| 12C: low rank versus LoRA | [Lab 1](../labs/lab1_diff.md) (the change is rank one) |
| 12D: weight editing versus activation steering | [Lab 6](../labs/lab6_steering.md) |
| 13.2: compare against a strong practical baseline | [Lab 5](../labs/lab5_decoding.md) |
| 13.3: keep development data separate | [Lab 8](../labs/lab8_capstone.md) |
| 13.4: measure more than accuracy | [Lab 7](../labs/lab7_side_effects.md) |
| 11, 12A, 12B, 12E, 12F, 13.1, 13.5, 13.6, 14 | No lab. Read them in the guide. |

We usually find weights by training. This study asks whether an LLM can instead read what weights compute, repair a broken network, or write a network from scratch. [rikiyat.github.io](https://rikiyat.github.io/blog/weight-space-programming/)

In the main language-model experiment, a small Qwen model often got stuck in repetitive reasoning, and a direct weight edit helped it finish many more answers. Understanding that repair requires five pieces:

1. How weights turn numbers into a computation.
2. How direct editing differs from training.
3. What attention heads and MLPs do inside a language model.
4. How vectors, projections, and small matrix updates change behavior.
5. What these experiments show and what stays open.

Every exercise has its worked solution right below "Check after trying." Cover the solution and work each one on paper first.

---

## 1. Start here: what is a neural-network weight?

A paddle moves left or right to stay under a ball. Its input has two coordinates:

$$
x=
\begin{pmatrix}
p\\
b
\end{pmatrix},
$$

where:

- $p$ is the paddle's horizontal position.
- $b$ is the ball's horizontal position.

With the paddle at $3$ and the ball at $8$:

$$
x=
\begin{pmatrix}
3\\
8
\end{pmatrix}.
$$

The ball is to the right, so we subtract:

$$
b-p.
$$

$$
8-3=5.
$$

A positive difference means the ball is to the right; a negative difference means it is to the left. Written with weights:

$$
(-1)p+(1)b.
$$

The weights are $-1$ and $1$, which set how much each input contributes. A weight is a numerical coefficient in a computation; when we know the rule we want, we can choose the weights ourselves instead of training them.

### Build a tiny network yourself

We want two hidden quantities that measure how far the ball is to the right or left:

$$
h_{\text{right}}=\max(0,b-p),
$$

$$
h_{\text{left}}=\max(0,p-b).
$$

The operation

$$
\operatorname{ReLU}(t)=\max(0,t)
$$

is pronounced "ree-loo." It keeps positive values and replaces negative values with zero.

**Try first:** What $2\times2$ matrix computes $b-p$ in its first row and $p-b$ in its second row?

*Hint: each row specifies one weighted sum of the input coordinates.*

### Check after trying

$$
W_1=
\begin{pmatrix}
-1&1\\
1&-1
\end{pmatrix}.
$$

Now multiply:

$$
W_1x
=
\begin{pmatrix}
-1&1\\
1&-1
\end{pmatrix}
\begin{pmatrix}
3\\
8
\end{pmatrix}
=
\begin{pmatrix}
(-1)(3)+(1)(8)\\
(1)(3)+(-1)(8)
\end{pmatrix}
=
\begin{pmatrix}
5\\
-5
\end{pmatrix}.
$$

Apply ReLU to each entry:

$$
h=
\operatorname{ReLU}(W_1x)
=
\begin{pmatrix}
5\\
0
\end{pmatrix}.
$$

Interpretation:

- Rightward distance: $5$.
- Leftward distance: $0$.

### Turn those hidden quantities into actions

Our three output scores will be ordered:

$$
\text{left},\quad \text{stay},\quad \text{right}.
$$

Define:

$$
z=
W_2h+c,
$$

with

$$
W_2=
\begin{pmatrix}
0&1\\
0&0\\
1&0
\end{pmatrix},
\qquad
c=
\begin{pmatrix}
0\\
1\\
0
\end{pmatrix}.
$$

The vector $c$ is a bias, an offset added after multiplication.

Calculate:

$$
z=
\begin{pmatrix}
0&1\\
0&0\\
1&0
\end{pmatrix}
\begin{pmatrix}
5\\
0
\end{pmatrix}
+
\begin{pmatrix}
0\\
1\\
0
\end{pmatrix}
=
\begin{pmatrix}
0\\
1\\
5
\end{pmatrix}.
$$

The highest score is "right." Because we knew the rule in advance, we could program the network by choosing its weights without a dataset.

### What did the bias accomplish?

If the ball and paddle have the same position:

$$
h=
\begin{pmatrix}
0\\
0
\end{pmatrix},
\qquad
z=
\begin{pmatrix}
0\\
1\\
0
\end{pmatrix}.
$$

The paddle stays still. "Stay" wins whenever the position difference is smaller than $1$; a tie needs a tie-breaking rule.

### Exercise: run it in the other direction

Now let:

$$
x=
\begin{pmatrix}
8\\
3
\end{pmatrix}.
$$

Compute the hidden vector and action scores.

### Check after trying

$$
W_1x=
\begin{pmatrix}
-5\\
5
\end{pmatrix},
\qquad
h=
\begin{pmatrix}
0\\
5
\end{pmatrix}.
$$

Therefore:

$$
z=
\begin{pmatrix}
5\\
1\\
0
\end{pmatrix}.
$$

The paddle moves left.

**What you should be able to explain:** the first layer computes relationships between inputs, and the second layer turns those relationships into action scores.

---

## 2. Architecture, weights, activations, and weight space

Keep these concepts separate:

| Term | Meaning | In our paddle network |
|---|---|---|
| **Architecture** | Which operations happen and how they connect | Matrix multiplication, ReLU, another matrix multiplication |
| **Weights** | Stored coefficients specifying the computation | Entries of $W_1$ and $W_2$ |
| **Biases** | Stored offsets | The $1$ that favors staying still |
| **Activations** | Intermediate values produced for a particular input | $h=(5,0)^T$ |
| **Logits** | Raw output scores before conversion to probabilities | $z=(0,1,5)^T$ |
| **Parameters** | Collective name for adjustable stored values | All weights and biases |

During inference, inputs and activations change from example to example while the stored parameters stay fixed. Collecting every parameter into one vector gives

$$
\theta,
$$

pronounced "theta." The network is the function:

$$
f_\theta(x).
$$

Read this as "the function applied to input $x$ with parameters $\theta$."

### What does "weight space" mean?

If a model has two weights:

$$
\theta=(w_1,w_2).
$$

Every setting is a point on a two-dimensional plane. A model with a million parameters lives in a million-dimensional space, with one coordinate per stored parameter. Editing weights moves the model to another point in that space.

Weight space differs from activation space. An activation vector holds the intermediate numbers produced on one input; a weight vector holds part of the stored program.

### A useful limitation of the "program" analogy

In ordinary code, a line might say:

```python
if ball_is_right:
    move_right()
```

Trained networks rarely separate into clean lines like this. One concept can spread across many neurons and matrices, and one neuron can take part in several behaviors. Our paddle network is easy to read because we designed each coordinate by hand; networks shaped by training are much harder to read.

---

## 3. How ordinary training differs from direct weight editing

Start with a one-weight model:

$$
\hat y=wx.
$$

Here:

- $x$: input.
- $w$: weight.
- $\hat y$, pronounced "y-hat": predicted output.

Suppose:

$$
x=2,\qquad y=6,\qquad w=1.
$$

The target output is $6$, and the model predicts:

$$
\hat y=1\cdot2=2.
$$

### Forward pass

A forward pass runs the input through the current weights to produce an output, here $2$.

### Loss

A loss function scores how far the prediction is from the target:

$$
L(w)=\frac12(wx-y)^2.
$$

At $w=1$:

$$
L(1)=\frac12(2-6)^2=8.
$$

### Backward pass

A backward pass computes how the loss changes when each weight changes (the derivative of the loss with respect to that weight):

$$
\frac{dL}{dw}=(wx-y)x.
$$

Substituting:

$$
\frac{dL}{dw}=(2-6)\cdot2=-8.
$$

Because the derivative is negative, nudging $w$ upward lowers the loss.

### Gradient-descent update

With learning rate $\eta=0.1$:

$$
w_{\text{new}}
=
w-\eta\frac{dL}{dw}
=
1-0.1(-8)
=
1.8.
$$

The new prediction is closer to $6$:

$$
1.8\cdot2=3.6.
$$

### Direct programming

When we know the exact relationship, we can solve directly:

$$
2w=6,
$$

so:

$$
w=3.
$$

**Try first:** Why does this example not prove that gradient descent is unnecessary for large networks?

### Check after trying

Here the specification gave the equation directly. In a large nonlinear network:

- We may not know the exact internal formula the task needs.
- Many weights interact across layers.
- Many parameter settings produce the same output on a given input.
- Fixing one behavior can break another.

Direct editing works when we can isolate the mechanism we want to change; training works when the mechanism has to be discovered from examples.

### Where weight-space programming fits

| Approach | How the target model is changed |
|---|---|
| Prompting | Change the input; keep its weights fixed |
| Ordinary fine-tuning | Compute gradients and update weights |
| An LLM writing a training script | The LLM designs the procedure; a training algorithm updates weights |
| Random or evolutionary search | Try parameter changes and retain promising ones |
| Weight-space programming in this work | The LLM reasons about the target and directly constructs parameter changes |

Keep two distinctions in mind:

$$
\boxed{
\text{No gradients does not imply no data, no feedback, or no computation.}
}
$$

Some experiments in this study are one-shot constructions, while others test and revise edits over many trials. Every experiment also involves two separate models:

- The editor: a frontier pretrained LLM.
- The target: the network being inspected or changed.

"No training" means no gradient updates on the target during the edit. Both the editor and any pretrained target were already trained beforehand.

---

## 4. How could anyone infer behavior from weights?

Return to a single weighted sum:

$$
z=-p+b.
$$

When we know what $p$ and $b$ mean, we can see that $z$ measures relative position. From the weight row alone,

$$
(-1,1),
$$

we cannot conclude "paddle controller." It could compute:

- Revenue minus cost.
- Target position minus robot position.
- Tomorrow's temperature minus today's temperature.

Weights get their meaning from the architecture and from the input and output coordinates they connect. In the game-identification experiments, the authors gave the editor the architecture and the input and output descriptions while withholding the game names, so the task was to infer the game from how the learned weights used those known inputs. [rikiyat.github.io](https://rikiyat.github.io/blog/weight-space-programming/)

### Why boundary layers can be revealing

Suppose the input is an image grid. First-layer weights that concentrate on the bottom row of the "agent" channel suggest an agent constrained near the bottom of the screen. Output weights that mainly separate left from right add another clue. Together, those clues can suggest a paddle game. That is meaningful inference, but it is a different achievement from explaining every internal computation.

### Exercise: the same numbers, a changed interface

A robot controller uses:

$$
z=x_1-x_2,
$$

where $x_1$ is the target position and $x_2$ is the robot position.

A new sensor interface reverses the order:

$$
x'=
\begin{pmatrix}
\text{robot position}\\
\text{target position}
\end{pmatrix}.
$$

What should the new weight row be?

### Check after trying

The original weight row was:

$$
(1,-1).
$$

The reversed interface needs:

$$
(-1,1).
$$

The behavior is unchanged, while the weights flip to match the coordinate order. This is why reading weights requires knowing what the coordinates mean.

### Another complication: different weights can implement the same function

Suppose:

$$
h_1=\operatorname{ReLU}(x_1),\qquad
h_2=\operatorname{ReLU}(x_2),
$$

and:

$$
y=2h_1+3h_2.
$$

Now swap the hidden neurons:

$$
h'_1=\operatorname{ReLU}(x_2),\qquad
h'_2=\operatorname{ReLU}(x_1).
$$

Use:

$$
y=3h'_1+2h'_2.
$$

Both versions compute the same function. For $x=(4,5)$, both produce:

$$
2(4)+3(5)=23.
$$

Because hidden neurons can be swapped without changing the output, "neuron 1" has no fixed meaning across networks; what matters is how a neuron connects to the layers around it.

---

## 5. The language-model machinery you need

### Tokens become vectors

A tokenizer splits text into tokens (words, subwords, or punctuation), and an embedding maps each token to a vector. At each token position, the model updates an internal vector:

$$
h.
$$

### The residual stream

As a useful approximation, each component adds its update to the running vector, and the existing vector continues forward:

$$
h_{\text{next}}=h+\text{update}.
$$

This running vector is the residual stream. Attention heads and MLPs read from it and add their outputs back into it, like a shared workspace made of numerical features rather than readable text.

### Attention moves information between positions

For a simplified attention head:

$$
q_i=W_Qh_i,
\qquad
k_j=W_Kh_j,
\qquad
v_j=W_Vh_j.
$$

Here:

- $i$ is the token position receiving information.
- $j$ is a candidate position to read from.
- $q_i$, the query, specifies what position $i$ looks for.
- $k_j$, the key, specifies what position $j$ offers.
- $v_j$, the value, holds the information that can be copied.

The matching score includes:

$$
q_i^Tk_j.
$$

The superscript $T$ means transpose, which turns a column vector into a row so the product is one number. Expanding the query and key shows why multiplying weight matrices together is informative:

$$
q_i^Tk_j
=
h_i^TW_Q^TW_Kh_j.
$$

The product

$$
W_Q^TW_K
$$

describes which token features match which, and the output path uses the product

$$
W_OW_V.
$$

In the transformer-circuits framework (shown here without positional encodings or normalization): [transformer-circuits.pub](https://transformer-circuits.pub/2021/framework/index.html)

- **QK:** selects where to read information from.
- **OV:** determines what is written back into the residual stream.

### What is an induction head?

When a sequence contains

> A B ... A

an induction circuit predicts

> B

by matching the current token to its earlier occurrence and copying what followed it.

**Try first:** In the pattern "red apple ... red," should a copying circuit copy the previous word "red"?

### Check after trying

No, the continuation is "apple." The circuit must link the earlier "red" to the token after it, which is why the researchers inspected products of matrices across heads rather than looking only for large single weights.

### MLPs compute and write features

An MLP (multilayer perceptron) transforms the representation at one position:

$$
a=\phi(W_{\text{up}}h),
$$

$$
m=W_{\text{down}}a,
$$

$$
h_{\text{next}}=h+m.
$$

Here $a$ holds the intermediate feature activations, and $W_{\text{down}}$ turns those activations into an update for the residual stream (real models may also use gating and normalization). For the Qwen target, the dimensions are:

$$
h\in\mathbb R^{1024},
\qquad
a\in\mathbb R^{3584},
\qquad
W_{\text{down}}\in\mathbb R^{1024\times3584}.
$$

The notation $\mathbb R^{1024}$ means "a vector of 1,024 real numbers." This Qwen model mixes Gated DeltaNet and gated-attention blocks, so standard attention does not describe every layer. [Hugging Face](https://huggingface.co/Qwen/Qwen3.5-0.8B)

---

## 6. The key insight: each MLP column writes a vector

Let the columns of $W_{\text{down}}$ be:

$$
w_1,w_2,\ldots,w_{3584}.
$$

Then matrix multiplication can be rewritten:

$$
\boxed{
W_{\text{down}}a
=
a_1w_1+a_2w_2+\cdots+a_{3584}w_{3584}.
}
$$

This is the most useful equation for understanding the edit. Each intermediate activation $a_j$ controls how much of column $w_j$ gets added:

- If $a_j=0$, that feature contributes nothing.
- If $a_j=2$, it contributes $2w_j$.
- Editing $w_j$ changes the vector that feature writes whenever it activates.

### Calculate it

Suppose:

$$
W=
\begin{pmatrix}
1&2\\
3&4
\end{pmatrix},
\qquad
a=
\begin{pmatrix}
2\\
0
\end{pmatrix}.
$$

**Try first:** Compute $Wa$ by thinking in columns.

### Check after trying

The columns are:

$$
w_1=
\begin{pmatrix}
1\\
3
\end{pmatrix},
\qquad
w_2=
\begin{pmatrix}
2\\
4
\end{pmatrix}.
$$

Therefore:

$$
Wa
=
2w_1+0w_2
=
\begin{pmatrix}
2\\
6
\end{pmatrix}.
$$

Even if the second column held huge numbers, it would contribute nothing on this input because its activation is zero. Looking only for the largest weights is not enough to find important behavior: a neuron's effect depends on both its stored vector and when it activates.

---

## 7. What went wrong with Qwen?

Language models generate text autoregressively:

1. Read the current context.
2. Predict the next token.
3. Append it.
4. Repeat.

During chain-of-thought reasoning, each forward pass is finite, yet the generated tokens can steer the context into repeating the same material until generation hits the token cap without a final answer.

### Why fixing repetition can improve "math accuracy"

Consider an invented illustration, not a decomposition measured in this study:

- A model finishes 10% of problems.
- Among the answers it finishes, 80% are correct.

The fraction of problems with a correct finished answer is:

$$
0.10\cdot0.80=0.08.
$$

If a repair raises completion to 60% while accuracy on finished answers stays 80%:

$$
0.60\cdot0.80=0.48.
$$

Overall accuracy rises from 8% to 48% without adding mathematical knowledge, because:

$$
P(\text{correct and finished})
=
P(\text{finished})
P(\text{correct}\mid\text{finished}).
$$

### The reported results

Under the authors' evaluation settings:

| Measurement | Original Qwen | Edited Qwen |
|---|---:|---:|
| MATH-500 accuracy | 7.9% | **48.1%** |
| GSM8K accuracy | 13.8% | **67.1%** |
| IFEval accuracy | 21.0% | **62.3%** |
| MATH-500 responses reaching the token cap | 93.0% | **33.1%** |

MATH-500 and GSM8K test mathematical problem solving; IFEval tests instruction compliance. The gain on IFEval fits a repair that reduces a shared generation failure across tasks. [Hugging Face](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming)

Reported scores average correctness across sampled responses (four responses per MATH-500 problem), rather than counting a problem solved if any of the four attempts succeeds. Evaluation used thinking mode, a 32,768-token limit, temperature $0.6$, top-p $0.95$, top-k $20$, and presence penalty $1.5$. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md)

### Exercise: distinguish two evaluation methods

A model answers one problem four times:

$$
\text{correct},\ \text{wrong},\ \text{wrong},\ \text{correct}.
$$

What is its average sample accuracy on that problem? What is its "at least one success among four" result?

### Check after trying

Average sample accuracy:

$$
\frac24=50\%.
$$

At least one success:

$$
100\%.
$$

Confusing these two metrics can make a model look much stronger than the experiment shows.

---

## 8. The linear algebra behind the repair

Changing a vector's contribution along one direction uses three operations: dot product, projection, and scaled subtraction.

### 8.1 A direction is a pattern across coordinates

Suppose:

$$
u=
\begin{pmatrix}
1\\
0
\end{pmatrix}.
$$

This points along the first coordinate.

Let:

$$
w=
\begin{pmatrix}
1\\
3
\end{pmatrix}.
$$

It contains:

- $1$ unit along $u$.
- $3$ units perpendicular to $u$.

In a real model, a direction typically spans many coordinates and need not align with a single neuron or coordinate axis.

### 8.2 The dot product measures the component

For a unit-length vector $u$, the scalar

$$
u^Tw
$$

gives the signed component of $w$ along $u$.

Here:

$$
u^Tw
=
\begin{pmatrix}
1&0
\end{pmatrix}
\begin{pmatrix}
1\\
3
\end{pmatrix}
=
1.
$$

Multiply that scalar by $u$:

$$
u(u^Tw)
=
\begin{pmatrix}
1\\
0
\end{pmatrix}.
$$

This is the vector component along $u$.

### 8.3 Remove that component

Subtract it:

$$
w'
=
w-u(u^Tw).
$$

Therefore:

$$
w'
=
\begin{pmatrix}
1\\
3
\end{pmatrix}
-
\begin{pmatrix}
1\\
0
\end{pmatrix}
=
\begin{pmatrix}
0\\
3
\end{pmatrix}.
$$

The component along $u$ is now zero while the perpendicular component remains. Geometrically, this is orthogonal projection onto the space perpendicular to $u$.

### 8.4 What if the direction is not unit length?

For any nonzero direction $d$, the denominator accounts for its length:

$$
\operatorname{proj}_d(w)
=
d\frac{d^Tw}{d^Td}.
$$

Or normalize $d$ to unit length first, where $\|d\|$ denotes its length:

$$
u=\frac{d}{\|d\|}.
$$

For:

$$
d=
\begin{pmatrix}
3\\
4
\end{pmatrix},
$$

we have:

$$
\|d\|=\sqrt{3^2+4^2}=5,
\qquad
u=
\begin{pmatrix}
3/5\\
4/5
\end{pmatrix}.
$$

### Exercise: remove a diagonal component

Let:

$$
d=
\begin{pmatrix}
1\\
1
\end{pmatrix},
\qquad
w=
\begin{pmatrix}
3\\
1
\end{pmatrix}.
$$

Find the part of $w$ along $d$, then subtract it.

*Hint: compute $d^Tw$ and $d^Td$ separately.*

### Check after trying

$$
d^Tw=3+1=4,
$$

$$
d^Td=1+1=2.
$$

Thus:

$$
\operatorname{proj}_d(w)
=
\begin{pmatrix}
1\\
1
\end{pmatrix}
\frac42
=
\begin{pmatrix}
2\\
2
\end{pmatrix}.
$$

Subtract:

$$
w'
=
\begin{pmatrix}
3\\
1
\end{pmatrix}
-
\begin{pmatrix}
2\\
2
\end{pmatrix}
=
\begin{pmatrix}
1\\
-1
\end{pmatrix}.
$$

Check perpendicularity:

$$
d^Tw'=1-1=0.
$$

Omitting the denominator and subtracting $(4,4)^T$ would overcorrect because $d$ is not normalized.

### 8.5 Generalize from removal to counteraction

Introduce an edit strength $\alpha$, pronounced "alpha":

$$
\boxed{
w'=w-\alpha u(u^Tw).
}
$$

For unit $u$:

$$
u^Tw'=(1-\alpha)u^Tw.
$$

Therefore:

| Strength | Effect on the component along $u$ |
|---|---|
| $\alpha=0$ | Leave it unchanged |
| $\alpha=0.5$ | Halve it |
| $\alpha=1$ | Remove it |
| $\alpha=2$ | Reverse it with equal magnitude |
| $\alpha=6$ | Reverse it with five times the original magnitude |

On the toy vector $w=(1,3)^T$ with the horizontal direction selected, raising the strength from $0$ to $1$ and then to $6$ changes only the horizontal component. The Qwen patch used strength $6$, reversing the selected component more strongly than projection removal.

---

## 9. What the successful Qwen edit actually did

### Step 1: compare productive and repetitive contexts

The editor built four matched pairs of math traces where one continuation made progress and the other repeated. It ran these fixed traces through the original model to record internal activations (teacher forcing, where the model reads a supplied sequence rather than sampling each next token).

At each relevant layer, it averaged the representations of the last 36 tokens of each trace and took a difference:

$$
d
=
\text{average representation during repetition}
-
\text{average representation during progress}.
$$

Then it normalized that difference:

$$
u=\frac{d}{\|d\|}.
$$

Each edited layer used its own direction vector rather than sharing one vector across the model. [Source: executed experiment code](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/tool_calls.jsonl)

**Interpretation:** $u$ is an activation pattern correlated with repetition. Correlation alone does not prove causation; weight interventions test whether changing it alters generation.

### Step 2: identify which features write that pattern

Recall:

$$
Wa=\sum_j a_jw_j.
$$

For feature $j$, two things matter:

1. Does it activate more strongly during repetition?
2. Does its output vector point along the repetition direction?

The selection score multiplied both terms:

$$
s_j
=
\underbrace{u^Tw_j}_{\text{writing along the direction}}
\underbrace{
\left(
\overline{a_j^{\text{repeat}}}
-
\overline{a_j^{\text{productive}}}
\right)
}_{\text{extra activation during repetition}}.
$$

The code selected the 64 highest-scoring neurons in each of two layers using the activation averages from the matched traces. [Source: neuron-selection code](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/tool_calls.jsonl)

### Exercise: choose the stronger candidate

Suppose:

| Neuron | Writing component $u^Tw_j$ | Extra activation during repetition |
|---|---:|---:|
| A | 3 | 2 |
| B | 100 | 0 |
| C | $-2$ | 4 |

Which receives the largest score?

### Check after trying

$$
s_A=3\cdot2=6,
$$

$$
s_B=100\cdot0=0,
$$

$$
s_C=(-2)\cdot4=-8.
$$

Neuron A gets the highest score. Neuron B aligns with the direction but shows no extra activation during repetition, and Neuron C pushes in the opposite direction.

### Step 3: edit selected output columns

The final patch changed 64 columns in each of:

```text
layers.15.mlp.down_proj.weight
layers.19.mlp.down_proj.weight
```

For each selected column:

$$
\boxed{
w_j'=w_j-6u(u^Tw_j).
}
$$

The successful run's [final report](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/work_notes/final_report.md) specifies that strength and those two matrices. Each column has 1,024 entries, so the edit touches:

$$
2\cdot64\cdot1024=131{,}072.
$$

Out of roughly 800 million parameters, that is:

$$
\frac{131{,}072}{800{,}000{,}000}\times100
\approx0.0164\%.
$$

That is the origin of the headline percentage. [Hugging Face](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming)

### Step 4: understand the matrix operation

Let $m$ be a mask: $1$ for a selected column, $0$ otherwise.

The implementation has the form:

```python
projection = u @ W
correction = np.outer(u, mask * projection)
W_edited = W - 6 * correction
```

An outer product multiplies a column vector by a row vector to build a matrix:

$$
\begin{pmatrix}
1\\
2
\end{pmatrix}
\begin{pmatrix}
3&4
\end{pmatrix}
=
\begin{pmatrix}
3&4\\
6&8
\end{pmatrix}.
$$

Every column of this matrix is a multiple of $(1,2)^T$. That makes the correction matrix rank at most one before rounding: it modifies only one output direction per edited matrix. Only the correction has this structure, not the entire model.

### A subtle but essential distinction

The edit does not subtract a fixed constant vector from every token. For an intermediate activation vector $a$:

$$
W'a-Wa
=
-6u
\sum_{j\in S}a_j(u^Tw_j),
$$

where $S$ is the selected neuron set. The push scales with how strongly those selected neurons fire on the current token. Preserving each selected column's perpendicular component in the edited matrix also does not freeze perpendicular activations in later layers, because downstream nonlinearities and future tokens respond to the updated stream.

---

## 10. Why believe this repaired repetition?

To check whether the edit worked by cutting repetition, the authors tested several controls:

- Sweeping edit strengths.
- Editing each layer on its own.
- Comparing against a random direction with matched edit norm.
- Evaluating on held-out practice problems after freezing the edit.

On a 1,152-problem held-out confirmation set, 579 problems flipped from wrong to right, and 576 of them had originally been truncated at the token cap. Aggregate accuracy on outputs that already finished before the edit did not change. [Source: successful run's final assessment](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/runs/p50c/final_answer.md)

That strongly supports the interpretation that the intervention rescued many answers by reducing a generation failure. It still does not prove that every form of repetition shares one mechanism, or that this direction represents "looping" in every context.

### The unsuccessful follow-up is informative

The authors also tried Gemma 3 270M, which rarely hit its token cap. Its MATH-500 score was 6.8% before editing and 7.0 to 7.7% across editing runs, showing no clear gain. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/gemma_followup/README.md)

That contrast separates two failure modes:

| Failure type | What may be needed |
|---|---|
| The model often knows enough but gets trapped in repetitive generation | A targeted execution repair may help |
| The model finishes but makes mathematical mistakes | It may need additional learning or a different intervention |

The Gemma follow-up does not show that direct editing can never add capabilities, only that these experiments did not demonstrate it.

---

## 11. What the other experiments demonstrate

Each experiment gave the editor different inputs and tools. Keep their conditions separate to avoid overgeneralizing:

| Experiment | What the editor could inspect or use | Reported outcome and its scope |
|---|---|---|
| **1. Identify a game** | Small policy weights, architecture, input/action descriptions; no tools | Correct game identification for four selected networks. [rikiyat.github.io](https://rikiyat.github.io/blog/weight-space-programming/) |
| **2. Identify generated shapes** | Small diffusion-network weights; no target forward passes | Correct coarse shape categories, but it described outlines when the models generated filled shapes. [Experiment 2](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp2_diffusion/README.md) |
| **3. Identify training text** | Middle transformer blocks and Python linear algebra; no target forward passes | Correct choice among Python, romanized Japanese, and English in two attempts per model. [Experiment 3](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp3_language/README.md) |
| **4. Repair a car controller** | Weights, scene, rules, baseline failures, one trajectory | A single edit succeeded in 1,024 tested episodes after the original policy failed. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp4_fix_car/README.md) |
| **5. Write game policies** | Game rules, source code, network specification; no trial gameplay | Written networks beat the reported 100-million-step PPO baselines in three games. [GitHub](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp5_write_from_scratch/README.md) |
| **6. Write a Go policy** | Rules, Python, and varying feedback budgets | Zero-feedback policies were useful; beating the tuned equal-sized PPO baselines required the feedback-assisted run. [Experiment 6](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp6_go_from_scratch/README.md) |
| **7. Identify sycophancy** | Initially weights alone; later attention patterns on supplied texts | Weights-only investigations failed to distinguish reliably; attention probes enabled three correct pair classifications. [Experiment 7](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp7_sycophancy/README.md) |

### Why the car example is interesting

To clear a ramp, the car had to back up before accelerating. Where the original policy kept repeating an unsuccessful approach, the editor wired a control pathway through selected units that used position and velocity to create a run-up before handing control back to the trained network. That edit changed about 3.5% of the actor's parameters. The result applies to the tested task distribution, without guaranteeing success in arbitrary driving situations. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp4_fix_car/README.md) This resembles our paddle example at a more sophisticated level: devise a controller, then encode it into a network.

### Why the game results need their full comparison

The reported scores were:

| Game | PPO: 100M steps | PPO: 500M steps | Directly written network |
|---|---:|---:|---:|
| Breakout | 15.8 | 25.3 | **517.4** |
| Seaquest | 5.6 | **88.9** | 53.1 |
| Asterix | 25.8 | **55.2** | 53.2 |

The written networks beat all three 100M-step baselines, but only on Breakout did they also beat the 500M-step baseline. [GitHub](https://github.com/RikiyaT/lm_understand-logs/blob/main/exp5_write_from_scratch/README.md) Because the editor received the game source code and knew the games, it could design heuristics and compile them into weights.

### What is the minimum diffusion background?

A diffusion generator repeatedly applies a learned denoiser to turn random noise into structured output. In the shape experiment, the editor identified the coarse shape category from the denoiser's weights, yet mistook filled shapes for outlines: spotting a structural signature in weights is weaker than reconstructing a generator's full output behavior.

### What is the minimum PPO background?

A policy maps a state to action probabilities. PPO (proximal policy optimization) is a reinforcement-learning algorithm that updates a policy from interaction rewards while limiting how sharply its action probabilities shift on each step. One environment step is a single interaction with the simulator, so it is not directly comparable to an LLM token, a full game, or a batched evaluation call.

### Why the sycophancy result matters

Sycophancy means following the user's suggested answer even when that suggestion is wrong. Seeing attention shift toward the suggestion shows where a head reads from, not whether that head causes the behavior: a head can attend strongly to a token while writing almost nothing through its value-output transformation. Causal interventions are stronger evidence than attention maps alone.

---

## 12. The distinctions you need for mastery

### A. Gradient-free versus feedback-free

The Qwen editor inspected practice problems, activations, and candidate outcomes. Its budget of "50 forward passes" meant 50 forward-tool requests, where one request could generate up to 512 multi-token responses or inspect up to 1,024 supplied texts. The winning run used 158 turns and 80 Python executions over 20.3 hours of elapsed time, including 4.7 hours of service unavailability. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md) This is gradient-free editing, yet it still involved a substantial experimental process.

### B. Sparse versus small

Sparse counts how many coordinates change; small measures how much the transformation or output changes. Changing one branch condition in a program can redirect nearly every run. Likewise, editing a tiny fraction of parameters can produce a large behavioral shift and does not guarantee that all other capabilities stay intact:

$$
\boxed{
\text{0.016\% of parameters changed}
\not\Rightarrow
\text{0.016\% of behavior changed}.
}
$$

### C. Low rank versus LoRA

Both the Qwen patch and LoRA write an update using low-rank factors:

$$
W'=W+BA.
$$

Ordinary LoRA learns those factors by gradient descent, whereas the Qwen patch constructed its rank-one update directly from activation contrasts. Low rank describes the shape of an update, not how it was found. [arxiv.org](https://arxiv.org/abs/2106.09685)

### D. Weight editing versus activation steering

Activation steering modifies temporary internal activations during an inference run; weight editing modifies the stored matrices that produce future activations. They can be related mathematically, while intervening at different places.

### E. Direct construction versus reverse engineering

Writing our paddle policy was straightforward because we chose the algorithm and coordinate meanings up front. Reading a trained network is harder because its computation can be distributed and redundant across many units. Building a network from scratch does not imply an ability to reverse engineer arbitrary trained networks.

### F. Novel combination versus entirely new ingredients

Each ingredient has prior work behind it:

- Mechanistic interpretability analyzes internal circuits.
- Model editing patches targeted behaviors.
- Tracr compiles human-readable programs into transformer weights, and it predates this work. [arxiv.org](https://arxiv.org/abs/2301.05062)
- Low-rank updates represent structured parameter deltas.

The interesting claim here is that a general-purpose LLM can perform several parts of this work itself, without specialized training as a weight generator.

---

## 13. How to evaluate the claims like a researcher

### 13.1 Separate three levels of understanding

Ask which level of claim the evidence supports:

1. **Identification:** "This network probably plays a paddle game."
2. **Prediction:** "On this input, it will choose right."
3. **Causal explanation:** "This pathway computes the position error, and changing it produces the predicted behavioral shift."

A correct label alone does not establish a causal explanation.

### 13.2 Compare against a strong practical baseline

Original and edited Qwen were tested under matched decoding settings, which supports a comparison under those conditions. However, the official model card recommends temperature $1.0$ for thinking text tasks, whereas the experiment used $0.6$, and notes that this small model can still enter thinking loops under recommended settings. [Hugging Face](https://huggingface.co/Qwen/Qwen3.5-0.8B)

To judge whether weight editing is the best practical fix, also compare against:

- Tuned decoding parameters.
- Thinking versus non-thinking modes.
- Repetition detection or other simple generation controls.
- Gradient-based fine-tuning under a similar resource budget.

This does not invalidate the measured edit effect. It determines whether editing is the best available intervention.

### 13.3 Keep development data separate from final evaluation

A sound workflow is:

```mermaid
flowchart TD
    A["Inspect failures and internals"] --> B["Propose structured edit"]
    B --> C["Test on development cases"]
    C -->|"Revise"| A
    C -->|"Select"| D["Freeze the edit"]
    D --> E["Evaluate on held-out tasks"]
```

Repeatedly checking the evaluation set while choosing layers, directions, or alpha turns that test set into development data. Final evaluation must run on held-out tasks after the edit is frozen.

### 13.4 Measure more than accuracy

For a repetition repair, track metrics that test the proposed mechanism alongside accuracy:

- Correct-answer rate and fraction hitting the token cap.
- Repetition rate, output length, and latency.
- Regressions on problems the unedited model solved.
- Performance on unrelated tasks.

### 13.5 Count the whole cost

A full cost comparison includes:

$$
\text{editor inference}
+
\text{target inference}
+
\text{analysis}
+
\text{development attempts}
+
\text{verification}.
$$

Skipping backpropagation does not automatically mean lower total cost.

### 13.6 Understand the current evidence boundary

The repository notes that earlier prompt trials are not all included in the released logs, making these runs exploratory demonstrations rather than a measure of success rate across models and tasks. [GitHub](https://github.com/RikiyaT/lm_understand-logs)

Our assessment is that the clearest near-term uses are:

- Targeted repairs of localized failure modes.
- Constructing small controllers from known rules.
- Proposing interpretable weight initializations.
- Automating parts of mechanistic investigation.

Installing broad new capabilities, auditing models reliably from weights alone, and replacing gradient-based training remain open research problems.

---

## 14. Use the demo as an active-learning experiment

Before opening the demo, solve its example yourself:

> Positive integers $x,y$ satisfy $x+y=20$ and $xy=91$. Find $x^2+y^2$.

**Try first:** Can you solve it without finding $x$ and $y$ separately?

*Hint: expand $(x+y)^2$.*

### Check after trying

$$
(x+y)^2=x^2+2xy+y^2.
$$

Rearrange:

$$
x^2+y^2=(x+y)^2-2xy.
$$

Substitute:

$$
x^2+y^2=20^2-2(91)=400-182=\boxed{218}.
$$

Now test both checkpoints in the [live comparison](https://huggingface.co/spaces/ML-Intern-lab/weight-space-programming-demo) and check:

1. Does each model reach a final answer?
2. Does it repeat a sentence or reasoning pattern?
3. Is the final answer correct?
4. Does a repeated run behave differently?

The edited model need not win on every individual prompt. The live app defaults to 4,096 tokens (the benchmark used 32,768) and omits the benchmark's presence penalty of $1.5$; the recorded-runs tab shows full-protocol examples. [Source: demo implementation](https://huggingface.co/spaces/ML-Intern-lab/weight-space-programming-demo/raw/main/app.py)

The model card also reports an independent full MATH-500 rerun of the edited checkpoint at 51.0%, paired with a 25-problem smoke test on the original model rather than a second full baseline evaluation. [Hugging Face](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming)

### How to read the paper's log repository

For the Qwen experiment, read in this order:

| File or folder | What it helps you understand |
|---|---|
| `exp8_math/README.md` | Rules, tools, evaluation protocol, and run mapping |
| `runs/p50c/final_answer.md` | Final choice and the main supporting evidence |
| `runs/p50c/work_notes/final_report.md` | Exact intervention and confirmation results |
| `runs/p50c/tool_calls.jsonl` | Executed computations behind the method |
| `runs/p50c/edit/` | The released parameter difference |
| `exp8_math/code/apply_edit.py` | Reconstructing and verifying the edited tensors |
| `exp8_math/eval/` | Benchmark scores and individual outcomes |

The reconstruction script verifies tensor hashes so you can confirm that the rebuilt weights match the released checkpoint. [raw.githubusercontent.com](https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/main/exp8_math/README.md) For understanding, the final report plus the edit script are more useful at first than the full conversation log.

---

## 15. Transfer exercise: predict collateral effects

Start with:

$$
W=
\begin{pmatrix}
1&0\\
0&1
\end{pmatrix}.
$$

This is the identity matrix:

$$
Wx=x.
$$

We want the input

$$
v=
\begin{pmatrix}
1\\
1
\end{pmatrix}
$$

to produce:

$$
\begin{pmatrix}
3\\
1
\end{pmatrix}
$$

instead of $(1,1)^T$.

One possible edit is:

$$
\Delta W=
\begin{pmatrix}
1&1\\
0&0
\end{pmatrix}.
$$

Then:

$$
W'=W+\Delta W
=
\begin{pmatrix}
2&1\\
0&1
\end{pmatrix}.
$$

Indeed:

$$
W'v=
\begin{pmatrix}
3\\
1
\end{pmatrix}.
$$

**Try first:** Does the edit also change the outputs for:

$$
x_A=
\begin{pmatrix}
1\\
-1
\end{pmatrix},
\qquad
x_B=
\begin{pmatrix}
1\\
0
\end{pmatrix}?
$$

### Check after trying

For $x_A$:

$$
\Delta Wx_A
=
\begin{pmatrix}
1&1\\
0&0
\end{pmatrix}
\begin{pmatrix}
1\\
-1
\end{pmatrix}
=
\begin{pmatrix}
0\\
0
\end{pmatrix}.
$$

Its output is unchanged.

For $x_B$:

$$
\Delta Wx_B
=
\begin{pmatrix}
1\\
0
\end{pmatrix}.
$$

Its output changes.

An edit designed for one target behavior also changes other inputs. That shared reach is the source of both useful generalization and possible collateral effects, and you need evaluation to determine which one you obtained.

---

# Compact mastery check

Test yourself on these questions before reading the answers.

## 1. Recall: explain weight-space programming to a beginner

What does the editor do, and what does "no training" mean in this study?

### Check after trying

> A neural network computes a function defined by its architecture and stored numerical weights. Weight-space programming asks a general-purpose LLM to inspect those weights and directly construct targeted updates, without running gradient-based training on the target model. Several experiments still used practice inputs, internal activation probes, and iterative testing, and both the editor and any pretrained targets were trained beforehand.

Your answer should separate the editor from the target and distinguish "no gradients" from "no feedback."

## 2. Intuition: why could a math repair improve instruction following?

### Check after trying

Repetitive looping can cut off generation on any task before the model emits its final answer. Suppressing that shared generation failure lets instruction-following responses finish too, without teaching each instruction separately. This is a mechanism-based prediction: improvement should transfer to other tasks affected by the same failure.

## 3. Calculation: compute the directional edit

Let:

$$
u=
\begin{pmatrix}
1\\
0
\end{pmatrix},
\qquad
w=
\begin{pmatrix}
2\\
5
\end{pmatrix}.
$$

Calculate:

$$
w'=w-\alpha u(u^Tw)
$$

for $\alpha=1$ and $\alpha=6$.

### Check after trying

First:

$$
u^Tw=2.
$$

For $\alpha=1$:

$$
w'=
\begin{pmatrix}
2\\
5
\end{pmatrix}
-
\begin{pmatrix}
2\\
0
\end{pmatrix}
=
\boxed{
\begin{pmatrix}
0\\
5
\end{pmatrix}
}.
$$

For $\alpha=6$:

$$
w'=
\begin{pmatrix}
2\\
5
\end{pmatrix}
-
6
\begin{pmatrix}
2\\
0
\end{pmatrix}
=
\boxed{
\begin{pmatrix}
-10\\
5
\end{pmatrix}
}.
$$

The original component $2$ becomes $-10$, which is $-5$ times the original. The perpendicular component remains $5$.

## 4. Discrimination: which claim is justified?

Someone says:

> "Only 0.016% of the weights changed, so all other capabilities must be preserved."

What is wrong with the argument?

### Check after trying

The percentage counts how many parameter coordinates changed, not how many behaviors depend on them. A small set of edited columns can sit on a pathway used across many tasks, so capability preservation must be tested on benchmarks rather than inferred from parameter count. A small edit count is useful evidence that the intervention is localized, but it is not a preservation guarantee.

## 5. Transfer: design a convincing experiment

You have a small model that often loops during tool-use planning. You propose editing a repetition-associated direction. What would make the result convincing?

### Check after trying

1. Measure the baseline failure under fixed decoding settings.
2. Use a development split to find candidate directions and choose the edit.
3. Check whether the proposed mechanism predicts the change in repetition rate.
4. Compare against controls, including a norm-matched random direction and simple decoding adjustments.
5. Freeze the edit before evaluating on held-out tasks.
6. Report task completion, accuracy, token usage, and regressions on previously solved cases.
7. Evaluate unrelated tasks for collateral side effects.
8. Include both editor and target compute when comparing total cost.

The explanation you should now be able to defend is:

> The promising result is that a frontier LLM can sometimes inspect a network's weights and activations, isolate a structured failure mode, construct a targeted matrix edit, and validate it. In the Qwen experiment, a rank-one update to two MLP matrices substantially reduced repetitive looping under the tested decoding protocol. Whether direct weight editing can install broad new capabilities or replace gradient training remains an open question.