weight-space-programming / labs /lab7_side_effects.html
thegovind's picture
Split the longest sentences and fit the install command on screen (from 338f312) (part 2)
09fbc15 verified
Raw History Blame Contribute Delete
155 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="color-scheme" content="only light">
<meta name="theme-color" content="#f5f5f5">
<title>Lab 7: What else did the edit change · Weight-space programming lab</title>
<meta name="description" content="Big thing: You will score the base model and 14 edited versions on 65,536 tokens of Wikipedia articles (WikiText-2) and on the agent&#x27;s 8 short math texts (4 productive, 4 looping). For each model…">
<link rel="icon" href="../static/favicon.svg" type="image/svg+xml">
<link rel="preload" href="../static/fonts/geist-latin-wght-normal.woff2" as="font" type="font/woff2" crossorigin>
<link rel="stylesheet" href="../static/site.css">
<link rel="stylesheet" href="../static/katex/katex.min.css">
<script>document.documentElement.classList.add("js")</script>
<script defer src="../static/site.js"></script>
</head>
<body data-copy="Copy" data-copied="Copied">
<a class="skip" href="#main">Skip to content</a>
<header class="top">
<div class="top-in">
<a class="brand" href="../index.html">Weight-space programming lab</a>
<nav class="top-links" aria-label="Site"><a href="../index.html">Home</a><a href="../quickstart.html">Quick start</a><a href="../index.html#the-nine-labs" aria-current="true">Labs</a><a href="../reference/theory-guide.html">Theory</a><a href="../results/summary.html">Results</a><a href="https://huggingface.co/spaces/thegovind/weight-space-programming/tree/main/code/" target="_blank" rel="noopener">Code</a></nav>
</div>
</header>
<div class="page">
<main id="main" class="doc">
<article class="prose">
<h1 id="lab-7-what-else-did-the-edit-change">Lab 7: What else did the edit change</h1>
<details class="tocm"><summary>On this page</summary><nav aria-label="On this page"><ul><li class="l2"><a href="#before-you-run-predict">Before you run: predict</a></li><li class="l2"><a href="#run-it">Run it</a></li><li class="l2"><a href="#what-you-should-see">What you should see</a></li><li class="l2"><a href="#why-it-works">Why it works</a></li><li class="l3"><a href="#1-testing-15-models-the-base-model-plus-14-edits-by-swapping-matrices-in-memory">1. Testing 15 models (the base model plus 14 edits) by swapping matrices in memory</a></li><li class="l3"><a href="#2-perplexity-and-kl-divergence-from-scratch">2. Perplexity and KL divergence from scratch</a></li><li class="l3"><a href="#3-reading-the-four-findings-in-the-table">3. Reading the four findings in the table</a></li><li class="l2"><a href="#check-yourself">Check yourself</a></li><li class="l2"><a href="#try-this-next">Try this next</a></li></ul></nav></details>
<p><strong>Big thing:</strong> You will score the base model and 14 edited versions on 65,536 tokens of Wikipedia articles (WikiText-2) and on the agent's 8 short math texts (4 productive, 4 looping). For each model, you measure how well it predicts the text (perplexity) and how far its next-token probabilities move from the base model's (KL divergence). The released edit (<code>p50c</code>) moves predictions 2.5 times more on looping texts than on productive ones (<code>0.0241</code> vs <code>0.0097</code>) and raises WikiText-2 perplexity by only <code>0.79%</code>.</p>
<p><strong>Why it matters:</strong> A loop fix is useless if it ruins ordinary language, since the model still has to read questions and write normal sentences. The 14 versions cover weaker and stronger edits (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span> from <code>-4</code> to <code>12</code>), single-layer edits, edits across all 3,584 neurons, and two random controls with the same weight-change size as the released edit. Comparing them shows what makes the edit act on loops and leave other text alone: the direction <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span>, the 64 chosen neurons, or both.</p>
<p><strong>Time and hardware:</strong> About 1.5 minutes on one A100 GPU (78 seconds in our run), plus a few seconds on the first run to download WikiText-2.</p>
<p><strong>Theory guide:</strong> Section 13.4 (measure more than accuracy).</p>
<h2 id="before-you-run-predict">Before you run: predict<a class="anchor" href="#before-you-run-predict" aria-hidden="true" tabindex="-1">#</a></h2>
<p>Write down a short guess for each question before running the script.</p>
<ol>
<li>The released edit (<code>p50c</code>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>) flips the loop push of 64 neurons in layers 15 and 19 and makes it 5 times larger. On plain Wikipedia text (no chat template, no thinking), will perplexity get much worse (say <code>+15%</code> or more), change by less than <code>1%</code>, or improve?</li>
<li>Two random controls test whether any change of the same size works. <code>randdir</code> edits the same 64 neurons along a random direction at right angles to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span>. <code>randcols</code> edits 64 random neurons along <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span>. Both match <code>p50c</code>'s weight-change size (matched before <code>bf16</code> rounding, so they agree closely but not exactly; the build script raised their <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span> to between about 20 and 31). On the agent's 8 texts, will they move predictions more on looping texts than on productive ones, or move both about equally?</li>
<li>Compare how far WikiText-2 predictions move when editing only layer 15 (<code>l15only</code>) versus both layers 15 and 19 (<code>p50c</code>). Must the two-layer edit move WikiText-2 predictions more than the single-layer edit?</li>
</ol>
<h2 id="run-it">Run it<a class="anchor" href="#run-it" aria-hidden="true" tabindex="-1">#</a></h2>
<p>Connect to <code>sampler</code> (or <code>trainer</code>). Then use <code>nvidia-smi</code> to pick a free GPU index and pass it in <code>CUDA_VISIBLE_DEVICES</code>.</p>
<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">ssh sampler
. /nvme/wsp-lab/code/scripts/env.sh &amp;&amp; cd /nvme/wsp-lab/code
nvidia-smi
CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py
</code></pre></div>
<p>The script loads the base model from <code>/nvme/wsp-lab/models/Qwen3.5-0.8B</code>, reads <code>edit_note.json</code> from each <code>/nvme/wsp-lab/models/qwen35-*</code> folder (built by <code>scripts/build_models.py</code>), and loads the <code>down_proj</code> matrices at the union of edited layers (layers 15 and 19 for the 14 standard edits, plus any extra edited layers from models you built in Labs 3 and 8). Pass <code>--models</code> to score a subset. It evaluates each model on <code>Salesforce/wikitext</code> (<code>wikitext-2-raw-v1</code>, <code>test</code> split) and <code>labs/data/astra_pairs.json</code>, then writes:</p>
<ul>
<li><code>results/lab7/side_effects.json</code> (perplexity, percent perplexity change, and KL divergences for <code>base</code> and each edited model)</li>
<li><code>results/figures/lab7_side_effects.png</code> (two-panel bar chart of WikiText-2 perplexity change and productive-versus-looping KL divergence)</li>
</ul>
<h2 id="what-you-should-see">What you should see<a class="anchor" href="#what-you-should-see" aria-hidden="true" tabindex="-1">#</a></h2>
<p>The script lists the 15 models in the default run (<code>base</code> plus 14 standard edits) and the swapped <code>down_proj</code> layers, then prints one row per model and the output path and runtime (yours will differ a little):</p>
<div class="code"><div class="code-head"><span class="code-lang">text</span></div><pre><code class="language-text">models: [&#x27;base&#x27;, &#x27;a-4&#x27;, &#x27;a1&#x27;, &#x27;a12&#x27;, &#x27;a2&#x27;, &#x27;a4&#x27;, &#x27;a8&#x27;, &#x27;all1&#x27;, &#x27;all2&#x27;, &#x27;l15only&#x27;, &#x27;l19only&#x27;, &#x27;p50c&#x27;, &#x27;randcols&#x27;, &#x27;randdir&#x27;, &#x27;rebuilt&#x27;] | down_proj swapped at layers [15, 19]
model ppl ppl +% KL wiki KL prod KL loop
base 20.009 0.00 0.00000 0.00000 0.00000
a-4 21.088 5.39 0.02058 0.00558 0.01466
a1 19.912 -0.49 0.00160 0.00095 0.00138
a12 22.878 14.34 0.16430 0.03661 0.10268
a2 19.871 -0.69 0.00445 0.00178 0.00346
a4 19.916 -0.47 0.01516 0.00485 0.01120
a8 20.675 3.33 0.06168 0.01645 0.04416
all1 19.925 -0.42 0.00248 0.00227 0.00289
all2 19.937 -0.36 0.00782 0.00634 0.00919
l15only 20.020 0.05 0.03476 0.00550 0.01938
l19only 20.340 1.65 0.01025 0.00464 0.00755
p50c 20.167 0.79 0.03327 0.00967 0.02405
randcols 19.989 -0.10 0.00346 0.00446 0.00383
randdir 20.070 0.30 0.00462 0.00479 0.00387
rebuilt 20.167 0.79 0.03326 0.00961 0.02384
wrote /nvme/wsp-lab/code/results/lab7 in 78 s
</code></pre></div>
<p>Grouping the 14 edited variants by what they test (rounded to 4 decimal places for KL, with base WikiText-2 perplexity <code>20.009</code>):</p>
<div class="table-wrap"><table>
<thead>
<tr>
<th>Model</th>
<th>What it is</th>
<th style="text-align:right">Perplexity change</th>
<th style="text-align:right">KL WikiText</th>
<th style="text-align:right">KL productive texts</th>
<th style="text-align:right">KL looping texts</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="cell"><span>p50c</span></code></td>
<td>Released edit (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>, 64 neurons, layers 15 and 19)</td>
<td style="text-align:right"><code class="cell"><span>+0.79%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0333</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0097</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0241</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>rebuilt</span></code></td>
<td>Lab 3's rebuild from the agent's 8 texts</td>
<td style="text-align:right"><code class="cell"><span>+0.79%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0333</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0096</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0238</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>a-4</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mo>−</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">\alpha = -4</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">−</span><span class="mord">4</span></span></span></span> (amplifies 5x toward loops)</td>
<td style="text-align:right"><code class="cell"><span>+5.39%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0206</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0056</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0147</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>a1</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\alpha = 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span> (erase loop component on 64 neurons)</td>
<td style="text-align:right"><code class="cell"><span>-0.49%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0016</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0010</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0014</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>a2</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">\alpha = 2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">2</span></span></span></span> (mirror loop component on 64 neurons)</td>
<td style="text-align:right"><code class="cell"><span>-0.69%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0044</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0018</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0035</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>a4</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">\alpha = 4</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">4</span></span></span></span> (flip and triple on 64 neurons)</td>
<td style="text-align:right"><code class="cell"><span>-0.47%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0152</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0048</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0112</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>a8</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">\alpha = 8</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">8</span></span></span></span> (flip and multiply by 7)</td>
<td style="text-align:right"><code class="cell"><span>+3.33%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0617</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0164</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0442</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>a12</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>12</mn></mrow><annotation encoding="application/x-tex">\alpha = 12</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">12</span></span></span></span> (flip and multiply by 11)</td>
<td style="text-align:right"><code class="cell"><span>+14.34%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.1643</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0366</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.1027</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>all1</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\alpha = 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span> on all 3,584 neurons</td>
<td style="text-align:right"><code class="cell"><span>-0.42%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0025</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0023</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0029</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>all2</span></code></td>
<td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">\alpha = 2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">2</span></span></span></span> on all 3,584 neurons</td>
<td style="text-align:right"><code class="cell"><span>-0.36%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0078</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0063</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0092</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>l15only</span></code></td>
<td>Released edit (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>, 64 neurons) on layer 15 only</td>
<td style="text-align:right"><code class="cell"><span>+0.05%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0348</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0055</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0194</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>l19only</span></code></td>
<td>Released edit (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>, 64 neurons) on layer 19 only</td>
<td style="text-align:right"><code class="cell"><span>+1.65%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0102</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0046</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0075</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>randcols</span></code></td>
<td>Control: direction <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span>, 64 random neurons, matched change size</td>
<td style="text-align:right"><code class="cell"><span>-0.10%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0035</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0045</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0038</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>randdir</span></code></td>
<td>Control: same 64 neurons, random orthogonal direction, matched size</td>
<td style="text-align:right"><code class="cell"><span>+0.30%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0046</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0048</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0039</span></code></td>
</tr>
</tbody>
</table></div>
<p>The script also saves <code>results/figures/lab7_side_effects.png</code>:</p>
<p><img src="../code/results/figures/lab7_side_effects.png" alt="Left: WikiText-2 perplexity change for each edit, highest for a12. Right: KL change on productive texts (green) and looping texts (red) for each edit, on a log scale." loading="lazy" decoding="async" width="1820" height="585" /></p>
<h2 id="why-it-works">Why it works<a class="anchor" href="#why-it-works" aria-hidden="true" tabindex="-1">#</a></h2>
<h3 id="1-testing-15-models-the-base-model-plus-14-edits-by-swapping-matrices-in-memory">1. Testing 15 models (the base model plus 14 edits) by swapping matrices in memory<a class="anchor" href="#1-testing-15-models-the-base-model-plus-14-edits-by-swapping-matrices-in-memory" aria-hidden="true" tabindex="-1">#</a></h3>
<p>A language model processes text one token (word piece) at a time through 24 blocks (or layers), each reading from and adding to the 1,024-number residual stream. Inside each block, the MLP (multi-layer perceptron) expands the stream to 3,584 neurons and projects them back through <code>down_proj</code> (a <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo separator="true">,</mo><mn>024</mn><mo>×</mo><mn>3</mn><mo separator="true">,</mo><mn>584</mn></mrow><annotation encoding="application/x-tex">1,024 \times 3,584</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">024</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">3</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">584</span></span></span></span> matrix stored in bf16, 16-bit floating point).</p>
<p>The base model has 488 weight tensors. In the default 15-model run (<code>base</code> plus 14 edits in <code>results/lab7/side_effects.json</code>), each edited model differs from the base model in at most two tensors: <code>layers.15.mlp.down_proj.weight</code> and <code>layers.19.mlp.down_proj.weight</code> (single-layer edits change only one). Instead of loading 15 separate 1.7 GB models from disk, <code>labs/lab7_side_effects.py</code> loads the base model onto the GPU once. It reads <code>edit_note.json</code> in each model folder, takes the union of edited layers (<code>LAYERS</code>), and swaps <code>down_proj</code> at those layers with <code>set_weights(w)</code> (printing <code>down_proj swapped at layers ...</code>). Any layer a model leaves untouched gets the base weights, which is how Lab 3's layer-21 experiment (<code>rebuilt-layers15-19-21</code>, layers 15, 19, and 21) is scored alongside 2-layer models.</p>
<p>For each chunk of text, the script uses teacher forcing: it feeds a fixed sequence of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo separator="true">,</mo><mn>024</mn></mrow><annotation encoding="application/x-tex">1,024</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">024</span></span></span></span> tokens in one forward pass and reads next-token predictions at positions <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn></mrow><annotation encoding="application/x-tex">0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0</span></span></span></span> through <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo separator="true">,</mo><mn>022</mn></mrow><annotation encoding="application/x-tex">1,022</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">022</span></span></span></span> (predicting tokens <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn></mrow><annotation encoding="application/x-tex">1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span> through <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo separator="true">,</mo><mn>023</mn></mrow><annotation encoding="application/x-tex">1,023</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">023</span></span></span></span>). On WikiText-2 test, 64 chunks of 1,024 tokens (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>65</mn><mo separator="true">,</mo><mn>536</mn></mrow><annotation encoding="application/x-tex">65,536</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">65</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">536</span></span></span></span> tokens total) yield 65,472 predictions, since the first token of each chunk has no prior context.</p>
<figure class="mermaid-svg" role="img" aria-label="Diagram: 1. Testing 15 models (the base model plus 14 edits) by swapping matrices in memory" style="--mw:415px"><svg id="m-e2898c1b5d83" width="100%" xmlns="http://www.w3.org/2000/svg" class="flowchart" style="max-width: 553.5px;" viewBox="0 0 553.5 526" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#m-e2898c1b5d83{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#m-e2898c1b5d83 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#m-e2898c1b5d83 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#m-e2898c1b5d83 .error-icon{fill:#552222;}#m-e2898c1b5d83 .error-text{fill:#552222;stroke:#552222;}#m-e2898c1b5d83 .edge-thickness-normal{stroke-width:1px;}#m-e2898c1b5d83 .edge-thickness-thick{stroke-width:3.5px;}#m-e2898c1b5d83 .edge-pattern-solid{stroke-dasharray:0;}#m-e2898c1b5d83 .edge-thickness-invisible{stroke-width:0;fill:none;}#m-e2898c1b5d83 .edge-pattern-dashed{stroke-dasharray:3;}#m-e2898c1b5d83 .edge-pattern-dotted{stroke-dasharray:2;}#m-e2898c1b5d83 .marker{fill:#666;stroke:#666;}#m-e2898c1b5d83 .marker.cross{stroke:#666;}#m-e2898c1b5d83 svg{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;}#m-e2898c1b5d83 p{margin:0;}#m-e2898c1b5d83 .label{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;color:#000000;}#m-e2898c1b5d83 .cluster-label text{fill:#333;}#m-e2898c1b5d83 .cluster-label span{color:#333;}#m-e2898c1b5d83 .cluster-label span p{background-color:transparent;}#m-e2898c1b5d83 .label text,#m-e2898c1b5d83 span{fill:#000000;color:#000000;}#m-e2898c1b5d83 .node rect,#m-e2898c1b5d83 .node circle,#m-e2898c1b5d83 .node ellipse,#m-e2898c1b5d83 .node polygon,#m-e2898c1b5d83 .node path{fill:#eee;stroke:#999;stroke-width:1px;}#m-e2898c1b5d83 .rough-node .label text,#m-e2898c1b5d83 .node .label text,#m-e2898c1b5d83 .image-shape .label,#m-e2898c1b5d83 .icon-shape .label{text-anchor:middle;}#m-e2898c1b5d83 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#m-e2898c1b5d83 .rough-node .label,#m-e2898c1b5d83 .node .label,#m-e2898c1b5d83 .image-shape .label,#m-e2898c1b5d83 .icon-shape .label{text-align:center;}#m-e2898c1b5d83 .node.clickable{cursor:pointer;}#m-e2898c1b5d83 .root .anchor path{fill:#666!important;stroke-width:0;stroke:#666;}#m-e2898c1b5d83 .arrowheadPath{fill:#333333;}#m-e2898c1b5d83 .edgePaths .path{stroke:#666;stroke-width:1px;}#m-e2898c1b5d83 .flowchart-link{stroke:#666;fill:none;}#m-e2898c1b5d83 .edgeLabel{background-color:white;text-align:center;}#m-e2898c1b5d83 .edgeLabel p{background-color:white;}#m-e2898c1b5d83 .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#m-e2898c1b5d83 .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#m-e2898c1b5d83 .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#m-e2898c1b5d83 .cluster text{fill:#333;}#m-e2898c1b5d83 .cluster span{color:#333;}#m-e2898c1b5d83 .node .collapsed-indicator{fill:#707070;stroke:none;opacity:0.6;}#m-e2898c1b5d83 .node .collapsed-separator{stroke:#707070;stroke-width:0.75px;}#m-e2898c1b5d83 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#m-e2898c1b5d83 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#m-e2898c1b5d83 rect.text{fill:none;stroke-width:0;}#m-e2898c1b5d83 .icon-shape,#m-e2898c1b5d83 .image-shape{background-color:white;text-align:center;}#m-e2898c1b5d83 .icon-shape p,#m-e2898c1b5d83 .image-shape p{background-color:white;padding:2px;}#m-e2898c1b5d83 .icon-shape .label rect,#m-e2898c1b5d83 .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#m-e2898c1b5d83 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#m-e2898c1b5d83 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#m-e2898c1b5d83 .node .neo-node{stroke:#999;}#m-e2898c1b5d83 [data-look="neo"].node rect,#m-e2898c1b5d83 [data-look="neo"].cluster rect,#m-e2898c1b5d83 [data-look="neo"].node polygon{stroke:url(#m-e2898c1b5d83-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-e2898c1b5d83 [data-look="neo"].swimlane.cluster rect{filter:none;}#m-e2898c1b5d83 [data-look="neo"].node path{stroke:url(#m-e2898c1b5d83-gradient);stroke-width:1px;}#m-e2898c1b5d83 [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-e2898c1b5d83 [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#m-e2898c1b5d83 [data-look="neo"].node circle{stroke:url(#m-e2898c1b5d83-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-e2898c1b5d83 [data-look="neo"].node circle .state-start{fill:#000000;}#m-e2898c1b5d83 [data-look="neo"].icon-shape .icon{fill:url(#m-e2898c1b5d83-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-e2898c1b5d83 [data-look="neo"].icon-shape .icon-neo path{stroke:url(#m-e2898c1b5d83-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-e2898c1b5d83 :root{--mermaid-font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;}</style><g><marker id="m-e2898c1b5d83_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-e2898c1b5d83_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-e2898c1b5d83_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></path></marker><marker id="m-e2898c1b5d83_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></polygon></marker><marker id="m-e2898c1b5d83_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-e2898c1b5d83_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-e2898c1b5d83_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-e2898c1b5d83_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-e2898c1b5d83_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-e2898c1b5d83_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-e2898c1b5d83_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5;"></path></marker><marker id="m-e2898c1b5d83_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5; stroke-dasharray: 1, 0;"></path></marker><g class="root"><g class="clusters"></g><g class="edgePaths"><path d="M135,214L135,218.167C135,222.333,135,230.667,142.292,238.688C149.583,246.71,164.167,254.42,171.458,258.275L178.75,262.13" id="m-e2898c1b5d83-L_B_C_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_B_C_0" data-points="W3sieCI6MTM1LCJ5IjoyMTR9LHsieCI6MTM1LCJ5IjoyMzl9LHsieCI6MTgyLjI4NjE4NDIxMDUyNjMsInkiOjI2NH1d" data-look="classic" marker-end="url(#m-e2898c1b5d83_flowchart-v2-pointEnd)"></path><path d="M422.5,86L422.5,90.167C422.5,94.333,422.5,102.667,422.5,110.333C422.5,118,422.5,125,422.5,128.5L422.5,132" id="m-e2898c1b5d83-L_V_S_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_V_S_0" data-points="W3sieCI6NDIyLjUsInkiOjg2fSx7IngiOjQyMi41LCJ5IjoxMTF9LHsieCI6NDIyLjUsInkiOjEzNn1d" data-look="classic" marker-end="url(#m-e2898c1b5d83_flowchart-v2-pointEnd)"></path><path d="M422.5,214L422.5,218.167C422.5,222.333,422.5,230.667,415.208,238.688C407.917,246.71,393.333,254.42,386.042,258.275L378.75,262.13" id="m-e2898c1b5d83-L_S_C_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_S_C_0" data-points="W3sieCI6NDIyLjUsInkiOjIxNH0seyJ4Ijo0MjIuNSwieSI6MjM5fSx7IngiOjM3NS4yMTM4MTU3ODk0NzM3LCJ5IjoyNjR9XQ==" data-look="classic" marker-end="url(#m-e2898c1b5d83_flowchart-v2-pointEnd)"></path><path d="M278.75,366L278.75,370.167C278.75,374.333,278.75,382.667,278.75,390.333C278.75,398,278.75,405,278.75,408.5L278.75,412" id="m-e2898c1b5d83-L_C_M_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_C_M_0" data-points="W3sieCI6Mjc4Ljc1LCJ5IjozNjZ9LHsieCI6Mjc4Ljc1LCJ5IjozOTF9LHsieCI6Mjc4Ljc1LCJ5Ijo0MTZ9XQ==" data-look="classic" marker-end="url(#m-e2898c1b5d83_flowchart-v2-pointEnd)"></path></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_B_C_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_V_S_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_S_C_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_C_M_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g></g><g class="nodes"><g class="node default" id="m-e2898c1b5d83-flowchart-B-0" data-look="classic" transform="translate(135, 175)"><rect class="basic label-container" style="" x="-127" y="-39" width="254" height="78"></rect><g class="label" style="" transform="translate(-97, -24)"><rect></rect><foreignObject width="194" height="48"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel"><p>Load base model once<br>save base W_15 and W_19</p></span></div></foreignObject></g></g><g class="node default" id="m-e2898c1b5d83-flowchart-C-1" data-look="classic" transform="translate(278.75, 315)"><rect class="basic label-container" style="" x="-130" y="-51" width="260" height="102"></rect><g class="label" style="" transform="translate(-100, -36)"><rect></rect><foreignObject width="200" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Feed 64 x 1,024 WikiText tokens<br>and the 8 math texts</p></span></div></foreignObject></g></g><g class="node default" id="m-e2898c1b5d83-flowchart-V-2" data-look="classic" transform="translate(422.5, 47)"><rect class="basic label-container" style="" x="-123" y="-39" width="246" height="78"></rect><g class="label" style="" transform="translate(-93, -24)"><rect></rect><foreignObject width="186" height="48"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel"><p>14 variant folders<br>load only W_15 and W_19</p></span></div></foreignObject></g></g><g class="node default" id="m-e2898c1b5d83-flowchart-S-3" data-look="classic" transform="translate(422.5, 175)"><rect class="basic label-container" style="" x="-110.5" y="-39" width="221" height="78"></rect><g class="label" style="" transform="translate(-80.5, -24)"><rect></rect><foreignObject width="161" height="48"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel"><p>Swap W_15 and W_19<br>in GPU memory</p></span></div></foreignObject></g></g><g class="node default" id="m-e2898c1b5d83-flowchart-M-7" data-look="classic" transform="translate(278.75, 467)"><rect class="basic label-container" style="" x="-130" y="-51" width="260" height="102"></rect><g class="label" style="" transform="translate(-100, -36)"><rect></rect><foreignObject width="200" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Compare base and edited log-probs<br>compute perplexity and KL</p></span></div></foreignObject></g></g></g></g></g><defs><filter id="m-e2898c1b5d83-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><defs><filter id="m-e2898c1b5d83-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><linearGradient id="m-e2898c1b5d83-gradient" gradientUnits="objectBoundingBox" x1="0%" y1="0%" x2="100%" y2="0%"><stop offset="0%" stop-color="hsl(0, 0%, 83.3333333333%)" stop-opacity="1"></stop><stop offset="100%" stop-color="hsl(0, 0%, 88.9215686275%)" stop-opacity="1"></stop></linearGradient></svg></figure>
<h3 id="2-perplexity-and-kl-divergence-from-scratch">2. Perplexity and KL divergence from scratch<a class="anchor" href="#2-perplexity-and-kl-divergence-from-scratch" aria-hidden="true" tabindex="-1">#</a></h3>
<p>At each token position, the model outputs logits (one raw score per vocabulary token), and softmax converts those scores into probabilities that sum to <code>1</code>. We compute two metrics from those probabilities.</p>
<p>Imagine a two-token vocabulary, <code>&quot;38&quot;</code> and <code>&quot;But&quot;</code>, at a step where <code>&quot;38&quot;</code> is the true next token:</p>
<ul>
<li>The base model predicts <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mtext>base</mtext></msub><mo stretchy="false">(</mo><mtext>&quot;38&quot;</mtext><mo stretchy="false">)</mo><mo>=</mo><mn>0.80</mn></mrow><annotation encoding="application/x-tex">p_{\text{base}}(\text{&quot;38&quot;}) = 0.80</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">base</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord text"><span class="mord">&quot;38&quot;</span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.80</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mtext>base</mtext></msub><mo stretchy="false">(</mo><mtext>&quot;But&quot;</mtext><mo stretchy="false">)</mo><mo>=</mo><mn>0.20</mn></mrow><annotation encoding="application/x-tex">p_{\text{base}}(\text{&quot;But&quot;}) = 0.20</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">base</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord text"><span class="mord">&quot;But&quot;</span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.20</span></span></span></span>.</li>
<li>An edited model predicts <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mtext>edit</mtext></msub><mo stretchy="false">(</mo><mtext>&quot;38&quot;</mtext><mo stretchy="false">)</mo><mo>=</mo><mn>0.50</mn></mrow><annotation encoding="application/x-tex">p_{\text{edit}}(\text{&quot;38&quot;}) = 0.50</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">edit</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord text"><span class="mord">&quot;38&quot;</span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.50</span></span></span></span> and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>p</mi><mtext>edit</mtext></msub><mo stretchy="false">(</mo><mtext>&quot;But&quot;</mtext><mo stretchy="false">)</mo><mo>=</mo><mn>0.50</mn></mrow><annotation encoding="application/x-tex">p_{\text{edit}}(\text{&quot;But&quot;}) = 0.50</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">edit</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord text"><span class="mord">&quot;But&quot;</span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.50</span></span></span></span>.</li>
</ul>
<p>Perplexity measures how well a model predicts the true next token in ground-truth text. First take the negative natural log of the true token's probability (<code>&quot;38&quot;</code>), called the negative log-likelihood (NLL):</p>
<ul>
<li>Base model: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>NLL</mtext><mo>=</mo><mo>−</mo><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mn>0.80</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0.2231</mn></mrow><annotation encoding="application/x-tex">\text{NLL} = -\ln(0.80) = 0.2231</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord text"><span class="mord">NLL</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">−</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord">0.80</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.2231</span></span></span></span>.</li>
<li>Edited model: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>NLL</mtext><mo>=</mo><mo>−</mo><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mn>0.50</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0.6931</mn></mrow><annotation encoding="application/x-tex">\text{NLL} = -\ln(0.50) = 0.6931</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord text"><span class="mord">NLL</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">−</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord">0.50</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.6931</span></span></span></span>.</li>
</ul>
<p>Average the NLL across all tokens and raise <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi></mrow><annotation encoding="application/x-tex">e</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">e</span></span></span></span> to that average: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>perplexity</mtext><mo>=</mo><msup><mi>e</mi><mtext>mean NLL</mtext></msup></mrow><annotation encoding="application/x-tex">\text{perplexity} = e^{\text{mean NLL}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord text"><span class="mord">perplexity</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.8413em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8413em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">mean NLL</span></span></span></span></span></span></span></span></span></span></span></span></span>. On this single token, perplexity is <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>0.2231</mn></msup><mo>=</mo><mn>1</mn><mi mathvariant="normal">/</mi><mn>0.80</mn><mo>=</mo><mn>1.25</mn></mrow><annotation encoding="application/x-tex">e^{0.2231} = 1 / 0.80 = 1.25</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">0.2231</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">1/0.80</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.25</span></span></span></span> for the base model and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>0.6931</mn></msup><mo>=</mo><mn>1</mn><mi mathvariant="normal">/</mi><mn>0.50</mn><mo>=</mo><mn>2.00</mn></mrow><annotation encoding="application/x-tex">e^{0.6931} = 1 / 0.50 = 2.00</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">0.6931</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">1/0.50</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">2.00</span></span></span></span> for the edited model. A model as uncertain as a fair <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.0715em;">K</span></span></span></span>-sided die at every token (assigning probability <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>K</mi></mrow><annotation encoding="application/x-tex">1/K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">1/</span><span class="mord mathnormal" style="margin-right:0.0715em;">K</span></span></span></span> to the true token) has perplexity <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.0715em;">K</span></span></span></span>. Lower perplexity means better prediction. On the 65,536 WikiText-2 tokens, the base model's perplexity is <code>20.009</code>.</p>
<p>KL divergence, written <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>KL</mtext><mo stretchy="false">(</mo><mtext>base</mtext><mo>∥</mo><mtext>edited</mtext><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\text{KL}(\text{base} \parallel \text{edited})</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord text"><span class="mord">KL</span></span><span class="mopen">(</span><span class="mord text"><span class="mord">base</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">∥</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord text"><span class="mord">edited</span></span><span class="mclose">)</span></span></span></span>, measures how far the edited model's full probability distribution moved from the base model's, regardless of which token came next. At one position, it averages <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><msub><mi>p</mi><mtext>base</mtext></msub><mo stretchy="false">(</mo><mi>v</mi><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><msub><mi>p</mi><mtext>edit</mtext></msub><mo stretchy="false">(</mo><mi>v</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\ln(p_{\text{base}}(v) / p_{\text{edit}}(v))</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">base</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span><span class="mclose">)</span><span class="mord">/</span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">edit</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span><span class="mclose">))</span></span></span></span> weighted by the base probabilities:</p>
<div class="math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>KL</mtext><mo stretchy="false">(</mo><msub><mi>p</mi><mtext>base</mtext></msub><mo>∥</mo><msub><mi>p</mi><mtext>edit</mtext></msub><mo stretchy="false">)</mo><mo>=</mo><munder><mo>∑</mo><mi>v</mi></munder><msub><mi>p</mi><mtext>base</mtext></msub><mo stretchy="false">(</mo><mi>v</mi><mo stretchy="false">)</mo><mrow><mo fence="true">(</mo><mi>ln</mi><mo>⁡</mo><msub><mi>p</mi><mtext>base</mtext></msub><mo stretchy="false">(</mo><mi>v</mi><mo stretchy="false">)</mo><mo>−</mo><mi>ln</mi><mo>⁡</mo><msub><mi>p</mi><mtext>edit</mtext></msub><mo stretchy="false">(</mo><mi>v</mi><mo stretchy="false">)</mo><mo fence="true">)</mo></mrow></mrow><annotation encoding="application/x-tex">\text{KL}(p_{\text{base}} \parallel p_{\text{edit}}) = \sum_{v} p_{\text{base}}(v) \left(\ln p_{\text{base}}(v) - \ln p_{\text{edit}}(v)\right)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord text"><span class="mord">KL</span></span><span class="mopen">(</span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">base</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">∥</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">edit</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:2.3em;vertical-align:-1.25em;"></span><span class="mop op-limits"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.05em;"><span style="top:-1.9em;margin-left:0em;"><span class="pstrut" style="height:3.05em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span style="top:-3.05em;"><span class="pstrut" style="height:3.05em;"></span><span><span class="mop op-symbol large-op">∑</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:1.25em;"><span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">base</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="minner"><span class="mopen delimcenter" style="top:0em;">(</span><span class="mop">ln</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">base</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mop">ln</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">edit</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mopen">(</span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span><span class="mclose">)</span><span class="mclose delimcenter" style="top:0em;">)</span></span></span></span></span></span></div>
<p>In our two-token example:</p>
<div class="math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>KL</mtext><mo>=</mo><mn>0.80</mn><mi>ln</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><mfrac><mn>0.80</mn><mn>0.50</mn></mfrac><mo fence="true">)</mo></mrow><mo>+</mo><mn>0.20</mn><mi>ln</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><mfrac><mn>0.20</mn><mn>0.50</mn></mfrac><mo fence="true">)</mo></mrow><mo>=</mo><mn>0.80</mn><mo stretchy="false">(</mo><mn>0.4700</mn><mo stretchy="false">)</mo><mo>+</mo><mn>0.20</mn><mo stretchy="false">(</mo><mo>−</mo><mn>0.9163</mn><mo stretchy="false">)</mo><mo>=</mo><mn>0.3760</mn><mo>−</mo><mn>0.1833</mn><mo>=</mo><mn>0.1927</mn></mrow><annotation encoding="application/x-tex">\text{KL} = 0.80 \ln\left(\frac{0.80}{0.50}\right) + 0.20 \ln\left(\frac{0.20}{0.50}\right) = 0.80(0.4700) + 0.20(-0.9163) = 0.3760 - 0.1833 = 0.1927</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord text"><span class="mord">KL</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:2.4em;vertical-align:-0.95em;"></span><span class="mord">0.80</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mop">ln</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="minner"><span class="mopen delimcenter" style="top:0em;"><span class="delimsizing size3">(</span></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.3214em;"><span style="top:-2.314em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord">0.50</span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord">0.80</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.686em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mclose delimcenter" style="top:0em;"><span class="delimsizing size3">)</span></span></span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:2.4em;vertical-align:-0.95em;"></span><span class="mord">0.20</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mop">ln</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="minner"><span class="mopen delimcenter" style="top:0em;"><span class="delimsizing size3">(</span></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.3214em;"><span style="top:-2.314em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord">0.50</span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord">0.20</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.686em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span><span class="mclose delimcenter" style="top:0em;"><span class="delimsizing size3">)</span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">0.80</span><span class="mopen">(</span><span class="mord">0.4700</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">+</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">0.20</span><span class="mopen">(</span><span class="mord">−</span><span class="mord">0.9163</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">0.3760</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.1833</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.1927</span></span></span></span></span></div>
<p>Identical distributions have KL divergence <code>0</code>. Even when perplexity barely changes because gains on some tokens offset losses on others, KL divergence is strictly positive whenever predictions move.</p>
<h3 id="3-reading-the-four-findings-in-the-table">3. Reading the four findings in the table<a class="anchor" href="#3-reading-the-four-findings-in-the-table" aria-hidden="true" tabindex="-1">#</a></h3>
<p>First, compare selectivity on looping versus productive text. Each variant in <code>configs/variants.json</code> modifies <code>down_proj</code> along a direction <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span> (a 1,024-number unit vector) with edit strength <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span> (scaling the chosen neurons' push along <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span> by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>−</mo><mi>α</mi></mrow><annotation encoding="application/x-tex">1 - \alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span>). In the right panel of <code>lab7_side_effects.png</code>, compare the red bars (<code>kl_looping_texts</code>) with the green bars (<code>kl_productive_texts</code>). The released edit <code>p50c</code> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>) moves next-token predictions about 2.5 times more on looping texts (<code>0.0241</code>) than on productive ones (<code>0.0097</code>). Lab 3's rebuild (<code>rebuilt</code>) matches <code>p50c</code> closely (<code>0.0238</code> vs <code>0.0096</code>, <code>+0.79%</code> perplexity).</p>
<p>Now compare <code>p50c</code> with the random controls <code>randdir</code> and <code>randcols</code>. Both set <code>norm_match: true</code>, rescaling <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span> so the Frobenius size of the weight change <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∥</mi><msup><mi>W</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo>−</mo><mi>W</mi><msub><mi mathvariant="normal">∥</mi><mi>F</mi></msub></mrow><annotation encoding="application/x-tex">\|W&#x27; - W\|_F</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.0019em;vertical-align:-0.25em;"></span><span class="mord">∥</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.1389em;">W</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.7519em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">′</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">W</span><span class="mord"><span class="mord">∥</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.1389em;">F</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> matches <code>p50c</code>'s change size before <code>bf16</code> rounding (<code>10.6%</code> of the matrix at layer 15 and <code>8.1%</code> at layer 19, from Lab 1). Despite equal change size, <code>randdir</code> (<code>0.0039</code> looping vs <code>0.0048</code> productive) and <code>randcols</code> (<code>0.0038</code> looping vs <code>0.0045</code> productive) move both text types by nearly the same small amount, shifting productive texts slightly more. Editing all 3,584 neurons (<code>all1</code>, <code>all2</code>) is also less selective than editing only the 64 chosen ones. At <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">\alpha = 2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">2</span></span></span></span>, <code>a2</code> (64 neurons) moves productive text by <code>0.0018</code> and looping text by <code>0.0035</code> (ratio about 1.9), whereas <code>all2</code> (3,584 neurons) moves them by <code>0.0063</code> and <code>0.0092</code> (ratio about 1.5), shifting productive text 3.5 times more than <code>a2</code>.</p>
<figure class="mermaid-svg" role="img" aria-label="Diagram: 3. Reading the four findings in the table" style="--mw:672px"><svg id="m-c823a720ff4c" width="100%" xmlns="http://www.w3.org/2000/svg" class="flowchart" style="max-width: 896px;" viewBox="0 0 896 470" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#m-c823a720ff4c{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#m-c823a720ff4c .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#m-c823a720ff4c .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#m-c823a720ff4c .error-icon{fill:#552222;}#m-c823a720ff4c .error-text{fill:#552222;stroke:#552222;}#m-c823a720ff4c .edge-thickness-normal{stroke-width:1px;}#m-c823a720ff4c .edge-thickness-thick{stroke-width:3.5px;}#m-c823a720ff4c .edge-pattern-solid{stroke-dasharray:0;}#m-c823a720ff4c .edge-thickness-invisible{stroke-width:0;fill:none;}#m-c823a720ff4c .edge-pattern-dashed{stroke-dasharray:3;}#m-c823a720ff4c .edge-pattern-dotted{stroke-dasharray:2;}#m-c823a720ff4c .marker{fill:#666;stroke:#666;}#m-c823a720ff4c .marker.cross{stroke:#666;}#m-c823a720ff4c svg{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;}#m-c823a720ff4c p{margin:0;}#m-c823a720ff4c .label{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;color:#000000;}#m-c823a720ff4c .cluster-label text{fill:#333;}#m-c823a720ff4c .cluster-label span{color:#333;}#m-c823a720ff4c .cluster-label span p{background-color:transparent;}#m-c823a720ff4c .label text,#m-c823a720ff4c span{fill:#000000;color:#000000;}#m-c823a720ff4c .node rect,#m-c823a720ff4c .node circle,#m-c823a720ff4c .node ellipse,#m-c823a720ff4c .node polygon,#m-c823a720ff4c .node path{fill:#eee;stroke:#999;stroke-width:1px;}#m-c823a720ff4c .rough-node .label text,#m-c823a720ff4c .node .label text,#m-c823a720ff4c .image-shape .label,#m-c823a720ff4c .icon-shape .label{text-anchor:middle;}#m-c823a720ff4c .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#m-c823a720ff4c .rough-node .label,#m-c823a720ff4c .node .label,#m-c823a720ff4c .image-shape .label,#m-c823a720ff4c .icon-shape .label{text-align:center;}#m-c823a720ff4c .node.clickable{cursor:pointer;}#m-c823a720ff4c .root .anchor path{fill:#666!important;stroke-width:0;stroke:#666;}#m-c823a720ff4c .arrowheadPath{fill:#333333;}#m-c823a720ff4c .edgePaths .path{stroke:#666;stroke-width:1px;}#m-c823a720ff4c .flowchart-link{stroke:#666;fill:none;}#m-c823a720ff4c .edgeLabel{background-color:white;text-align:center;}#m-c823a720ff4c .edgeLabel p{background-color:white;}#m-c823a720ff4c .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#m-c823a720ff4c .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#m-c823a720ff4c .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#m-c823a720ff4c .cluster text{fill:#333;}#m-c823a720ff4c .cluster span{color:#333;}#m-c823a720ff4c .node .collapsed-indicator{fill:#707070;stroke:none;opacity:0.6;}#m-c823a720ff4c .node .collapsed-separator{stroke:#707070;stroke-width:0.75px;}#m-c823a720ff4c div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#m-c823a720ff4c .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#m-c823a720ff4c rect.text{fill:none;stroke-width:0;}#m-c823a720ff4c .icon-shape,#m-c823a720ff4c .image-shape{background-color:white;text-align:center;}#m-c823a720ff4c .icon-shape p,#m-c823a720ff4c .image-shape p{background-color:white;padding:2px;}#m-c823a720ff4c .icon-shape .label rect,#m-c823a720ff4c .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#m-c823a720ff4c .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#m-c823a720ff4c .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#m-c823a720ff4c .node .neo-node{stroke:#999;}#m-c823a720ff4c [data-look="neo"].node rect,#m-c823a720ff4c [data-look="neo"].cluster rect,#m-c823a720ff4c [data-look="neo"].node polygon{stroke:url(#m-c823a720ff4c-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-c823a720ff4c [data-look="neo"].swimlane.cluster rect{filter:none;}#m-c823a720ff4c [data-look="neo"].node path{stroke:url(#m-c823a720ff4c-gradient);stroke-width:1px;}#m-c823a720ff4c [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-c823a720ff4c [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#m-c823a720ff4c [data-look="neo"].node circle{stroke:url(#m-c823a720ff4c-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-c823a720ff4c [data-look="neo"].node circle .state-start{fill:#000000;}#m-c823a720ff4c [data-look="neo"].icon-shape .icon{fill:url(#m-c823a720ff4c-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-c823a720ff4c [data-look="neo"].icon-shape .icon-neo path{stroke:url(#m-c823a720ff4c-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-c823a720ff4c :root{--mermaid-font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;}</style><g><marker id="m-c823a720ff4c_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-c823a720ff4c_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-c823a720ff4c_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></path></marker><marker id="m-c823a720ff4c_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></polygon></marker><marker id="m-c823a720ff4c_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-c823a720ff4c_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-c823a720ff4c_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-c823a720ff4c_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-c823a720ff4c_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-c823a720ff4c_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-c823a720ff4c_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5;"></path></marker><marker id="m-c823a720ff4c_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5; stroke-dasharray: 1, 0;"></path></marker><g class="root"><g class="clusters"></g><g class="edgePaths"><path d="M318,90.871L288,98.226C258,105.581,198,120.29,168,133.145C138,146,138,157,138,162.5L138,168" id="m-c823a720ff4c-L_E_T_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_E_T_0" data-points="W3sieCI6MzE4LCJ5Ijo5MC44NzA5Njc3NDE5MzU0OX0seyJ4IjoxMzgsInkiOjEzNX0seyJ4IjoxMzgsInkiOjE3Mn1d" data-look="classic" marker-end="url(#m-c823a720ff4c_flowchart-v2-pointEnd)"></path><path d="M448,110L448,114.167C448,118.333,448,126.667,448,134.333C448,142,448,149,448,152.5L448,156" id="m-c823a720ff4c-L_E_R1_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_E_R1_0" data-points="W3sieCI6NDQ4LCJ5IjoxMTB9LHsieCI6NDQ4LCJ5IjoxMzV9LHsieCI6NDQ4LCJ5IjoxNjB9XQ==" data-look="classic" marker-end="url(#m-c823a720ff4c_flowchart-v2-pointEnd)"></path><path d="M578,90.871L608,98.226C638,105.581,698,120.29,728,131.145C758,142,758,149,758,152.5L758,156" id="m-c823a720ff4c-L_E_R2_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_E_R2_0" data-points="W3sieCI6NTc4LCJ5Ijo5MC44NzA5Njc3NDE5MzU0OX0seyJ4Ijo3NTgsInkiOjEzNX0seyJ4Ijo3NTgsInkiOjE2MH1d" data-look="classic" marker-end="url(#m-c823a720ff4c_flowchart-v2-pointEnd)"></path><path d="M138,274L138,280.167C138,286.333,138,298.667,138,308.333C138,318,138,325,138,328.5L138,332" id="m-c823a720ff4c-L_T_TO_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_T_TO_0" data-points="W3sieCI6MTM4LCJ5IjoyNzR9LHsieCI6MTM4LCJ5IjozMTF9LHsieCI6MTM4LCJ5IjozMzZ9XQ==" data-look="classic" marker-end="url(#m-c823a720ff4c_flowchart-v2-pointEnd)"></path><path d="M448,286L448,290.167C448,294.333,448,302.667,448,312.333C448,322,448,333,448,338.5L448,344" id="m-c823a720ff4c-L_R1_RO1_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_R1_RO1_0" data-points="W3sieCI6NDQ4LCJ5IjoyODZ9LHsieCI6NDQ4LCJ5IjozMTF9LHsieCI6NDQ4LCJ5IjozNDh9XQ==" data-look="classic" marker-end="url(#m-c823a720ff4c_flowchart-v2-pointEnd)"></path><path d="M758,286L758,290.167C758,294.333,758,302.667,758,312.333C758,322,758,333,758,338.5L758,344" id="m-c823a720ff4c-L_R2_RO2_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_R2_RO2_0" data-points="W3sieCI6NzU4LCJ5IjoyODZ9LHsieCI6NzU4LCJ5IjozMTF9LHsieCI6NzU4LCJ5IjozNDh9XQ==" data-look="classic" marker-end="url(#m-c823a720ff4c_flowchart-v2-pointEnd)"></path></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_E_T_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_E_R1_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_E_R2_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_T_TO_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_R1_RO1_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_R2_RO2_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g></g><g class="nodes"><g class="node default" id="m-c823a720ff4c-flowchart-E-0" data-look="classic" transform="translate(448, 59)"><rect class="basic label-container" style="" x="-130" y="-51" width="260" height="102"></rect><g class="label" style="" transform="translate(-100, -36)"><rect></rect><foreignObject width="200" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Same Frobenius weight change size<br>at layers 15 and 19</p></span></div></foreignObject></g></g><g class="node default" id="m-c823a720ff4c-flowchart-T-1" data-look="classic" transform="translate(138, 223)"><rect class="basic label-container" style="" x="-130" y="-51" width="260" height="102"></rect><g class="label" style="" transform="translate(-100, -36)"><rect></rect><foreignObject width="200" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Targeted edit (p50c)<br>top 64 loop neurons + loop direction u</p></span></div></foreignObject></g></g><g class="node default" id="m-c823a720ff4c-flowchart-R1-3" data-look="classic" transform="translate(448, 223)"><rect class="basic label-container" style="" x="-130" y="-63" width="260" height="126"></rect><g class="label" style="" transform="translate(-100, -48)"><rect></rect><foreignObject width="200" height="96"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Random direction control (randdir)<br>same 64 neurons + random orthogonal u</p></span></div></foreignObject></g></g><g class="node default" id="m-c823a720ff4c-flowchart-R2-5" data-look="classic" transform="translate(758, 223)"><rect class="basic label-container" style="" x="-130" y="-63" width="260" height="126"></rect><g class="label" style="" transform="translate(-100, -48)"><rect></rect><foreignObject width="200" height="96"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Random columns control (randcols)<br>64 other neurons + loop direction u</p></span></div></foreignObject></g></g><g class="node default" id="m-c823a720ff4c-flowchart-TO-7" data-look="classic" transform="translate(138, 399)"><rect class="basic label-container" style="" x="-130" y="-63" width="260" height="126"></rect><g class="label" style="" transform="translate(-100, -48)"><rect></rect><foreignObject width="200" height="96"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Selective shift:<br>KL looping 0.0241 vs productive 0.0097 (about 2.5x)</p></span></div></foreignObject></g></g><g class="node default" id="m-c823a720ff4c-flowchart-RO1-9" data-look="classic" transform="translate(448, 399)"><rect class="basic label-container" style="" x="-130" y="-51" width="260" height="102"></rect><g class="label" style="" transform="translate(-100, -36)"><rect></rect><foreignObject width="200" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Unselective shift:<br>KL looping 0.0039 vs productive 0.0048</p></span></div></foreignObject></g></g><g class="node default" id="m-c823a720ff4c-flowchart-RO2-11" data-look="classic" transform="translate(758, 399)"><rect class="basic label-container" style="" x="-130" y="-51" width="260" height="102"></rect><g class="label" style="" transform="translate(-100, -36)"><rect></rect><foreignObject width="200" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Unselective shift:<br>KL looping 0.0038 vs productive 0.0045</p></span></div></foreignObject></g></g></g></g></g><defs><filter id="m-c823a720ff4c-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><defs><filter id="m-c823a720ff4c-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><linearGradient id="m-c823a720ff4c-gradient" gradientUnits="objectBoundingBox" x1="0%" y1="0%" x2="100%" y2="0%"><stop offset="0%" stop-color="hsl(0, 0%, 83.3333333333%)" stop-opacity="1"></stop><stop offset="100%" stop-color="hsl(0, 0%, 88.9215686275%)" stop-opacity="1"></stop></linearGradient></svg></figure>
<p>KL has no sign: it measures how far predictions move, not which way. <code>a-4</code> pushes the 64 neurons toward loops instead of away, yet it is selective too (looping texts move 2.6 times more than productive ones). It is selective because it acts in the same place as <code>p50c</code>: where those 64 neurons push hard along <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span>, which happens on looping text. Selectivity shows where an edit acts; Lab 4's accuracy runs show whether it helps or hurts.</p>
<p>Second, trace the WikiText-2 perplexity dose curve across <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span> (<code>a-4</code>, <code>a1</code>, <code>a2</code>, <code>a4</code>, <code>p50c</code> at <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>, <code>a8</code>, <code>a12</code>) in the left panel of <code>lab7_side_effects.png</code>:</p>
<ul>
<li>Setting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span> to <code>-4</code> (<code>a-4</code>, pushing 5 times harder in the loop direction) raises perplexity by <code>+5.39%</code> (<code>21.088</code>).</li>
<li>Small positive edits (<code>a1</code> <code>-0.49%</code>, <code>a2</code> <code>-0.69%</code>, <code>a4</code> <code>-0.47%</code>, <code>all1</code> <code>-0.42%</code>, <code>all2</code> <code>-0.36%</code>) lower WikiText-2 perplexity slightly below the base model's <code>20.009</code>, reaching <code>19.871</code> at <code>a2</code>.</li>
<li>At the released strength <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span> (<code>p50c</code>), flipping and amplifying 5x costs <code>+0.79%</code> perplexity (<code>20.167</code>).</li>
<li>Above <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>, perplexity climbs fast: <code>a8</code> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">\alpha = 8</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">8</span></span></span></span>) adds <code>+3.33%</code> (<code>20.675</code>), and <code>a12</code> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>12</mn></mrow><annotation encoding="application/x-tex">\alpha = 12</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">12</span></span></span></span>) adds <code>+14.34%</code> (<code>22.878</code>) with WikiText KL <code>0.1643</code>.</li>
</ul>
<p>Third, the single-layer rows <code>l15only</code> and <code>l19only</code> show that KL does not add across edits. On WikiText-2, editing layer 15 alone moves predictions slightly more (<code>wikitext_kl = 0.0348</code>) than editing both layers 15 and 19 (<code>p50c</code>, <code>0.0333</code>). Yet <code>l15only</code> barely changes perplexity (<code>+0.05%</code>), while <code>l19only</code> moves predictions less (KL <code>0.0102</code>) and raises perplexity more (<code>+1.65%</code>). This table cannot say why. One plausible story: layer 19 sits four blocks after layer 15, so the layer-15 edit changes the input reaching layer 19's 64 neurons. Because the two loop directions point along similar lines (absolute cosine similarity <code>0.849</code> in Lab 1, where <code>1</code> means the same direction and <code>0</code> means at right angles), the layer-19 edit may undo part of the layer-15 shift on ordinary text. <code>l15only</code> is also the most selective edit in the table, moving looping texts 3.5 times more than productive ones, and in Lab 4 it fixes almost as many problems as the full edit (<code>60.4%</code> correct on the dev set vs <code>61.5%</code> for <code>p50c</code>).</p>
<p>Fourth, WikiText-2 is plain encyclopedia text scored with no chat template and no thinking. This table measures damage to ordinary language modeling, not to math reasoning or following instructions. Lab 4's dose runs measure the math side. Here, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>12</mn></mrow><annotation encoding="application/x-tex">\alpha = 12</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">12</span></span></span></span> hurts ordinary text a lot and <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span> barely touches it.</p>
<h2 id="check-yourself">Check yourself<a class="anchor" href="#check-yourself" aria-hidden="true" tabindex="-1">#</a></h2>
<ol>
<li>Suppose a model assigns probability <code>0.25</code> to the true next token at every position of a test passage. What is its average negative log-likelihood (NLL), and what is its perplexity?</li>
</ol>
<details>
<summary>Answer</summary>
<p>At every token, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>NLL</mtext><mo>=</mo><mo>−</mo><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mn>0.25</mn><mo stretchy="false">)</mo><mo>=</mo><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mn>4</mn><mo stretchy="false">)</mo><mo>≈</mo><mn>1.3863</mn></mrow><annotation encoding="application/x-tex">\text{NLL} = -\ln(0.25) = \ln(4) \approx 1.3863</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord text"><span class="mord">NLL</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">−</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord">0.25</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mop">ln</span><span class="mopen">(</span><span class="mord">4</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.3863</span></span></span></span>. Raising <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>e</mi></mrow><annotation encoding="application/x-tex">e</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">e</span></span></span></span> to the mean NLL gives <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>perplexity</mtext><mo>=</mo><msup><mi>e</mi><mrow><mi>ln</mi><mo>⁡</mo><mo stretchy="false">(</mo><mn>4</mn><mo stretchy="false">)</mo></mrow></msup><mo>=</mo><mn>4.0</mn></mrow><annotation encoding="application/x-tex">\text{perplexity} = e^{\ln(4)} = 4.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord text"><span class="mord">perplexity</span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.888em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.888em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mop mtight"><span class="mtight">l</span><span class="mtight">n</span></span><span class="mopen mtight">(</span><span class="mord mtight">4</span><span class="mclose mtight">)</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">4.0</span></span></span></span>. A constant true-token probability of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi>K</mi></mrow><annotation encoding="application/x-tex">1/K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">1/</span><span class="mord mathnormal" style="margin-right:0.0715em;">K</span></span></span></span> always gives perplexity <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi></mrow><annotation encoding="application/x-tex">K</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.0715em;">K</span></span></span></span>.</p>
</details>
<ol start="2">
<li>How can <code>a2</code> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">\alpha = 2</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">2</span></span></span></span>) have a positive KL divergence on WikiText-2 (<code>0.0044</code>) while its WikiText-2 perplexity change is negative (<code>-0.69%</code>)?</li>
</ol>
<details>
<summary>Answer</summary>
<p>KL divergence measures how far the edited probabilities moved from the base probabilities in either direction, so any change makes KL positive. Perplexity measures how much probability lands on the actual next token in WikiText-2. For <code>a2</code>, the small shift puts slightly more probability on the true next token on average, so perplexity drops from <code>20.009</code> to <code>19.871</code> (<code>-0.69%</code>) while KL records the <code>0.0044</code> shift.</p>
</details>
<ol start="3">
<li>What specific alternative explanation does <code>randdir</code> rule out, and what specific alternative explanation does <code>randcols</code> rule out?</li>
</ol>
<details>
<summary>Answer</summary>
<p><code>randdir</code> edits the same 64 neurons as <code>p50c</code> with the same change norm along a random direction orthogonal to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span>; its flat response (<code>0.0039</code> looping vs <code>0.0048</code> productive) argues against the idea that perturbing those 64 neurons in any direction would target loops. <code>randcols</code> uses the loop direction <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span> and the same change norm on 64 random neurons; its flat response (<code>0.0038</code> looping vs <code>0.0045</code> productive) argues against the idea that projecting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span> out of any 64 neurons works just as well. In these tests, selectivity required both the loop direction <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span> and the neurons that fire during loops. Each control is a single random draw, so treat this as good evidence rather than proof.</p>
</details>
<ol start="4">
<li>Why can <code>l15only</code> have a larger <code>wikitext_kl</code> (<code>0.0348</code>) than <code>p50c</code> (<code>0.0333</code>), even though <code>p50c</code> includes the exact same layer-15 edit plus the layer-19 edit?</li>
</ol>
<details>
<summary>Answer</summary>
<p>KL doesn't add across edits. The layer-19 edit acts on a stream that the layer-15 edit already changed, so the two shifts can partly cancel instead of stacking. On WikiText-2 they seem to cancel a little, bringing predictions slightly back toward the base model's. This table shows that it happens, not how. Measuring the KL between <code>l15only</code> and <code>p50c</code> directly would show how much the layer-19 edit moves predictions once the layer-15 edit is in place.</p>
</details>
<h2 id="try-this-next">Try this next<a class="anchor" href="#try-this-next" aria-hidden="true" tabindex="-1">#</a></h2>
<p>Each experiment writes its JSON file and figure to <code>results/lab7/try-&lt;name&gt;/</code>, leaving <code>results/lab7/side_effects.json</code> untouched. Our try folders are committed so you can compare your run with ours.</p>
<ol>
<li>
<p><strong>Rank the edits by how selective they are.</strong> For each model, divide its KL on the looping texts by its KL on the productive texts, and sort:</p>
<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">python -c &#x27;import json; m=json.load(open(&quot;results/lab7/side_effects.json&quot;))[&quot;models&quot;]; rows=[(v[&quot;kl_looping_texts&quot;]/v[&quot;kl_productive_texts&quot;], k, v[&quot;ppl_change_pct&quot;]) for k,v in m.items() if k != &quot;base&quot;]; print(&quot;\n&quot;.join(f&quot;{k:10s} ratio={r:.2f}x ppl_change={p:+.2f}%&quot; for r,k,p in sorted(rows, reverse=True)))&#x27;
</code></pre></div>
<p>Our output:</p>
<div class="code"><div class="code-head"><span class="code-lang">text</span></div><pre><code class="language-text">l15only ratio=3.52x ppl_change=+0.05%
a12 ratio=2.81x ppl_change=+14.34%
a8 ratio=2.68x ppl_change=+3.33%
a-4 ratio=2.63x ppl_change=+5.39%
p50c ratio=2.49x ppl_change=+0.79%
rebuilt ratio=2.48x ppl_change=+0.79%
a4 ratio=2.31x ppl_change=-0.47%
a2 ratio=1.95x ppl_change=-0.69%
l19only ratio=1.63x ppl_change=+1.65%
all2 ratio=1.45x ppl_change=-0.36%
a1 ratio=1.44x ppl_change=-0.49%
all1 ratio=1.28x ppl_change=-0.42%
randcols ratio=0.86x ppl_change=-0.10%
randdir ratio=0.81x ppl_change=+0.30%
</code></pre></div>
<p>The layer-15-only edit ranks first, ahead of the released edit. The <code>a-4</code> edit, which pushes toward loops, also ranks above <code>p50c</code> because KL measures how far predictions move, not which way. Both random controls rank last, below 1, moving productive text slightly more than looping text.</p>
</li>
<li>
<p><strong>Score only the last 36 tokens of each text.</strong> In each pair, the productive and looping texts share their prompt and opening lines before the looping text starts repeating. Those shared tokens get the same KL, pulling the two KL columns together. <code>--last 36</code> scores only the last 36 tokens of each of the 8 texts (the shortest has 98 tokens):</p>
<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --last 36
</code></pre></div>
<p>This writes <code>results/lab7/try-last-36/</code>, with WikiText-2 columns unchanged. Point the ranking one-liner at <code>results/lab7/try-last-36/side_effects.json</code> to compare:</p>
<div class="table-wrap"><table>
<thead>
<tr>
<th>Model</th>
<th style="text-align:right">Ratio over all tokens</th>
<th style="text-align:right">Ratio over the last 36 tokens</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="cell"><span>l15only</span></code></td>
<td style="text-align:right"><code class="cell"><span>3.52</span></code></td>
<td style="text-align:right"><code class="cell"><span>12.96</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>p50c</span></code></td>
<td style="text-align:right"><code class="cell"><span>2.49</span></code></td>
<td style="text-align:right"><code class="cell"><span>6.31</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>rebuilt</span></code></td>
<td style="text-align:right"><code class="cell"><span>2.48</span></code></td>
<td style="text-align:right"><code class="cell"><span>6.18</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>all1</span></code></td>
<td style="text-align:right"><code class="cell"><span>1.28</span></code></td>
<td style="text-align:right"><code class="cell"><span>2.07</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>randcols</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.86</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.49</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>randdir</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.81</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.34</span></code></td>
</tr>
</tbody>
</table></div>
<p>Every edit along <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span> gets more selective, while the random controls get less selective. For <code>p50c</code>, productive KL barely moves (<code>0.0097</code> to <code>0.0093</code>), while looping KL more than doubles (<code>0.0241</code> to <code>0.0584</code>). This fits Lab 6, where the trigger <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi><mo stretchy="false">(</mo><mi>t</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">s(t)</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord mathnormal">s</span><span class="mopen">(</span><span class="mord mathnormal">t</span><span class="mclose">)</span></span></span></span> rises in the last 512 tokens before a loop and peaks inside the repetition (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>15</mn></msub></mrow><annotation encoding="application/x-tex">s_{15}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal">s</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3011em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">15</span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> goes <code>0.096 -&gt; 0.178 -&gt; 0.424</code> from early in a looping answer, to just before the loop, to inside it).</p>
</li>
<li>
<p><strong>Score less WikiText-2.</strong> <code>--chunks 8</code> scores the first 8 chunks (8,192 tokens) instead of all 64:</p>
<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --chunks 8
</code></pre></div>
<p>This writes <code>results/lab7/try-chunks-8/</code>. Base perplexity is <code>22.804</code> on these 8,192 tokens versus <code>20.009</code> on all 65,536, which is fine because every model is compared with the base model on the same tokens. The WikiText KL ranking matches the 64-chunk run except that <code>p50c</code> and <code>rebuilt</code> (tied to 4 decimal places) swap places. Perplexity changes are less stable: the four largest keep their order (<code>a12</code> <code>+12.21%</code>, <code>a-4</code> <code>+5.16%</code>, <code>a8</code> <code>+2.43%</code>, <code>l19only</code> <code>+1.47%</code>), while shifts under 1% shuffle (<code>p50c</code> drops from <code>+0.79%</code> to <code>+0.20%</code>, and <code>l15only</code> flips from <code>+0.05%</code> to <code>-0.54%</code>). Because KL averages non-negative per-position values, positions cannot cancel out, which helps explain why its ranking held up. A perplexity change subtracts two nearly equal averages, so when it is under 1%, even its sign depends on which text you score. The run took 70 seconds versus 78 for 64 chunks: scoring the text is the fast part.</p>
</li>
<li>
<p><strong>Score Lab 3's rebuild experiments (<code>try-lab3-tries</code>).</strong> After building the three Lab 3 experiment models (<code>rebuilt-pairs2</code>, <code>rebuilt-windows36-36</code>, and <code>rebuilt-layers15-19-21</code>), compare them against the main 4-pair rebuild:</p>
<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models rebuilt rebuilt-pairs2 rebuilt-windows36-36 rebuilt-layers15-19-21 --try-name lab3-tries
</code></pre></div>
<p>Because <code>rebuilt-layers15-19-21</code> also edits layer 21, the script swaps <code>down_proj</code> at layers <code>[15, 19, 21]</code> and writes <code>results/lab7/try-lab3-tries/side_effects.json</code> (base perplexity <code>20.009</code>):</p>
<div class="table-wrap"><table>
<thead>
<tr>
<th>Model</th>
<th style="text-align:right">Perplexity</th>
<th style="text-align:right">Change</th>
<th style="text-align:right">KL WikiText</th>
<th style="text-align:right">KL productive</th>
<th style="text-align:right">KL looping</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="cell"><span>rebuilt</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.167</span></code></td>
<td style="text-align:right"><code class="cell"><span>+0.79%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0333</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0096</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0238</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>rebuilt-pairs2</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.096</span></code></td>
<td style="text-align:right"><code class="cell"><span>+0.43%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0194</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0075</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0109</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>rebuilt-windows36-36</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.151</span></code></td>
<td style="text-align:right"><code class="cell"><span>+0.71%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0326</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0096</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0241</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>rebuilt-layers15-19-21</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.307</span></code></td>
<td style="text-align:right"><code class="cell"><span>+1.49%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0394</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0114</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0280</span></code></td>
</tr>
</tbody>
</table></div>
<p>All four models score between <code>61.3%</code> and <code>62.3%</code> on <code>dev256</code>, yet their side effects separate cleanly. Using only 2 pairs (<code>rebuilt-pairs2</code>) produces a gentler, less selective edit (<code>+0.43%</code> perplexity, looping-to-productive KL ratio <code>1.46</code> vs <code>2.48</code>), while widening the neuron window to 36 tokens (<code>rebuilt-windows36-36</code>) changes almost nothing. Adding layer 21 (<code>rebuilt-layers15-19-21</code>) nearly doubles the WikiText-2 perplexity cost (<code>+1.49%</code> vs <code>+0.79%</code>) without helping accuracy (<code>61.3%</code> vs <code>61.5%</code>).</p>
</li>
<li>
<p><strong>Score your main capstone edits (after Lab 8).</strong> Lab 8 adds two models built from the unedited model's 225 loops at <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span>: <code>capA-a6</code> (loop minus just before) and <code>capB-a6</code> (loop minus well before). Compare them with the released edit and the most selective edit:</p>
<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models p50c l15only capA-a6 capB-a6
</code></pre></div>
<p>This writes <code>results/lab7/try-models-p50c-l15only-capA-a6-capB-a6/</code>. Our output:</p>
<div class="code"><div class="code-head"><span class="code-lang">text</span></div><pre><code class="language-text">models: [&#x27;base&#x27;, &#x27;p50c&#x27;, &#x27;l15only&#x27;, &#x27;capA-a6&#x27;, &#x27;capB-a6&#x27;]
model ppl ppl +% KL wiki KL prod KL loop
base 20.009 0.00 0.00000 0.00000 0.00000
p50c 20.167 0.79 0.03327 0.00967 0.02405
l15only 20.020 0.05 0.03476 0.00550 0.01938
capA-a6 20.111 0.51 0.01778 0.01025 0.01624
capB-a6 20.194 0.92 0.02711 0.01166 0.02078
wrote /nvme/wsp-lab/code/results/lab7/try-models-p50c-l15only-capA-a6-capB-a6 in 34 s
</code></pre></div>
<p>Both capstone edits move productive texts more than the released edit does, and looping texts less. Their selectivity ratios are <code>1.58</code> and <code>1.78</code> versus <code>2.49</code> for <code>p50c</code>. In Lab 8, <code>capA-a6</code> also helps less: it scores <code>43.0%</code> on the dev set with <code>52.5%</code> of answers cut off, versus <code>61.5%</code> accuracy and <code>19.7%</code> cut for <code>p50c</code> (<code>capB-a6</code> reaches <code>58.2%</code> accuracy with <code>30.7%</code> cut).</p>
</li>
<li>
<p><strong>Score Lab 8's capstone experiments (<code>try-lab8-tries</code>).</strong> Lab 8 tests building the capstone edits from only the first 16 loops (<code>capA-max-samples-16</code>, <code>capB-max-samples-16</code>) or raising the strength from <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span> to <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">\alpha = 8</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">8</span></span></span></span> on all 225 loops (<code>capA-alpha-8</code>, <code>capB-alpha-8</code>):</p>
<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">CUDA_VISIBLE_DEVICES=7 python labs/lab7_side_effects.py --models capA-max-samples-16 capB-max-samples-16 capA-alpha-8 capB-alpha-8 --try-name lab8-tries
</code></pre></div>
<p>This writes <code>results/lab7/try-lab8-tries/side_effects.json</code> in about 33 seconds. Here are those four models alongside the main <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi><mo>=</mo><mn>6</mn></mrow><annotation encoding="application/x-tex">\alpha = 6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">6</span></span></span></span> capstone edits (<code>capA-a6</code> and <code>capB-a6</code>, base perplexity <code>20.009</code>):</p>
<div class="table-wrap"><table>
<thead>
<tr>
<th>Model</th>
<th style="text-align:right">Loops used</th>
<th style="text-align:right"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span></th>
<th><code class="cell"><span>dev256</span></code> accuracy %</th>
<th style="text-align:right">Perplexity</th>
<th style="text-align:right">Change</th>
<th style="text-align:right">KL WikiText</th>
<th style="text-align:right">KL productive</th>
<th style="text-align:right">KL looping</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="cell"><span>capA-a6</span></code> (main)</td>
<td style="text-align:right"><code class="cell"><span>225</span></code></td>
<td style="text-align:right"><code class="cell"><span>6</span></code></td>
<td><code class="cell"><span>43.0 +/- 2.6</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.111</span></code></td>
<td style="text-align:right"><code class="cell"><span>+0.51%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0178</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0103</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0162</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>capA-max-samples-16</span></code></td>
<td style="text-align:right"><code class="cell"><span>16</span></code></td>
<td style="text-align:right"><code class="cell"><span>6</span></code></td>
<td><code class="cell"><span>17.6 +/- 1.9</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.029</span></code></td>
<td style="text-align:right"><code class="cell"><span>+0.10%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0081</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0063</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0086</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>capA-alpha-8</span></code></td>
<td style="text-align:right"><code class="cell"><span>225</span></code></td>
<td style="text-align:right"><code class="cell"><span>8</span></code></td>
<td><code class="cell"><span>42.8 +/- 2.8</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.331</span></code></td>
<td style="text-align:right"><code class="cell"><span>+1.61%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0298</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0183</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0265</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>capB-a6</span></code> (main)</td>
<td style="text-align:right"><code class="cell"><span>225</span></code></td>
<td style="text-align:right"><code class="cell"><span>6</span></code></td>
<td><code class="cell"><span>58.2 +/- 2.9</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.194</span></code></td>
<td style="text-align:right"><code class="cell"><span>+0.92%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0271</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0117</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0208</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>capB-max-samples-16</span></code></td>
<td style="text-align:right"><code class="cell"><span>16</span></code></td>
<td style="text-align:right"><code class="cell"><span>6</span></code></td>
<td><code class="cell"><span>53.7 +/- 2.8</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.021</span></code></td>
<td style="text-align:right"><code class="cell"><span>+0.06%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0204</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0102</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0173</span></code></td>
</tr>
<tr>
<td><code class="cell"><span>capB-alpha-8</span></code></td>
<td style="text-align:right"><code class="cell"><span>225</span></code></td>
<td style="text-align:right"><code class="cell"><span>8</span></code></td>
<td><code class="cell"><span>54.5 +/- 2.9</span></code></td>
<td style="text-align:right"><code class="cell"><span>20.562</span></code></td>
<td style="text-align:right"><code class="cell"><span>+2.76%</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0468</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0206</span></code></td>
<td style="text-align:right"><code class="cell"><span>0.0340</span></code></td>
</tr>
</tbody>
</table></div>
<p>Raising <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>α</mi></mrow><annotation encoding="application/x-tex">\alpha</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0037em;">α</span></span></span></span> from <code>6</code> to <code>8</code> roughly triples the WikiText-2 perplexity cost (<code>+0.51%</code> to <code>+1.61%</code> for recipe A, <code>+0.92%</code> to <code>+2.76%</code> for recipe B) and nearly doubles the KL shifts (<code>0.0178</code> to <code>0.0298</code> and <code>0.0271</code> to <code>0.0468</code> on WikiText; <code>0.0103</code> to <code>0.0183</code> and <code>0.0117</code> to <code>0.0206</code> on productive texts). Yet <code>dev256</code> accuracy does not improve: recipe A scores <code>42.8%</code> vs <code>43.0%</code> (paired change <code>-0.2 +/- 2.3</code> points), and recipe B scores <code>54.5%</code> vs <code>58.2%</code> (paired drop <code>-3.7 +/- 1.6</code> points). Pushing harder along an imperfect direction scales the off-target part of <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>u</mi></mrow><annotation encoding="application/x-tex">u</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal">u</span></span></span></span> along with the useful part. Using only 16 loops (<code>capB-max-samples-16</code>) keeps ordinary-text side effects tiny (<code>+0.06%</code> perplexity, <code>0.0204</code> WikiText KL) while reaching <code>53.7%</code> on <code>dev256</code> (<code>+41.6</code> points over the unedited model).</p>
<p><code>capA-max-samples-16</code> shows why you should not read this table on its own. It has some of the smallest side effects here (<code>+0.10%</code> perplexity, <code>0.0081</code> WikiText KL), yet it reaches only <code>17.6%</code> on <code>dev256</code> (<code>5.5 +/- 1.9</code> points above the unedited model) with <code>82.6%</code> of answers still cut off. By comparison, <code>all2</code> has a similar WikiText KL (<code>0.0078</code>) and reaches <code>49.4%</code>. Small side effects are good news only when the edit also does its job. Read side effects next to the accuracy column. <a href="lab8_capstone.html">Lab 8</a> explains why this edit fails.</p>
</li>
</ol>
</article>
<nav class="pager" aria-label="Previous and next"><a class="prev" href="lab6_steering.html" rel="prev"><span>Previous</span><b>Lab 6: The weight edit is steering in disguise</b></a><a class="next" href="lab8_capstone.html" rel="next"><span>Next</span><b>Lab 8 (capstone): build your own edit from the model&#x27;s own loops</b></a></nav>
</main>
<aside class="toc"><nav aria-label="On this page"><p class="toc-title">On this page</p><ul><li class="l2"><a href="#before-you-run-predict">Before you run: predict</a></li><li class="l2"><a href="#run-it">Run it</a></li><li class="l2"><a href="#what-you-should-see">What you should see</a></li><li class="l2"><a href="#why-it-works">Why it works</a></li><li class="l3"><a href="#1-testing-15-models-the-base-model-plus-14-edits-by-swapping-matrices-in-memory">1. Testing 15 models (the base model plus 14 edits) by swapping matrices in memory</a></li><li class="l3"><a href="#2-perplexity-and-kl-divergence-from-scratch">2. Perplexity and KL divergence from scratch</a></li><li class="l3"><a href="#3-reading-the-four-findings-in-the-table">3. Reading the four findings in the table</a></li><li class="l2"><a href="#check-yourself">Check yourself</a></li><li class="l2"><a href="#try-this-next">Try this next</a></li></ul></nav></aside>
</div>
<footer class="foot">
<div class="foot-in">
<p class="foot-note">Based on Experiment 8 of "<a href="https://rikiyat.github.io/blog/weight-space-programming/" target="_blank" rel="noopener">Sparks of Weight-Space Programming</a>". Logs and the agent's texts come from <a href="https://github.com/RikiyaT/lm_understand-logs" target="_blank" rel="noopener">RikiyaT/lm_understand-logs</a>, <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" rel="noopener">CC BY 4.0</a>.</p>
<p class="foot-links"><a href="https://huggingface.co/spaces/thegovind/weight-space-programming/blob/main/code/labs/lab7_side_effects.md" target="_blank" rel="noopener">View source</a></p>
</div>
</footer>
</body>
</html>