| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8" /> |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> |
| <title>Weight-Space Geometry of Offline Reasoning Training</title> |
| <meta name="description" content="An interactive look at the weight-space geometry of six offline reasoning losses trained on identical data." /> |
| <link rel="preconnect" href="https://fonts.googleapis.com" /> |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin /> |
| <link href="https://fonts.googleapis.com/css2?family=Fraunces:opsz,wght@9..144,400;9..144,500;9..144,600&family=Newsreader:ital,opsz,wght@0,6..72,400;0,6..72,500;0,6..72,600;1,6..72,400;1,6..72,500&family=IBM+Plex+Mono:wght@400;500&display=swap" rel="stylesheet" /> |
| <script src="https://cdn.plot.ly/plotly-2.35.2.min.js" charset="utf-8"></script> |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/katex.min.css" /> |
| <script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/katex.min.js"></script> |
| <link rel="stylesheet" href="style.css" /> |
| </head> |
| <body> |
|
|
| <a class="skip" href="#abstract">Skip to content</a> |
|
|
| <nav id="toc" aria-label="Table of contents"> |
| <div class="toc-inner"> |
| <div class="toc-title">Contents</div> |
| <ol> |
| <li><a href="#abstract">Abstract</a></li> |
| <li><a href="#reading">Reading the geometry</a></li> |
| <li><a href="#methods">The six losses</a></li> |
| <li><a href="#cosine">Cosine map</a></li> |
| <li><a href="#perlayer">Layer by layer</a></li> |
| <li><a href="#cka">Representations (CKA)</a></li> |
| <li><a href="#svd">Output subspace</a></li> |
| <li><a href="#angles">Principal angles</a></li> |
| <li><a href="#geometry">Update geometry</a></li> |
| <li><a href="#lmc">Mode connectivity</a></li> |
| <li><a href="#accuracy">Accuracy</a></li> |
| <li><a href="#seedlr">Seed & LR</a></li> |
| <li><a href="#takeaways">Takeaways</a></li> |
| </ol> |
| </div> |
| </nav> |
|
|
| <main> |
| <header class="hero"> |
| <div class="venue" id="venue"></div> |
| <h1 id="title"></h1> |
| <p class="lede" id="subtitle"></p> |
| <div class="byline"> |
| <a id="paperlink" href="#" target="_blank" rel="noopener">paper ↗</a> |
| · interactive companion · |
| <a id="repolink" href="#" target="_blank" rel="noopener">source & data ↗</a> |
| </div> |
| </header> |
|
|
| <section id="abstract" class="aside"> |
| <h2>Abstract</h2> |
| <p id="abstract-text" class="dropcap"></p> |
| <p> |
| Six losses, one base model, one fixed set of math rollouts. If the data is held |
| identical, what does the <em>choice of loss</em> actually do to the weights? This page |
| lets you turn the same knobs we did — pick method pairs, scrub across all 36 layers, |
| and watch each geometric metric respond. |
| </p> |
| </section> |
|
|
| <section id="reading" class="aside"> |
| <h2>Reading the geometry</h2> |
| <p> |
| Every chart below is computed on the LoRA weight update <strong>ΔW</strong> — the small |
| change each method writes into the base model — or on the representations that update |
| produces. Four tools, each asking a different version of <em>“are these two the same?”</em> |
| </p> |
| <div class="note"> |
| <dl class="glossary"> |
| <div> |
| <dt>Cosine similarity</dt> |
| <dd>Do two weight updates point the same way? <b>+1</b> = identical direction, <b>0</b> = orthogonal (unrelated), <b>below 0</b> = opposed. This is the headline number, taken on the stacked ΔW.</dd> |
| </div> |
| <div> |
| <dt>Principal angles</dt> |
| <dd>How far apart are the <b>subspaces</b> the two updates span — a basis-free generalization of cosine. A few degrees means effectively the same subspace; near 90° means disjoint.</dd> |
| </div> |
| <div> |
| <dt>Mode connectivity<span>linear · LMC</span></dt> |
| <dd>Interpolate between two trained adapters and watch the loss. A flat path means they sit in the <b>same basin</b>; a bump in the middle is a barrier separating two different solutions.</dd> |
| </div> |
| <div> |
| <dt>CKA<span>centered kernel alignment</span></dt> |
| <dd>Do the two models compute the same thing <b>inside</b>? Unlike the others, CKA compares hidden representations, not weights. <b>≈1</b> = near-identical computation; lower means the circuit has been rewired.</dd> |
| </div> |
| </dl> |
| </div> |
| </section> |
|
|
| <section id="methods"> |
| <h2>The six losses</h2> |
| <p> |
| Every method is trained on the same rollouts from Qwen3-4B-Instruct with attention-only |
| LoRA (q, k, v, o; rank 32). They differ only in how the loss treats negatives, reward, |
| and a reference policy. |
| </p> |
| <figure class="wide"> |
| <table id="methods-table" class="methods"></table> |
| </figure> |
| <p class="block-label">The objectives, written out</p> |
| <div class="note"> |
| <dl class="glossary objectives" id="objectives"></dl> |
| </div> |
| <p class="footnote"> |
| Every objective above is token-level <strong>cross-entropy</strong>: the term |
| <span id="ce-eq">−𝔼 log π_θ(y∣x)</span> is exactly the CE between a rollout and the |
| model. SFT, RFT and RIFT are the <em>same</em> CE, only reweighted per rollout — |
| over all tokens, over positives only, or by reward. DFT reweights it by the |
| stop-gradient probability; GRPO and DPO leave the CE form entirely. |
| </p> |
| </section> |
|
|
| <section id="cosine"> |
| <h2>A map of directions</h2> |
| <p> |
| Start global. Stack each method's LoRA update into a single vector ΔW and measure the |
| cosine between every pair. Three blocks of the matrix tell the whole story: a hot |
| reward-weighted cluster (SFT / RFT / RIFT), a lukewarm Offline GRPO, and a cold, |
| near-orthogonal DPO. Add the on-policy methods and they detach from everything offline. |
| </p> |
| <figure class="wide"> |
| <div class="controls"> |
| <div class="seg" role="group" aria-label="matrix size"> |
| <button class="seg-btn active" data-mx="6">6 offline losses</button> |
| <button class="seg-btn" data-mx="8">+ online RL (8×8)</button> |
| </div> |
| <span class="hint">hover a cell for the value · click to inspect that pair below</span> |
| </div> |
| <div id="chart-cosine" class="plot square"></div> |
| <figcaption data-cap="cosine"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="perlayer"> |
| <h2>Layer by layer</h2> |
| <p> |
| A single number hides where methods agree. Here is the cosine of ΔW computed |
| independently in each of the 36 transformer blocks. Toggle pairs and drag the slider: |
| the SFT family is colinear from embedding to head, while DPO and online RL stay pinned |
| near zero — and Offline GRPO peels away in the <em>late</em> layers. |
| </p> |
| <figure class="wide"> |
| <div class="controls" id="perlayer-pairs"></div> |
| <div id="chart-perlayer" class="plot"></div> |
| <div class="slider-row"> |
| <label for="layer-slider">Layer</label> |
| <input type="range" id="layer-slider" min="0" max="35" value="30" step="1" /> |
| <output id="layer-readout"></output> |
| </div> |
| <figcaption data-cap="perlayer"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="cka"> |
| <h2>Does it rewire the computation?</h2> |
| <p> |
| Cosine compares <em>updates</em>. CKA compares what the network actually computes — |
| the hidden representations. Most methods leave them almost untouched (CKA ≈ 1). DPO is |
| the exception: its representation similarity collapses in the final blocks, the |
| fingerprint of a method that changes the circuit, not just the write direction. The |
| layer slider is shared with the chart above. |
| </p> |
| <figure class="wide"> |
| <div class="controls" id="cka-pairs"></div> |
| <div id="chart-cka" class="plot"></div> |
| <figcaption data-cap="cka"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="svd"> |
| <h2>Same answer, different basis</h2> |
| <p> |
| Low cosine does not always mean a different solution. Decompose each ΔW and compare only |
| the dominant <em>output</em> direction (the top left-singular vector u). Across the SFT |
| family these stay aligned even where the raw vectors diverge — the updates point the same |
| way in output space while differing in their input-side basis, an artifact of random LoRA |
| initialization rather than a genuinely different circuit. |
| </p> |
| <figure class="wide"> |
| <div class="controls" id="svd-pairs"></div> |
| <div id="chart-svd" class="plot"></div> |
| <figcaption data-cap="svd"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="angles"> |
| <h2>How far apart are the subspaces?</h2> |
| <p> |
| Principal angles measure the gap between the subspaces two updates span — a basis-free |
| version of cosine. SFT and RFT sit about 7° apart (effectively the same subspace); |
| SFT and DPO open up to ~55°. Each bar is the median over 144 modules; the whisker shows |
| the spread of the worst of the top-10 angles. |
| </p> |
| <figure class="wide"> |
| <div id="chart-angles" class="plot"></div> |
| <figcaption data-cap="angles"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="geometry"> |
| <h2>Size and rank of the move</h2> |
| <p> |
| Direction is only half of it. How <em>far</em> does each loss push, and how concentrated |
| is the push? The SFT family travels far along a low-rank direction; DPO barely moves yet |
| spreads that tiny step across a much higher effective rank — a small, broad nudge versus |
| a large, focused shove. |
| </p> |
| <figure class="wide"> |
| <div id="chart-geometry" class="plot"></div> |
| <figcaption data-cap="geometry"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="lmc"> |
| <h2>One basin or two?</h2> |
| <p> |
| Linearly interpolate between two trained adapters and watch the loss. A flat or monotone |
| path means the two solutions share a basin; a bump in the middle is an energy barrier |
| separating them. SFT ↔ Offline GRPO is barrier-free — same basin. Paths into DPO climb a |
| wall. |
| </p> |
| <figure class="wide"> |
| <div class="controls" id="lmc-pairs"></div> |
| <div id="chart-lmc" class="plot"></div> |
| <figcaption data-cap="lmc"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="accuracy"> |
| <h2>Does the geometry show up in accuracy?</h2> |
| <p> |
| Yes — and it inverts the usual intuition. The methods that move <em>orthogonally</em> to |
| the SFT direction (DPO and on-policy RL) hold onto the base model's accuracy, while the |
| colinear SFT family drags GSM8K below base. Online GRPO posts the best AIME26. |
| </p> |
| <figure class="wide"> |
| <div class="controls"> |
| <div class="seg" role="group" aria-label="benchmark"> |
| <button class="seg-btn active" data-bench="gsm8k">GSM8K</button> |
| <button class="seg-btn" data-bench="aime26">AIME26</button> |
| </div> |
| </div> |
| <div id="chart-accuracy" class="plot"></div> |
| <figcaption data-cap="accuracy"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="seedlr"> |
| <h2>Is the geometry an artifact of seed or learning rate?</h2> |
| <p> |
| A fair worry: maybe the directions are just noise. They are not. Two seeds of the same |
| loss produce a low raw weight-cosine — yet the top-1 <em>output</em> direction stays at |
| ~0.99. The disagreement is entirely in the input-side basis (random LoRA A-init), not in |
| the solution. Separately, a 10× learning-rate change <em>rotates</em> ΔW rather than |
| merely rescaling it — so DPO's smaller LR is genuinely part of its geometry. |
| </p> |
| <figure class="wide"> |
| <div class="controls"> |
| <div class="seg" role="group" aria-label="seed view"> |
| <button class="seg-btn active" data-seed="cos">Raw cosine vs output dir</button> |
| <button class="seg-btn" data-seed="lr">LR rotates ΔW</button> |
| </div> |
| </div> |
| <div id="chart-seedlr" class="plot"></div> |
| <figcaption data-cap="seedlr"></figcaption> |
| </figure> |
| </section> |
|
|
| <section id="takeaways"> |
| <h2>Takeaways</h2> |
| <ul class="takeaways"> |
| <li><strong>The reward-weighted MLE family is one direction.</strong> SFT, RFT, and RIFT have cosine ≥ 0.94 and ~7° top-1 principal angle — interchangeable in weight space.</li> |
| <li><strong>DFT diverges the most among offline losses</strong> despite seeing identical data — the stop-gradient reshaping matters geometrically.</li> |
| <li><strong>Offline GRPO stays in the SFT basin but adds a large orthogonal late-layer component</strong> (up to ~86% off-SFT in the final blocks).</li> |
| <li><strong>DPO is the outlier:</strong> near-orthogonal subspace, a mode-connectivity barrier, late-layer CKA collapse — and the best accuracy, at a 10× smaller learning rate.</li> |
| <li><strong>On-policy RL is geometrically unlike everything offline.</strong> Online GRPO/DAPO are near-orthogonal to every offline loss and to each other: shared-rollout colinearity is partly an artifact of training on the same fixed data.</li> |
| </ul> |
| <p class="repro"> |
| Base model Qwen3-4B-Instruct-2507 · attention-only LoRA (q,k,v,o, r32 a64) · DeepScaleR |
| math rollouts · math-verify reward. All metrics on this page are computed from the |
| published analysis JSON. <a id="repolink2" href="#" target="_blank" rel="noopener">Code, adapters, and raw results ↗</a> |
| </p> |
| </section> |
|
|
| <footer> |
| <p>Built as an interactive companion to <span id="footer-title"></span>. Figures rendered client-side with Plotly from the paper's released metrics.</p> |
| </footer> |
| </main> |
|
|
| <script src="app.js"></script> |
| </body> |
| </html> |
|
|
