Download labs/lab5_decoding.html from thegovind/weight-space-programming: direct link, hf CLI and curl.
- Browser
- Download file 146 kB
-
https://huggingface.co/spaces/thegovind/weight-space-programming/resolve/main/labs/lab5_decoding.html
- Command line
-
hf download hf://spaces/thegovind/weight-space-programming/labs/lab5_decoding.html
-
curl -L -o lab5_decoding.html https://huggingface.co/spaces/thegovind/weight-space-programming/resolve/main/labs/lab5_decoding.html
146 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <meta name="color-scheme" content="only light"> | |
| <meta name="theme-color" content="#f5f5f5"> | |
| <title>Lab 5: can decoding settings do the same job? · Weight-space programming lab</title> | |
| <meta name="description" content="Big thing: We compare the 131,072-weight edit against six decoding changes on unedited Qwen3.5-0.8B (temperature 1.0, presence penalty 0 and 2.0, repetition penalty 1.1, a shorter 4,096-token cap…"> | |
| <link rel="icon" href="../static/favicon.svg" type="image/svg+xml"> | |
| <link rel="preload" href="../static/fonts/geist-latin-wght-normal.woff2" as="font" type="font/woff2" crossorigin> | |
| <link rel="stylesheet" href="../static/site.css"> | |
| <link rel="stylesheet" href="../static/katex/katex.min.css"> | |
| <script>document.documentElement.classList.add("js")</script> | |
| <script defer src="../static/site.js"></script> | |
| </head> | |
| <body data-copy="Copy" data-copied="Copied"> | |
| <a class="skip" href="#main">Skip to content</a> | |
| <header class="top"> | |
| <div class="top-in"> | |
| <a class="brand" href="../index.html">Weight-space programming lab</a> | |
| <nav class="top-links" aria-label="Site"><a href="../index.html">Home</a><a href="../quickstart.html">Quick start</a><a href="../index.html#the-nine-labs" aria-current="true">Labs</a><a href="../reference/theory-guide.html">Theory</a><a href="../results/summary.html">Results</a><a href="https://huggingface.co/spaces/thegovind/weight-space-programming/tree/main/code/" target="_blank" rel="noopener">Code</a></nav> | |
| </div> | |
| </header> | |
| <div class="page"> | |
| <main id="main" class="doc"> | |
| <article class="prose"> | |
| <h1 id="lab-5-can-decoding-settings-do-the-same-job">Lab 5: can decoding settings do the same job?</h1> | |
| <details class="tocm"><summary>On this page</summary><nav aria-label="On this page"><ul><li class="l2"><a href="#before-you-run-predict">Before you run: predict</a></li><li class="l2"><a href="#run-it">Run it</a></li><li class="l2"><a href="#what-you-should-see">What you should see</a></li><li class="l3"><a href="#1-all-nine-runs-on-dev256-512-answers-per-run">1. All nine runs on <code>dev256</code> (<code>512</code> answers per run)</a></li><li class="l3"><a href="#2-breakdown-by-problem-source-gsm8k-vs-math-and-median-length-of-correct-answers">2. Breakdown by problem source (GSM8K vs MATH) and median length of correct answers</a></li><li class="l3"><a href="#3-how-every-thinking-mode-answer-ended-resultslab5endingsjson">3. How every thinking-mode answer ended (<code>results/lab5/endings.json</code>)</a></li><li class="l2"><a href="#why-it-works">Why it works</a></li><li class="l3"><a href="#1-decoding-from-first-principles-how-vllm-turns-logits-into-a-chosen-token">1. Decoding from first principles: how vLLM turns logits into a chosen token</a></li><li class="l3"><a href="#2-a-worked-example-why-token-penalties-fail-to-break-a-strong-loop">2. A worked example: why token penalties fail to break a strong loop</a></li><li class="l3"><a href="#3-why-turning-think-off-scores-521-and-why-the-edit-adds-nothing-there">3. Why turning <code><think></code> off scores 52.1%, and why the edit adds nothing there</a></li><li class="l3"><a href="#4-the-papers-gemma-3-270m-follow-up-points-the-same-way">4. The paper's Gemma 3 270M follow-up points the same way</a></li><li class="l2"><a href="#check-yourself">Check yourself</a></li><li class="l2"><a href="#try-this-next">Try this next</a></li></ul></nav></details> | |
| <p><strong>Big thing:</strong> We compare the 131,072-weight edit against six decoding changes on unedited Qwen3.5-0.8B (temperature <code>1.0</code>, presence penalty <code>0</code> and <code>2.0</code>, repetition penalty <code>1.1</code>, a shorter <code>4,096</code>-token cap, and <code><think></code> mode off) across all 512 <code>dev256</code> answers. Penalties and temperature raise accuracy at most from <code>12.1%</code> to <code>34.4%</code>, turning thinking off reaches <code>52.1%</code> in 13 times fewer tokens than the unedited thinking baseline, and the weight edit with thinking on reaches <code>61.5%</code> (<code>74.6%</code> on the GSM8K half).</p> | |
| <p><strong>Why it matters:</strong> Always test whether a weight edit beats simpler decoding changes that leave the weights alone. Ending counts suggest penalties mostly add normal closes, while the weight edit adds both normal closes and loop escapes. These counts do not prove which loops were prevented, because the check reads only the text right before <code></think></code>.</p> | |
| <p><strong>Time and hardware:</strong> We already ran it (<code>results/summary.md</code>, <code>results/lab5/endings.json</code>, and <code>results/figures/lab5_decoding.png</code>). Re-running one unedited <code>dev256</code> job on 1 NVIDIA A100 GPU takes <code>34</code> to <code>49</code> minutes with thinking on at the <code>32,768</code>-token cap, or about <code>3</code> minutes with thinking off or a <code>4,096</code>-token cap.</p> | |
| <p><strong>Theory guide:</strong> Section 13.2.</p> | |
| <h2 id="before-you-run-predict">Before you run: predict<a class="anchor" href="#before-you-run-predict" aria-hidden="true" tabindex="-1">#</a></h2> | |
| <p>Write down a guess for each question before looking at the table.</p> | |
| <ol> | |
| <li>The paper's baseline (<code>"think35"</code>) uses <code>presence_penalty = 1.5</code>, which subtracts <code>1.5</code> from the score of any token that has already appeared in the answer. If you turn that penalty off (<code>presence_penalty = 0</code>), what happens to the share of answers cut at the 32,768-token cap (now <code>88.7%</code>)?</li> | |
| <li>The unedited model's correct <code>dev256</code> answers have a median length of <code>5,600</code> tokens, while looping answers run to <code>32,768</code> tokens. What happens to accuracy if you lower the token cap to <code>4,096</code> tokens?</li> | |
| <li>What if you turn <code><think></code> mode off (<code>enable_thinking=False</code>), so the model writes a direct step-by-step solution instead of an open-ended internal scratchpad? Will it beat the unedited thinking model (<code>12.1%</code>), and will the weight edit (<code>p50c</code>) still help when thinking is off?</li> | |
| </ol> | |
| <h2 id="run-it">Run it<a class="anchor" href="#run-it" aria-hidden="true" tabindex="-1">#</a></h2> | |
| <p>All seven decoding runs are at the bottom of <code>plans/sampler.json</code> and ran on <code>sampler</code> alongside the Lab 4 variants (<code>dev/base</code> and <code>dev/p50c</code> at the top of <code>plans/sampler.json</code> are the baselines).</p> | |
| <p>Open <code>plans/sampler.json</code> and check the last seven entries:</p> | |
| <div class="code"><div class="code-head"><span class="code-lang">json</span></div><pre><code class="language-json"> {"name": "dev/base+temp1.0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["temperature=1.0"]}, | |
| {"name": "dev/base+pp0", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=0"]}, | |
| {"name": "dev/base+pp2", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 15, "set": ["presence_penalty=2.0"]}, | |
| {"name": "dev/base+rep1.1", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 14, "set": ["repetition_penalty=1.1"]}, | |
| {"name": "dev/base+max4k", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "set": ["max_tokens=4096"]}, | |
| {"name": "dev/base+nothink", "model": "base", "data": "dev256", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"}, | |
| {"name": "dev/p50c+nothink", "model": "qwen35-p50c", "data": "dev256", "n": 2, "shards": 1, "est": 1, "preset": "nothink35"} | |
| </code></pre></div> | |
| <p>Each entry starts from a preset in <code>wsp/protocol.py</code>:</p> | |
| <ul> | |
| <li><code>"think35"</code> (the default when <code>"preset"</code> is omitted): <code>temperature = 0.6</code>, <code>top_p = 0.95</code>, <code>top_k = 20</code>, <code>presence_penalty = 1.5</code>, <code>repetition_penalty = 1.0</code>, <code>max_tokens = 32768</code>, <code>thinking = True</code>.</li> | |
| <li><code>"nothink35"</code> (Qwen's model-card settings for non-thinking mode): <code>temperature = 0.7</code>, <code>top_p = 0.8</code>, <code>top_k = 20</code>, <code>presence_penalty = 1.5</code>, <code>repetition_penalty = 1.0</code>, <code>max_tokens = 8192</code>, <code>thinking = False</code>.</li> | |
| <li>A <code>"set"</code> list overrides fields in the preset (<code>scripts/run_jobs.py</code> passes each <code>"key=value"</code> string to <code>scripts/gen_worker.py</code> as <code>--set</code>).</li> | |
| </ul> | |
| <p>These runs finished on <code>sampler</code> and are in <code>results/runs/</code> on this laptop. On 1 A100 GPU, a full <code>dev256</code> run takes <code>17</code> to <code>49</code> minutes (<code>33.8</code> to <code>48.9</code> minutes for the long unedited runs; <code>dev/p50c</code> took <code>20.6</code> minutes), while <code>max_tokens=4096</code> and <code>nothink35</code> take about <code>3</code> minutes (<code>2.7</code> to <code>3.2</code> minutes).</p> | |
| <p>To inspect how answers ended or read single answers on <code>sampler</code> (every VM command starts with <code>. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code</code>):</p> | |
| <div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">ssh sampler | |
| . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code | |
| export CUDA_VISIBLE_DEVICES="" | |
| # Classify how every answer ended across the Lab 5 runs (writes results/lab5/endings.json) | |
| python labs/lab4_endings.py \ | |
| dev/base dev/base+temp1.0 dev/base+pp0 dev/base+pp2 dev/base+rep1.1 dev/base+max4k \ | |
| dev/base+nothink dev/p50c+nothink \ | |
| --out lab5/endings | |
| # List or read individual answers from any decoding run on sampler | |
| python labs/peek.py dev/base+rep1.1 --id gsm8k_train_5628 | |
| python labs/peek.py dev/base+nothink --id gsm8k_train_5628 | |
| </code></pre></div> | |
| <p><code>labs/lab4_endings.py</code> skips <code>dev/base+nothink</code> and <code>dev/p50c+nothink</code> with a message (<code>thinking is off, so its answers have no </think> to look for</code>).</p> | |
| <p>To redraw <code>results/figures/lab5_decoding.png</code> on this laptop:</p> | |
| <div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash"># On the local laptop, inside ~/weight-space-programming/code | |
| python3 scripts/summarize.py results/runs | |
| uv run -q --with numpy --with matplotlib python scripts/plot_runs.py | |
| </code></pre></div> | |
| <h2 id="what-you-should-see">What you should see<a class="anchor" href="#what-you-should-see" aria-hidden="true" tabindex="-1">#</a></h2> | |
| <h3 id="1-all-nine-runs-on-dev256-512-answers-per-run">1. All nine runs on <code>dev256</code> (<code>512</code> answers per run)<a class="anchor" href="#1-all-nine-runs-on-dev256-512-answers-per-run" aria-hidden="true" tabindex="-1">#</a></h3> | |
| <p>Every run uses the same 256 unseen practice problems (128 GSM8K-train + 128 MATH-train, 2 answers each = 512 answers). Paired gain compares each run against <code>dev/base</code>, except <code>dev/p50c+nothink</code>, which <code>scripts/summarize.py</code> pairs against <code>dev/base+nothink</code> to isolate the weight edit when thinking is off:</p> | |
| <div class="table-wrap"><table> | |
| <thead> | |
| <tr> | |
| <th>Run</th> | |
| <th>Setting</th> | |
| <th>Accuracy %</th> | |
| <th>Cut at the cap %</th> | |
| <th>Mean tokens</th> | |
| <th>Correct when finished %</th> | |
| <th>Cut answers ending in repetition %</th> | |
| <th>Paired gain vs baseline (pp)</th> | |
| <th>Rescued / lost problems</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td><code class="cell"><span>dev/base</span></code></td> | |
| <td>Unedited, paper settings (<code class="cell"><span>think35</span></code>)</td> | |
| <td><code class="cell"><span>12.1 +/- 1.6</span></code></td> | |
| <td><code class="cell"><span>88.7</span></code></td> | |
| <td><code class="cell"><span>29,683</span></code></td> | |
| <td><code class="cell"><span>100.0</span></code></td> | |
| <td><code class="cell"><span>96.9</span></code></td> | |
| <td><code class="cell"><span>-</span></code></td> | |
| <td><code class="cell"><span>-</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+temp1.0</span></code></td> | |
| <td>Temperature <code class="cell"><span>1.0</span></code> (from <code class="cell"><span>0.6</span></code>)</td> | |
| <td><code class="cell"><span>27.9 +/- 2.5</span></code></td> | |
| <td><code class="cell"><span>71.3</span></code></td> | |
| <td><code class="cell"><span>27,430</span></code></td> | |
| <td><code class="cell"><span>95.9</span></code></td> | |
| <td><code class="cell"><span>59.5</span></code></td> | |
| <td><code class="cell"><span>+15.8 +/- 2.1</span></code></td> | |
| <td><code class="cell"><span>52 / 9</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+pp0</span></code></td> | |
| <td>Presence penalty <code class="cell"><span>0</span></code> (from <code class="cell"><span>1.5</span></code>)</td> | |
| <td><code class="cell"><span>4.5 +/- 0.9</span></code></td> | |
| <td><code class="cell"><span>99.0</span></code></td> | |
| <td><code class="cell"><span>32,472</span></code></td> | |
| <td><code class="cell"><span>100.0</span></code></td> | |
| <td><code class="cell"><span>98.4</span></code></td> | |
| <td><code class="cell"><span>-7.6 +/- 1.6</span></code></td> | |
| <td><code class="cell"><span>11 / 42</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+pp2</span></code></td> | |
| <td>Presence penalty <code class="cell"><span>2.0</span></code> (from <code class="cell"><span>1.5</span></code>)</td> | |
| <td><code class="cell"><span>16.2 +/- 1.9</span></code></td> | |
| <td><code class="cell"><span>84.6</span></code></td> | |
| <td><code class="cell"><span>28,614</span></code></td> | |
| <td><code class="cell"><span>98.7</span></code></td> | |
| <td><code class="cell"><span>96.8</span></code></td> | |
| <td><code class="cell"><span>+4.1 +/- 1.7</span></code></td> | |
| <td><code class="cell"><span>24 / 13</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+rep1.1</span></code></td> | |
| <td>Repetition penalty <code class="cell"><span>1.1</span></code> (from <code class="cell"><span>1.0</span></code>)</td> | |
| <td><code class="cell"><span>34.4 +/- 2.6</span></code></td> | |
| <td><code class="cell"><span>61.7</span></code></td> | |
| <td><code class="cell"><span>23,301</span></code></td> | |
| <td><code class="cell"><span>89.8</span></code></td> | |
| <td><code class="cell"><span>88.3</span></code></td> | |
| <td><code class="cell"><span>+22.3 +/- 2.2</span></code></td> | |
| <td><code class="cell"><span>62 / 3</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+max4k</span></code></td> | |
| <td>Cap at <code class="cell"><span>4,096</span></code> tokens (from <code class="cell"><span>32,768</span></code>)</td> | |
| <td><code class="cell"><span>5.9 +/- 1.1</span></code></td> | |
| <td><code class="cell"><span>95.5</span></code></td> | |
| <td><code class="cell"><span>4,019</span></code></td> | |
| <td><code class="cell"><span>100.0</span></code></td> | |
| <td><code class="cell"><span>18.2</span></code></td> | |
| <td><code class="cell"><span>-6.2 +/- 1.2</span></code></td> | |
| <td><code class="cell"><span>3 / 30</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+nothink</span></code></td> | |
| <td>Thinking off, cap <code class="cell"><span>8,192</span></code> (<code class="cell"><span>nothink35</span></code>)</td> | |
| <td><code class="cell"><span>52.1 +/- 2.7</span></code></td> | |
| <td><code class="cell"><span>18.0</span></code></td> | |
| <td><code class="cell"><span>2,270</span></code></td> | |
| <td><code class="cell"><span>63.6</span></code></td> | |
| <td><code class="cell"><span>7.6</span></code></td> | |
| <td><code class="cell"><span>+40.0 +/- 2.5</span></code></td> | |
| <td><code class="cell"><span>115 / 2</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/p50c</span></code></td> | |
| <td>Edited (<code class="cell"><span>p50c</span></code>), paper settings</td> | |
| <td><code class="cell"><span>61.5 +/- 2.8</span></code></td> | |
| <td><code class="cell"><span>19.7</span></code></td> | |
| <td><code class="cell"><span>16,104</span></code></td> | |
| <td><code class="cell"><span>76.6</span></code></td> | |
| <td><code class="cell"><span>24.8</span></code></td> | |
| <td><code class="cell"><span>+49.4 +/- 2.6</span></code></td> | |
| <td><code class="cell"><span>125 / 0</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/p50c+nothink</span></code></td> | |
| <td>Edited (<code class="cell"><span>p50c</span></code>), thinking off</td> | |
| <td><code class="cell"><span>53.5 +/- 2.8</span></code></td> | |
| <td><code class="cell"><span>16.4</span></code></td> | |
| <td><code class="cell"><span>2,306</span></code></td> | |
| <td><code class="cell"><span>64.0</span></code></td> | |
| <td><code class="cell"><span>0.0</span></code></td> | |
| <td><code class="cell"><span>+1.4 +/- 1.9</span></code> (vs <code class="cell"><span>base+nothink</span></code>)</td> | |
| <td><code class="cell"><span>22 / 26</span></code></td> | |
| </tr> | |
| </tbody> | |
| </table></div> | |
| <p><img src="../code/results/figures/lab5_decoding.png" alt="Three bar charts on the same 256 practice problems: accuracy, percent of answers cut at the token cap, and mean answer length, for the unedited model with six decoding changes and for the edited model with thinking on and off." loading="lazy" decoding="async" width="2210" height="721" /></p> | |
| <h3 id="2-breakdown-by-problem-source-gsm8k-vs-math-and-median-length-of-correct-answers">2. Breakdown by problem source (GSM8K vs MATH) and median length of correct answers<a class="anchor" href="#2-breakdown-by-problem-source-gsm8k-vs-math-and-median-length-of-correct-answers" aria-hidden="true" tabindex="-1">#</a></h3> | |
| <p>Splitting <code>dev256</code> into its 128 GSM8K problems and 128 harder MATH problems (<code>results/summary.json</code>) shows where the tokens go and where thinking mode pays off:</p> | |
| <div class="table-wrap"><table> | |
| <thead> | |
| <tr> | |
| <th>Run</th> | |
| <th>GSM8K accuracy % / cut %</th> | |
| <th>MATH accuracy % / cut %</th> | |
| <th>Median length of correct answers (tokens)</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td><code class="cell"><span>dev/base</span></code> (unedited, thinking on)</td> | |
| <td><code class="cell"><span>15.2</span></code> / <code class="cell"><span>86.3</span></code></td> | |
| <td><code class="cell"><span>9.0</span></code> / <code class="cell"><span>91.0</span></code></td> | |
| <td><code class="cell"><span>5,600</span></code> (<code class="cell"><span>5,600.5</span></code>)</td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+temp1.0</span></code></td> | |
| <td><code class="cell"><span>35.9</span></code> / <code class="cell"><span>62.1</span></code></td> | |
| <td><code class="cell"><span>19.9</span></code> / <code class="cell"><span>80.5</span></code></td> | |
| <td><code class="cell"><span>12,548</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+pp0</span></code> (presence penalty <code class="cell"><span>0</span></code>)</td> | |
| <td><code class="cell"><span>6.2</span></code> / <code class="cell"><span>98.4</span></code></td> | |
| <td><code class="cell"><span>2.7</span></code> / <code class="cell"><span>99.6</span></code></td> | |
| <td><code class="cell"><span>32,768</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+pp2</span></code> (presence penalty <code class="cell"><span>2.0</span></code>)</td> | |
| <td><code class="cell"><span>20.7</span></code> / <code class="cell"><span>80.1</span></code></td> | |
| <td><code class="cell"><span>11.7</span></code> / <code class="cell"><span>89.1</span></code></td> | |
| <td><code class="cell"><span>5,353</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+rep1.1</span></code> (repetition penalty <code class="cell"><span>1.1</span></code>)</td> | |
| <td><code class="cell"><span>46.5</span></code> / <code class="cell"><span>48.8</span></code></td> | |
| <td><code class="cell"><span>22.3</span></code> / <code class="cell"><span>74.6</span></code></td> | |
| <td><code class="cell"><span>5,700.5</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+max4k</span></code> (cap <code class="cell"><span>4,096</span></code>)</td> | |
| <td><code class="cell"><span>7.4</span></code> / <code class="cell"><span>94.1</span></code></td> | |
| <td><code class="cell"><span>4.3</span></code> / <code class="cell"><span>96.9</span></code></td> | |
| <td><code class="cell"><span>3,307</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+nothink</span></code> (thinking off)</td> | |
| <td><code class="cell"><span>58.2</span></code> / <code class="cell"><span>4.3</span></code></td> | |
| <td><code class="cell"><span>46.1</span></code> / <code class="cell"><span>31.6</span></code></td> | |
| <td><code class="cell"><span>452</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/p50c</span></code> (edited, thinking on)</td> | |
| <td><code class="cell"><span>74.6</span></code> / <code class="cell"><span>4.7</span></code></td> | |
| <td><code class="cell"><span>48.4</span></code> / <code class="cell"><span>34.8</span></code></td> | |
| <td><code class="cell"><span>7,998</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/p50c+nothink</span></code> (edited, thinking off)</td> | |
| <td><code class="cell"><span>62.1</span></code> / <code class="cell"><span>2.0</span></code></td> | |
| <td><code class="cell"><span>44.9</span></code> / <code class="cell"><span>30.9</span></code></td> | |
| <td><code class="cell"><span>480</span></code> (<code class="cell"><span>479.5</span></code>)</td> | |
| </tr> | |
| </tbody> | |
| </table></div> | |
| <p>In <code>dev/base+pp0</code>, removing the presence penalty sends <code>99.0%</code> of answers (<code>98.4%</code> on GSM8K, <code>99.6%</code> on MATH) to the <code>32,768</code>-token cap. The median length of its <code>23</code> correct answers is <code>32,768</code> tokens because <code>18</code> of those <code>23</code> were cut while thinking after boxing an answer mid-scratchpad. In <code>dev/base+max4k</code>, because the unedited model's median correct answer is <code>5,600</code> tokens, a <code>4,096</code>-token cap cuts off valid reasoning before completion and drops accuracy from <code>12.1%</code> to <code>5.9%</code> (<code>30</code> problems lost, <code>3</code> rescued).</p> | |
| <h3 id="3-how-every-thinking-mode-answer-ended-resultslab5endingsjson">3. How every thinking-mode answer ended (<code>results/lab5/endings.json</code>)<a class="anchor" href="#3-how-every-thinking-mode-answer-ended-resultslab5endingsjson" aria-hidden="true" tabindex="-1">#</a></h3> | |
| <p>Ending counts from <code>results/lab5/endings.json</code> (plus <code>dev/p50c</code> from <code>results/lab4/endings.json</code>), with correct answers in parentheses:</p> | |
| <div class="table-wrap"><table> | |
| <thead> | |
| <tr> | |
| <th>Run (<code class="cell"><span>512</span></code> answers each)</th> | |
| <th>Total correct</th> | |
| <th>Cut while thinking (correct)</th> | |
| <th>Stopped while thinking (correct)</th> | |
| <th>Closed out of a loop (correct)</th> | |
| <th>Closed normally (correct)</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td><code class="cell"><span>dev/base</span></code></td> | |
| <td><code class="cell"><span>62</span></code></td> | |
| <td><code class="cell"><span>453 (4)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>3 (2)</span></code></td> | |
| <td><code class="cell"><span>56 (56)</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+temp1.0</span></code></td> | |
| <td><code class="cell"><span>143</span></code></td> | |
| <td><code class="cell"><span>352 (2)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>3 (1)</span></code></td> | |
| <td><code class="cell"><span>157 (140)</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+pp0</span></code></td> | |
| <td><code class="cell"><span>23</span></code></td> | |
| <td><code class="cell"><span>507 (18)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>1 (1)</span></code></td> | |
| <td><code class="cell"><span>4 (4)</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+pp2</span></code></td> | |
| <td><code class="cell"><span>83</span></code></td> | |
| <td><code class="cell"><span>433 (5)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>4 (4)</span></code></td> | |
| <td><code class="cell"><span>75 (74)</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+rep1.1</span></code></td> | |
| <td><code class="cell"><span>176</span></code></td> | |
| <td><code class="cell"><span>294 (0)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>28 (11)</span></code></td> | |
| <td><code class="cell"><span>190 (165)</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/base+max4k</span></code></td> | |
| <td><code class="cell"><span>30</span></code></td> | |
| <td><code class="cell"><span>484 (7)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>28 (23)</span></code></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>dev/p50c</span></code> (weight edit)</td> | |
| <td><code class="cell"><span>315</span></code></td> | |
| <td><code class="cell"><span>72 (0)</span></code></td> | |
| <td><code class="cell"><span>0 (0)</span></code></td> | |
| <td><code class="cell"><span>96 (29)</span></code></td> | |
| <td><code class="cell"><span>344 (286)</span></code></td> | |
| </tr> | |
| </tbody> | |
| </table></div> | |
| <p>Compare the last two columns (<code>closed out of a loop</code> and <code>closed normally</code>):</p> | |
| <ul> | |
| <li>Raising temperature to <code>1.0</code> increases <code>closed normally</code> from <code>56</code> to <code>157</code>, while <code>closed out of a loop</code> stays at <code>3</code>.</li> | |
| <li>Raising presence penalty to <code>2.0</code> gives <code>4</code> loop escapes; adding repetition penalty <code>1.1</code> gives <code>28</code> loop escapes and <code>190</code> normal closes.</li> | |
| <li>The weight edit (<code>dev/p50c</code>) produces <code>344</code> normal closes and <code>96</code> loop escapes. These endings suggest decoding tweaks mostly add normal closes, while the weight edit adds both. A normal close can still follow an earlier loop, because the check reads only the last <code>6,000</code> characters before <code></think></code>.</li> | |
| </ul> | |
| <h2 id="why-it-works">Why it works<a class="anchor" href="#why-it-works" aria-hidden="true" tabindex="-1">#</a></h2> | |
| <h3 id="1-decoding-from-first-principles-how-vllm-turns-logits-into-a-chosen-token">1. Decoding from first principles: how vLLM turns logits into a chosen token<a class="anchor" href="#1-decoding-from-first-principles-how-vllm-turns-logits-into-a-chosen-token" aria-hidden="true" tabindex="-1">#</a></h3> | |
| <p>At each generation step, the language model outputs a vector of raw scores called logits, one number <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>v</mi></msub></mrow><annotation encoding="application/x-tex">z_v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> for every token <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span></span></span></span> in the vocabulary. Without decoding tweaks, the softmax function turns those logits into probabilities by exponentiating each score and dividing by the sum:</p> | |
| <div class="math-display"><span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>p</mi><mi>v</mi></msub><mo>=</mo><mfrac><msup><mi>e</mi><msub><mi>z</mi><mi>v</mi></msub></msup><mrow><munder><mo>∑</mo><mi>w</mi></munder><msup><mi>e</mi><msub><mi>z</mi><mi>w</mi></msub></msup></mrow></mfrac></mrow><annotation encoding="application/x-tex">p_v = \frac{e^{z_v}}{\sum_{w} e^{z_w}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:2.3271em;vertical-align:-0.9857em;"></span><span class="mord"><span class="mopen nulldelimiter"></span><span class="mfrac"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.3414em;"><span style="top:-2.314em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mop"><span class="mop op-symbol small-op" style="position:relative;top:0em;">∑</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.0017em;"><span style="top:-2.4003em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.0269em;">w</span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.2997em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.5904em;"><span style="top:-2.989em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1645em;"><span style="top:-2.357em;margin-left:-0.044em;margin-right:0.0714em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0269em;">w</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.143em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span style="top:-3.23em;"><span class="pstrut" style="height:3em;"></span><span class="frac-line" style="border-bottom-width:0.04em;"></span></span><span style="top:-3.677em;"><span class="pstrut" style="height:3em;"></span><span class="mord"><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1645em;"><span style="top:-2.357em;margin-left:-0.044em;margin-right:0.0714em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.143em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.9857em;"><span></span></span></span></span></span><span class="mclose nulldelimiter"></span></span></span></span></span></span></div> | |
| <p>Because <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><msub><mi>z</mi><mi>v</mi></msub></msup></mrow><annotation encoding="application/x-tex">e^{z_v}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight"><span class="mord mathnormal mtight" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1645em;"><span style="top:-2.357em;margin-left:-0.044em;margin-right:0.0714em;"><span class="pstrut" style="height:2.5em;"></span><span class="sizing reset-size3 size1 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.143em;"><span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> grows exponentially, adding <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>+</mo><mn>3</mn></mrow><annotation encoding="application/x-tex">+3</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">+</span><span class="mord">3</span></span></span></span> to a token's logit multiplies its relative odds by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>3</mn></msup><mo>≈</mo><mn>20</mn></mrow><annotation encoding="application/x-tex">e^3 \approx 20</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight">3</span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">20</span></span></span></span>.</p> | |
| <p>In vLLM <code>0.33.0</code>, sampling follows the four steps below. The helper <code>pick_prob</code> in <code>wsp/protocol.py</code> copies this order for presence penalty, temperature, top-k, and top-p, and rejects non-default repetition penalties.</p> | |
| <figure class="mermaid-svg" role="img" aria-label="Diagram: 1. Decoding from first principles: how vLLM turns logits into a chosen token" style="--mw:207px"><svg id="m-0180c66d1d81" width="100%" xmlns="http://www.w3.org/2000/svg" class="flowchart" style="max-width: 276px;" viewBox="0 0 276 774" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#m-0180c66d1d81{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#m-0180c66d1d81 .edge-animation-slow{stroke-dasharray:9,5;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#m-0180c66d1d81 .edge-animation-fast{stroke-dasharray:9,5;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#m-0180c66d1d81 .error-icon{fill:#552222;}#m-0180c66d1d81 .error-text{fill:#552222;stroke:#552222;}#m-0180c66d1d81 .edge-thickness-normal{stroke-width:1px;}#m-0180c66d1d81 .edge-thickness-thick{stroke-width:3.5px;}#m-0180c66d1d81 .edge-pattern-solid{stroke-dasharray:0;}#m-0180c66d1d81 .edge-thickness-invisible{stroke-width:0;fill:none;}#m-0180c66d1d81 .edge-pattern-dashed{stroke-dasharray:3;}#m-0180c66d1d81 .edge-pattern-dotted{stroke-dasharray:2;}#m-0180c66d1d81 .marker{fill:#666;stroke:#666;}#m-0180c66d1d81 .marker.cross{stroke:#666;}#m-0180c66d1d81 svg{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;}#m-0180c66d1d81 p{margin:0;}#m-0180c66d1d81 .label{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;color:#000000;}#m-0180c66d1d81 .cluster-label text{fill:#333;}#m-0180c66d1d81 .cluster-label span{color:#333;}#m-0180c66d1d81 .cluster-label span p{background-color:transparent;}#m-0180c66d1d81 .label text,#m-0180c66d1d81 span{fill:#000000;color:#000000;}#m-0180c66d1d81 .node rect,#m-0180c66d1d81 .node circle,#m-0180c66d1d81 .node ellipse,#m-0180c66d1d81 .node polygon,#m-0180c66d1d81 .node path{fill:#eee;stroke:#999;stroke-width:1px;}#m-0180c66d1d81 .rough-node .label text,#m-0180c66d1d81 .node .label text,#m-0180c66d1d81 .image-shape .label,#m-0180c66d1d81 .icon-shape .label{text-anchor:middle;}#m-0180c66d1d81 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#m-0180c66d1d81 .rough-node .label,#m-0180c66d1d81 .node .label,#m-0180c66d1d81 .image-shape .label,#m-0180c66d1d81 .icon-shape .label{text-align:center;}#m-0180c66d1d81 .node.clickable{cursor:pointer;}#m-0180c66d1d81 .root .anchor path{fill:#666;stroke-width:0;stroke:#666;}#m-0180c66d1d81 .arrowheadPath{fill:#333333;}#m-0180c66d1d81 .edgePaths .path{stroke:#666;stroke-width:1px;}#m-0180c66d1d81 .flowchart-link{stroke:#666;fill:none;}#m-0180c66d1d81 .edgeLabel{background-color:white;text-align:center;}#m-0180c66d1d81 .edgeLabel p{background-color:white;}#m-0180c66d1d81 .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#m-0180c66d1d81 .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#m-0180c66d1d81 .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#m-0180c66d1d81 .cluster text{fill:#333;}#m-0180c66d1d81 .cluster span{color:#333;}#m-0180c66d1d81 .node .collapsed-indicator{fill:#707070;stroke:none;opacity:0.6;}#m-0180c66d1d81 .node .collapsed-separator{stroke:#707070;stroke-width:0.75px;}#m-0180c66d1d81 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#m-0180c66d1d81 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#m-0180c66d1d81 rect.text{fill:none;stroke-width:0;}#m-0180c66d1d81 .icon-shape,#m-0180c66d1d81 .image-shape{background-color:white;text-align:center;}#m-0180c66d1d81 .icon-shape p,#m-0180c66d1d81 .image-shape p{background-color:white;padding:2px;}#m-0180c66d1d81 .icon-shape .label rect,#m-0180c66d1d81 .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#m-0180c66d1d81 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#m-0180c66d1d81 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#m-0180c66d1d81 .node .neo-node{stroke:#999;}#m-0180c66d1d81 [data-look="neo"].node rect,#m-0180c66d1d81 [data-look="neo"].cluster rect,#m-0180c66d1d81 [data-look="neo"].node polygon{stroke:url(#m-0180c66d1d81-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-0180c66d1d81 [data-look="neo"].swimlane.cluster rect{filter:none;}#m-0180c66d1d81 [data-look="neo"].node path{stroke:url(#m-0180c66d1d81-gradient);stroke-width:1px;}#m-0180c66d1d81 [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-0180c66d1d81 [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#m-0180c66d1d81 [data-look="neo"].node circle{stroke:url(#m-0180c66d1d81-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-0180c66d1d81 [data-look="neo"].node circle .state-start{fill:#000000;}#m-0180c66d1d81 [data-look="neo"].icon-shape .icon{fill:url(#m-0180c66d1d81-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-0180c66d1d81 [data-look="neo"].icon-shape .icon-neo path{stroke:url(#m-0180c66d1d81-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-0180c66d1d81 :root{--mermaid-font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;}</style><g><marker id="m-0180c66d1d81_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-0180c66d1d81_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-0180c66d1d81_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></path></marker><marker id="m-0180c66d1d81_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></polygon></marker><marker id="m-0180c66d1d81_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-0180c66d1d81_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-0180c66d1d81_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-0180c66d1d81_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-0180c66d1d81_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-0180c66d1d81_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-0180c66d1d81_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5;"></path></marker><marker id="m-0180c66d1d81_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5; stroke-dasharray: 1, 0;"></path></marker><g class="root"><g class="clusters"></g><g class="edgePaths"><path d="M138,86L138,90.167C138,94.333,138,102.667,138,110.333C138,118,138,125,138,128.5L138,132" id="m-0180c66d1d81-L_L_Pen_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_L_Pen_0" data-points="W3sieCI6MTM4LCJ5Ijo4Nn0seyJ4IjoxMzgsInkiOjExMX0seyJ4IjoxMzgsInkiOjEzNn1d" data-look="classic" marker-end="url(#m-0180c66d1d81_flowchart-v2-pointEnd)"></path><path d="M138,238L138,242.167C138,246.333,138,254.667,138,262.333C138,270,138,277,138,280.5L138,284" id="m-0180c66d1d81-L_Pen_Temp_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_Pen_Temp_0" data-points="W3sieCI6MTM4LCJ5IjoyMzh9LHsieCI6MTM4LCJ5IjoyNjN9LHsieCI6MTM4LCJ5IjoyODh9XQ==" data-look="classic" marker-end="url(#m-0180c66d1d81_flowchart-v2-pointEnd)"></path><path d="M138,414L138,418.167C138,422.333,138,430.667,138,438.333C138,446,138,453,138,456.5L138,460" id="m-0180c66d1d81-L_Temp_Filter_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_Temp_Filter_0" data-points="W3sieCI6MTM4LCJ5Ijo0MTR9LHsieCI6MTM4LCJ5Ijo0Mzl9LHsieCI6MTM4LCJ5Ijo0NjR9XQ==" data-look="classic" marker-end="url(#m-0180c66d1d81_flowchart-v2-pointEnd)"></path><path d="M138,614L138,618.167C138,622.333,138,630.667,138,638.333C138,646,138,653,138,656.5L138,660" id="m-0180c66d1d81-L_Filter_Pick_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_Filter_Pick_0" data-points="W3sieCI6MTM4LCJ5Ijo2MTR9LHsieCI6MTM4LCJ5Ijo2Mzl9LHsieCI6MTM4LCJ5Ijo2NjR9XQ==" data-look="classic" marker-end="url(#m-0180c66d1d81_flowchart-v2-pointEnd)"></path></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_L_Pen_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_Pen_Temp_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_Temp_Filter_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_Filter_Pick_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g></g><g class="nodes"><g class="node default" id="m-0180c66d1d81-flowchart-L-0" data-look="classic" transform="translate(138, 47)"><rect class="basic label-container" style="" x="-130" y="-39" width="260" height="78"></rect><g class="label" style="" transform="translate(-100, -24)"><rect></rect><foreignObject width="200" height="48"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Raw logits z_v<br>from the model's final layer</p></span></div></foreignObject></g></g><g class="node default" id="m-0180c66d1d81-flowchart-Pen-1" data-look="classic" transform="translate(138, 187)"><rect class="basic label-container" style="" x="-127" y="-51" width="254" height="102"></rect><g class="label" style="" transform="translate(-97, -36)"><rect></rect><foreignObject width="194" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel"><p>1. Penalties:<br>repetition penalty (/r or *r),<br>then presence penalty (-p)</p></span></div></foreignObject></g></g><g class="node default" id="m-0180c66d1d81-flowchart-Temp-3" data-look="classic" transform="translate(138, 351)"><rect class="basic label-container" style="" x="-130" y="-63" width="260" height="126"></rect><g class="label" style="" transform="translate(-100, -48)"><rect></rect><foreignObject width="200" height="96"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>2. Temperature:<br>divide all logits by T<br>(T < 1 sharpens, T > 1 flattens)</p></span></div></foreignObject></g></g><g class="node default" id="m-0180c66d1d81-flowchart-Filter-5" data-look="classic" transform="translate(138, 539)"><rect class="basic label-container" style="" x="-130" y="-75" width="260" height="150"></rect><g class="label" style="" transform="translate(-100, -60)"><rect></rect><foreignObject width="200" height="120"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>3. Top-k and Top-p:<br>keep top 20 tokens, then keep<br>smallest set with >= 95% mass</p></span></div></foreignObject></g></g><g class="node default" id="m-0180c66d1d81-flowchart-Pick-7" data-look="classic" transform="translate(138, 715)"><rect class="basic label-container" style="" x="-111" y="-51" width="222" height="102"></rect><g class="label" style="" transform="translate(-81, -36)"><rect></rect><foreignObject width="162" height="72"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel"><p>4. Sample one token<br>from the renormalized<br>probabilities</p></span></div></foreignObject></g></g></g></g></g><defs><filter id="m-0180c66d1d81-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><defs><filter id="m-0180c66d1d81-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><linearGradient id="m-0180c66d1d81-gradient" gradientUnits="objectBoundingBox" x1="0%" y1="0%" x2="100%" y2="0%"><stop offset="0%" stop-color="hsl(0, 0%, 83.3333333333%)" stop-opacity="1"></stop><stop offset="100%" stop-color="hsl(0, 0%, 88.9215686275%)" stop-opacity="1"></stop></linearGradient></svg></figure> | |
| <p>Each step transforms a token's score:</p> | |
| <ol> | |
| <li><strong>Penalties (<code>repetition_penalty</code> and <code>presence_penalty</code>):</strong> | |
| <ul> | |
| <li><strong>Repetition penalty <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span></span></span></span> (<code>1.1</code> in <code>dev/base+rep1.1</code>):</strong> For every token <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span></span></span></span> that has appeared anywhere in the prompt or generated answer so far, divide <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>v</mi></msub></mrow><annotation encoding="application/x-tex">z_v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span></span></span></span> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>v</mi></msub><mo>←</mo><msub><mi>z</mi><mi>v</mi></msub><mi mathvariant="normal">/</mi><mi>r</mi></mrow><annotation encoding="application/x-tex">z_v \leftarrow z_v / r</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">←</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord">/</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span></span></span></span>) if <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>v</mi></msub></mrow><annotation encoding="application/x-tex">z_v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span></span></span></span> is positive, or multiply by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span></span></span></span> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>v</mi></msub><mo>←</mo><msub><mi>z</mi><mi>v</mi></msub><mo>×</mo><mi>r</mi></mrow><annotation encoding="application/x-tex">z_v \leftarrow z_v \times r</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">←</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.7333em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">×</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span></span></span></span>) if negative. Both push the logit down in proportion to its size.</li> | |
| <li><strong>Presence penalty <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">p</span></span></span></span> (<code>1.5</code> in <code>"think35"</code>):</strong> For every token <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>v</mi></mrow><annotation encoding="application/x-tex">v</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.4306em;"></span><span class="mord mathnormal" style="margin-right:0.0359em;">v</span></span></span></span> that has appeared in the generated answer so far (ignoring the prompt), subtract <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">p</span></span></span></span> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>v</mi></msub><mo>←</mo><msub><mi>z</mi><mi>v</mi></msub><mo>−</mo><mi>p</mi></mrow><annotation encoding="application/x-tex">z_v \leftarrow z_v - p</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">←</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.7333em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.625em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">p</span></span></span></span>). This penalty is binary: a token used once and a token used 200 times both lose <code>1.5</code>.</li> | |
| </ul> | |
| </li> | |
| <li><strong>Temperature <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi></mrow><annotation encoding="application/x-tex">T</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span></span></span></span> (<code>0.6</code> in <code>"think35"</code>, <code>1.0</code> in <code>dev/base+temp1.0</code>):</strong> Divide every logit by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi></mrow><annotation encoding="application/x-tex">T</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span></span></span></span> (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>v</mi></msub><mo>←</mo><msub><mi>z</mi><mi>v</mi></msub><mi mathvariant="normal">/</mi><mi>T</mi></mrow><annotation encoding="application/x-tex">z_v \leftarrow z_v / T</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">←</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0359em;">v</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord">/</span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span></span></span></span>). Dividing by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.6</mn></mrow><annotation encoding="application/x-tex">0.6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.6</span></span></span></span> multiplies logit gaps by <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mn>0.6</mn><mo>=</mo><mn>1.67</mn></mrow><annotation encoding="application/x-tex">1 / 0.6 = 1.67</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">1/0.6</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.67</span></span></span></span>, favoring the top token. Setting <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>1.0</mn></mrow><annotation encoding="application/x-tex">T = 1.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.0</span></span></span></span> leaves gaps unstretched, giving runner-up tokens a larger chance.</li> | |
| <li><strong>Top-k (<code>20</code>) and Top-p (<code>0.95</code>):</strong> Keep the <code>20</code> highest-logit tokens, compute their softmax probabilities, sort them, and drop the lowest-probability tokens whose cumulative probability is at most <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>−</mo><mn>0.95</mn><mo>=</mo><mn>0.05</mn></mrow><annotation encoding="application/x-tex">1 - 0.95 = 0.05</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">1</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.95</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.05</span></span></span></span> (always keeping the top token).</li> | |
| <li><strong>Renormalize and sample:</strong> Divide the kept probabilities by their sum so they add to <code>1.0</code>, then pick one token at random.</li> | |
| </ol> | |
| <h3 id="2-a-worked-example-why-token-penalties-fail-to-break-a-strong-loop">2. A worked example: why token penalties fail to break a strong loop<a class="anchor" href="#2-a-worked-example-why-token-penalties-fail-to-break-a-strong-loop" aria-hidden="true" tabindex="-1">#</a></h3> | |
| <p>Suppose the model is inside a loop and about to repeat <code>"Let's"</code> (using made-up numbers):</p> | |
| <ul> | |
| <li><code>"Let's"</code> has raw logit <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mtext>loop</mtext></msub><mo>=</mo><mn>12.0</mn></mrow><annotation encoding="application/x-tex">z_{\text{loop}} = 12.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7167em;vertical-align:-0.2861em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">loop</span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.2861em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">12.0</span></span></span></span>, and <code>"</think>"</code> (not yet in the answer) has raw logit <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mtext>close</mtext></msub><mo>=</mo><mn>5.0</mn></mrow><annotation encoding="application/x-tex">z_{\text{close}} = 5.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.5806em;vertical-align:-0.15em;"></span><span class="mord"><span class="mord mathnormal" style="margin-right:0.044em;">z</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3361em;"><span style="top:-2.55em;margin-left:-0.044em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">close</span></span></span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">5.0</span></span></span></span>.</li> | |
| <li><code>"Let's"</code> and the other scratchpad words have already appeared in the answer.</li> | |
| </ul> | |
| <div class="table-wrap"><table> | |
| <thead> | |
| <tr> | |
| <th>Setting</th> | |
| <th>Logit of <code class="cell"><span>"Let's"</span></code> after penalties</th> | |
| <th>Logit of <code class="cell"><span>"</think>"</span></code> after penalties</th> | |
| <th>Gap after dividing by temperature <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi></mrow><annotation encoding="application/x-tex">T</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span></span></span></span></th> | |
| <th>Odds ratio <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mtext>gap</mtext></msup></mrow><annotation encoding="application/x-tex">e^{\text{gap}}</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6644em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.6644em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">gap</span></span></span></span></span></span></span></span></span></span></span></span></span> (<code class="cell"><span>"Let's"</span></code> vs <code class="cell"><span>"</think>"</span></code>)</th> | |
| </tr> | |
| </thead> | |
| <tbody> | |
| <tr> | |
| <td><code class="cell"><span>presence_penalty = 0</span></code>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>0.6</mn></mrow><annotation encoding="application/x-tex">T = 0.6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.6</span></span></span></span> (<code class="cell"><span>dev/base+pp0</span></code>)</td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>12.0</mn></mrow><annotation encoding="application/x-tex">12.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">12.0</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>5.0</mn></mrow><annotation encoding="application/x-tex">5.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">5.0</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>12.0</mn><mo>−</mo><mn>5.0</mn><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mn>0.6</mn><mo>=</mo><mn>11.67</mn></mrow><annotation encoding="application/x-tex">(12.0 - 5.0) / 0.6 = 11.67</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord">12.0</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">5.0</span><span class="mclose">)</span><span class="mord">/0.6</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">11.67</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>11.67</mn></msup><mo>≈</mo><mn>117</mn><mo separator="true">,</mo><mn>000</mn><mo>:</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">e^{11.67} \approx 117,000 : 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">11.67</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">117</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">000</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span></td> | |
| </tr> | |
| <tr> | |
| <td>Paper settings: <code class="cell"><span>presence_penalty = 1.5</span></code>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>0.6</mn></mrow><annotation encoding="application/x-tex">T = 0.6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.6</span></span></span></span> (<code class="cell"><span>dev/base</span></code>)</td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>12.0</mn><mo>−</mo><mn>1.5</mn><mo>=</mo><mn>10.5</mn></mrow><annotation encoding="application/x-tex">12.0 - 1.5 = 10.5</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">12.0</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.5</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">10.5</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>5.0</mn></mrow><annotation encoding="application/x-tex">5.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">5.0</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>10.5</mn><mo>−</mo><mn>5.0</mn><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mn>0.6</mn><mo>=</mo><mn>9.17</mn></mrow><annotation encoding="application/x-tex">(10.5 - 5.0) / 0.6 = 9.17</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord">10.5</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">5.0</span><span class="mclose">)</span><span class="mord">/0.6</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">9.17</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>9.17</mn></msup><mo>≈</mo><mn>9</mn><mo separator="true">,</mo><mn>600</mn><mo>:</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">e^{9.17} \approx 9,600 : 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">9.17</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">9</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">600</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>presence_penalty = 2.0</span></code>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>0.6</mn></mrow><annotation encoding="application/x-tex">T = 0.6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.6</span></span></span></span> (<code class="cell"><span>dev/base+pp2</span></code>)</td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>12.0</mn><mo>−</mo><mn>2.0</mn><mo>=</mo><mn>10.0</mn></mrow><annotation encoding="application/x-tex">12.0 - 2.0 = 10.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">12.0</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">2.0</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">10.0</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>5.0</mn></mrow><annotation encoding="application/x-tex">5.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">5.0</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>10.0</mn><mo>−</mo><mn>5.0</mn><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mn>0.6</mn><mo>=</mo><mn>8.33</mn></mrow><annotation encoding="application/x-tex">(10.0 - 5.0) / 0.6 = 8.33</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord">10.0</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">5.0</span><span class="mclose">)</span><span class="mord">/0.6</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">8.33</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>8.33</mn></msup><mo>≈</mo><mn>4</mn><mo separator="true">,</mo><mn>150</mn><mo>:</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">e^{8.33} \approx 4,150 : 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">8.33</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">4</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">150</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>presence_penalty = 1.5</span></code>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>1.0</mn></mrow><annotation encoding="application/x-tex">T = 1.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.0</span></span></span></span> (<code class="cell"><span>dev/base+temp1.0</span></code>)</td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>12.0</mn><mo>−</mo><mn>1.5</mn><mo>=</mo><mn>10.5</mn></mrow><annotation encoding="application/x-tex">12.0 - 1.5 = 10.5</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.7278em;vertical-align:-0.0833em;"></span><span class="mord">12.0</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.5</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">10.5</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>5.0</mn></mrow><annotation encoding="application/x-tex">5.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">5.0</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>10.5</mn><mo>−</mo><mn>5.0</mn><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mn>1.0</mn><mo>=</mo><mn>5.50</mn></mrow><annotation encoding="application/x-tex">(10.5 - 5.0) / 1.0 = 5.50</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord">10.5</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">5.0</span><span class="mclose">)</span><span class="mord">/1.0</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">5.50</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>5.50</mn></msup><mo>≈</mo><mn>245</mn><mo>:</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">e^{5.50} \approx 245 : 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">5.50</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">245</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span></td> | |
| </tr> | |
| <tr> | |
| <td><code class="cell"><span>presence_penalty = 1.5</span></code>, <code class="cell"><span>repetition_penalty = 1.1</span></code>, <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>T</mi><mo>=</mo><mn>0.6</mn></mrow><annotation encoding="application/x-tex">T = 0.6</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6833em;"></span><span class="mord mathnormal" style="margin-right:0.1389em;">T</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.6</span></span></span></span> (<code class="cell"><span>dev/base+rep1.1</span></code>)</td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>12.0</mn><mi mathvariant="normal">/</mi><mn>1.1</mn><mo stretchy="false">)</mo><mo>−</mo><mn>1.5</mn><mo>=</mo><mn>9.41</mn></mrow><annotation encoding="application/x-tex">(12.0 / 1.1) - 1.5 = 9.41</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord">12.0/1.1</span><span class="mclose">)</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.5</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">9.41</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>5.0</mn></mrow><annotation encoding="application/x-tex">5.0</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">5.0</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>9.41</mn><mo>−</mo><mn>5.0</mn><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mn>0.6</mn><mo>=</mo><mn>7.35</mn></mrow><annotation encoding="application/x-tex">(9.41 - 5.0) / 0.6 = 7.35</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mopen">(</span><span class="mord">9.41</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:1em;vertical-align:-0.25em;"></span><span class="mord">5.0</span><span class="mclose">)</span><span class="mord">/0.6</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">7.35</span></span></span></span></td> | |
| <td><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mn>7.35</mn></msup><mo>≈</mo><mn>1</mn><mo separator="true">,</mo><mn>550</mn><mo>:</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">e^{7.35} \approx 1,550 : 1</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8141em;"></span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t"><span class="vlist-r"><span class="vlist" style="height:0.8141em;"><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord mtight">7.35</span></span></span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">≈</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.8389em;vertical-align:-0.1944em;"></span><span class="mord">1</span><span class="mpunct">,</span><span class="mspace" style="margin-right:0.1667em;"></span><span class="mord">550</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">:</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1</span></span></span></span></td> | |
| </tr> | |
| </tbody> | |
| </table></div> | |
| <p>This arithmetic explains three patterns in the data:</p> | |
| <ol> | |
| <li><strong>Why presence penalty does not grow as the loop repeats:</strong> <code>"Let's"</code> appeared in the first paragraph, so it pays <code>-1.5</code> on cycle 1 and the same <code>-1.5</code> on cycle 200. Normal math words (<code>"Therefore"</code>, <code>"answer"</code>, <code>"total"</code>, <code>"240"</code>) also appeared earlier in the scratchpad, so each pays <code>-1.5</code> too, leaving the gap between <code>"Let's"</code> and <code>"Therefore"</code> unchanged.</li> | |
| <li><strong>Why <code>repetition_penalty = 1.1</code> does better (<code>34.4%</code>) than <code>presence_penalty = 2.0</code> (<code>16.2%</code>):</strong> Dividing a positive logit by <code>1.1</code> takes more from a large logit (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>12.0</mn><mo>→</mo><mn>10.91</mn></mrow><annotation encoding="application/x-tex">12.0 \to 10.91</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">12.0</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">→</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">10.91</span></span></span></span>, losing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1.09</mn></mrow><annotation encoding="application/x-tex">1.09</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.09</span></span></span></span> on top of the <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1.5</mn></mrow><annotation encoding="application/x-tex">1.5</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">1.5</span></span></span></span> presence penalty) than from a moderate runner-up (<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>8.0</mn><mo>→</mo><mn>7.27</mn></mrow><annotation encoding="application/x-tex">8.0 \to 7.27</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">8.0</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">→</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">7.27</span></span></span></span>, losing <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.73</mn></mrow><annotation encoding="application/x-tex">0.73</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.6444em;"></span><span class="mord">0.73</span></span></span></span>). Even so, <code>61.7%</code> of <code>dev/base+rep1.1</code> answers hit the cap.</li> | |
| <li><strong>How penalties might prevent some loops yet fail to break others:</strong> When looping and productive continuations have similar logits, higher temperature (<code>1.0</code>) or <code>repetition_penalty = 1.1</code> can tip the choice away from the first repeat. Once repeated cycles fill the context, loop-token logits can grow high enough that <code>top_p = 0.95</code> drops every non-loop token to probability zero. Lab 6 Part 3 measures the per-token pick probability inside 48 real loops: <code>0.99995</code> in the unedited model vs <code>0.979</code> with the weight edit.</li> | |
| </ol> | |
| <h3 id="3-why-turning-think-off-scores-521-and-why-the-edit-adds-nothing-there">3. Why turning <code><think></code> off scores 52.1%, and why the edit adds nothing there<a class="anchor" href="#3-why-turning-think-off-scores-521-and-why-the-edit-adds-nothing-there" aria-hidden="true" tabindex="-1">#</a></h3> | |
| <p>Switching from <code>"think35"</code> to <code>"nothink35"</code> (<code>dev/base+nothink</code>) raises <code>dev256</code> accuracy from <code>12.1%</code> to <code>52.1%</code> and cuts mean length from <code>29,683</code> tokens to <code>2,270</code> (median length on correct answers is <code>452</code> tokens).</p> | |
| <figure class="mermaid-svg" role="img" aria-label="Diagram: 3. Why turning <think> off scores 52.1%, and why the edit adds nothing there" style="--mw:657px"><svg id="m-69f15a42968f" width="100%" xmlns="http://www.w3.org/2000/svg" class="flowchart" style="max-width: 876px;" viewBox="0 0 876 814" role="graphics-document document" aria-roledescription="flowchart-v2"><style>#m-69f15a42968f{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#m-69f15a42968f .edge-animation-slow{stroke-dasharray:9,5;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#m-69f15a42968f .edge-animation-fast{stroke-dasharray:9,5;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#m-69f15a42968f .error-icon{fill:#552222;}#m-69f15a42968f .error-text{fill:#552222;stroke:#552222;}#m-69f15a42968f .edge-thickness-normal{stroke-width:1px;}#m-69f15a42968f .edge-thickness-thick{stroke-width:3.5px;}#m-69f15a42968f .edge-pattern-solid{stroke-dasharray:0;}#m-69f15a42968f .edge-thickness-invisible{stroke-width:0;fill:none;}#m-69f15a42968f .edge-pattern-dashed{stroke-dasharray:3;}#m-69f15a42968f .edge-pattern-dotted{stroke-dasharray:2;}#m-69f15a42968f .marker{fill:#666;stroke:#666;}#m-69f15a42968f .marker.cross{stroke:#666;}#m-69f15a42968f svg{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:16px;}#m-69f15a42968f p{margin:0;}#m-69f15a42968f .label{font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;color:#000000;}#m-69f15a42968f .cluster-label text{fill:#333;}#m-69f15a42968f .cluster-label span{color:#333;}#m-69f15a42968f .cluster-label span p{background-color:transparent;}#m-69f15a42968f .label text,#m-69f15a42968f span{fill:#000000;color:#000000;}#m-69f15a42968f .node rect,#m-69f15a42968f .node circle,#m-69f15a42968f .node ellipse,#m-69f15a42968f .node polygon,#m-69f15a42968f .node path{fill:#eee;stroke:#999;stroke-width:1px;}#m-69f15a42968f .rough-node .label text,#m-69f15a42968f .node .label text,#m-69f15a42968f .image-shape .label,#m-69f15a42968f .icon-shape .label{text-anchor:middle;}#m-69f15a42968f .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#m-69f15a42968f .rough-node .label,#m-69f15a42968f .node .label,#m-69f15a42968f .image-shape .label,#m-69f15a42968f .icon-shape .label{text-align:center;}#m-69f15a42968f .node.clickable{cursor:pointer;}#m-69f15a42968f .root .anchor path{fill:#666;stroke-width:0;stroke:#666;}#m-69f15a42968f .arrowheadPath{fill:#333333;}#m-69f15a42968f .edgePaths .path{stroke:#666;stroke-width:1px;}#m-69f15a42968f .flowchart-link{stroke:#666;fill:none;}#m-69f15a42968f .edgeLabel{background-color:white;text-align:center;}#m-69f15a42968f .edgeLabel p{background-color:white;}#m-69f15a42968f .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#m-69f15a42968f .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#m-69f15a42968f .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#m-69f15a42968f .cluster text{fill:#333;}#m-69f15a42968f .cluster span{color:#333;}#m-69f15a42968f .node .collapsed-indicator{fill:#707070;stroke:none;opacity:0.6;}#m-69f15a42968f .node .collapsed-separator{stroke:#707070;stroke-width:0.75px;}#m-69f15a42968f div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#m-69f15a42968f .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#m-69f15a42968f rect.text{fill:none;stroke-width:0;}#m-69f15a42968f .icon-shape,#m-69f15a42968f .image-shape{background-color:white;text-align:center;}#m-69f15a42968f .icon-shape p,#m-69f15a42968f .image-shape p{background-color:white;padding:2px;}#m-69f15a42968f .icon-shape .label rect,#m-69f15a42968f .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#m-69f15a42968f .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#m-69f15a42968f .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#m-69f15a42968f .node .neo-node{stroke:#999;}#m-69f15a42968f [data-look="neo"].node rect,#m-69f15a42968f [data-look="neo"].cluster rect,#m-69f15a42968f [data-look="neo"].node polygon{stroke:url(#m-69f15a42968f-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-69f15a42968f [data-look="neo"].swimlane.cluster rect{filter:none;}#m-69f15a42968f [data-look="neo"].node path{stroke:url(#m-69f15a42968f-gradient);stroke-width:1px;}#m-69f15a42968f [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-69f15a42968f [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#m-69f15a42968f [data-look="neo"].node circle{stroke:url(#m-69f15a42968f-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-69f15a42968f [data-look="neo"].node circle .state-start{fill:#000000;}#m-69f15a42968f [data-look="neo"].icon-shape .icon{fill:url(#m-69f15a42968f-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-69f15a42968f [data-look="neo"].icon-shape .icon-neo path{stroke:url(#m-69f15a42968f-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#m-69f15a42968f :root{--mermaid-font-family:Geist,Inter,ui-sans-serif,system-ui,sans-serif;}</style><g><marker id="m-69f15a42968f_flowchart-v2-pointEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 0 L 10 5 L 0 10 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-69f15a42968f_flowchart-v2-pointStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="4.5" refY="5" markerUnits="userSpaceOnUse" markerWidth="8" markerHeight="8" orient="auto"><path d="M 0 5 L 10 10 L 10 0 z" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></path></marker><marker id="m-69f15a42968f_flowchart-v2-pointEnd-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="11.5" refY="7" markerUnits="userSpaceOnUse" markerWidth="10.5" markerHeight="14" orient="auto"><path d="M 0 0 L 11.5 7 L 0 14 z" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></path></marker><marker id="m-69f15a42968f_flowchart-v2-pointStart-margin" class="marker flowchart-v2" viewBox="0 0 11.5 14" refX="1" refY="7" markerUnits="userSpaceOnUse" markerWidth="11.5" markerHeight="14" orient="auto"><polygon points="0,7 11.5,14 11.5,0" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></polygon></marker><marker id="m-69f15a42968f_flowchart-v2-circleEnd" class="marker flowchart-v2" viewBox="0 0 10 10" refX="11" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-69f15a42968f_flowchart-v2-circleStart" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-1" refY="5" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 1; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-69f15a42968f_flowchart-v2-circleEnd-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refY="5" refX="12.25" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-69f15a42968f_flowchart-v2-circleStart-margin" class="marker flowchart-v2" viewBox="0 0 10 10" refX="-2" refY="5" markerUnits="userSpaceOnUse" markerWidth="14" markerHeight="14" orient="auto"><circle cx="5" cy="5" r="5" class="arrowMarkerPath" style="stroke-width: 0; stroke-dasharray: 1, 0;"></circle></marker><marker id="m-69f15a42968f_flowchart-v2-crossEnd" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="12" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-69f15a42968f_flowchart-v2-crossStart" class="marker cross flowchart-v2" viewBox="0 0 11 11" refX="-1" refY="5.2" markerUnits="userSpaceOnUse" markerWidth="11" markerHeight="11" orient="auto"><path d="M 1,1 l 9,9 M 10,1 l -9,9" class="arrowMarkerPath" style="stroke-width: 2; stroke-dasharray: 1, 0;"></path></marker><marker id="m-69f15a42968f_flowchart-v2-crossEnd-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="17.7" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5;"></path></marker><marker id="m-69f15a42968f_flowchart-v2-crossStart-margin" class="marker cross flowchart-v2" viewBox="0 0 15 15" refX="-3.5" refY="7.5" markerUnits="userSpaceOnUse" markerWidth="12" markerHeight="12" orient="auto"><path d="M 1,1 L 14,14 M 1,14 L 14,1" class="arrowMarkerPath" style="stroke-width: 2.5; stroke-dasharray: 1, 0;"></path></marker><g class="root"><g class="clusters"></g><g class="edgePaths"><path d="M145.959,374L167.132,342.167C188.306,310.333,230.653,246.667,255.326,214.833C280,183,287,183,290.5,183L294,183" id="m-69f15a42968f-L_Q_TOn_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_Q_TOn_0" data-points="W3sieCI6MTQ1Ljk1ODcxNTU5NjMzMDI4LCJ5IjozNzR9LHsieCI6MjczLCJ5IjoxODN9LHsieCI6Mjk4LCJ5IjoxODN9XQ==" data-look="classic" marker-end="url(#m-69f15a42968f_flowchart-v2-pointEnd)"></path><path d="M147.005,428L168.004,457.833C189.003,487.667,231.002,547.333,255.501,577.167C280,607,287,607,290.5,607L294,607" id="m-69f15a42968f-L_Q_TOff_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_Q_TOff_0" data-points="W3sieCI6MTQ3LjAwNDg1NDM2ODkzMjAzLCJ5Ijo0Mjh9LHsieCI6MjczLCJ5Ijo2MDd9LHsieCI6Mjk4LCJ5Ijo2MDd9XQ==" data-look="classic" marker-end="url(#m-69f15a42968f_flowchart-v2-pointEnd)"></path><path d="M525.65,120L535.208,113.833C544.767,107.667,563.883,95.333,576.942,89.167C590,83,597,83,600.5,83L604,83" id="m-69f15a42968f-L_TOn_BaseOn_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_TOn_BaseOn_0" data-points="W3sieCI6NTI1LjY1LCJ5IjoxMjB9LHsieCI6NTgzLCJ5Ijo4M30seyJ4Ijo2MDgsInkiOjgzfV0=" data-look="classic" marker-end="url(#m-69f15a42968f_flowchart-v2-pointEnd)"></path><path d="M525.65,246L535.208,252.167C544.767,258.333,563.883,270.667,576.942,276.833C590,283,597,283,600.5,283L604,283" id="m-69f15a42968f-L_TOn_EditOn_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_TOn_EditOn_0" data-points="W3sieCI6NTI1LjY1LCJ5IjoyNDZ9LHsieCI6NTgzLCJ5IjoyODN9LHsieCI6NjA4LCJ5IjoyODN9XQ==" data-look="classic" marker-end="url(#m-69f15a42968f_flowchart-v2-pointEnd)"></path><path d="M515.188,544L526.49,535.833C537.792,527.667,560.396,511.333,575.198,503.167C590,495,597,495,600.5,495L604,495" id="m-69f15a42968f-L_TOff_BaseOff_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_TOff_BaseOff_0" data-points="W3sieCI6NTE1LjE4NzUsInkiOjU0NH0seyJ4Ijo1ODMsInkiOjQ5NX0seyJ4Ijo2MDgsInkiOjQ5NX1d" data-look="classic" marker-end="url(#m-69f15a42968f_flowchart-v2-pointEnd)"></path><path d="M515.188,670L526.49,678.167C537.792,686.333,560.396,702.667,575.198,710.833C590,719,597,719,600.5,719L604,719" id="m-69f15a42968f-L_TOff_EditOff_0" class="edge-thickness-normal edge-pattern-solid edge-thickness-normal edge-pattern-solid flowchart-link" style=";" data-edge="true" data-et="edge" data-id="L_TOff_EditOff_0" data-points="W3sieCI6NTE1LjE4NzUsInkiOjY3MH0seyJ4Ijo1ODMsInkiOjcxOX0seyJ4Ijo2MDgsInkiOjcxOX1d" data-look="classic" marker-end="url(#m-69f15a42968f_flowchart-v2-pointEnd)"></path></g><g class="edgeLabels"><g class="edgeLabel"><g class="label" data-id="L_Q_TOn_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_Q_TOff_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_TOn_BaseOn_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_TOn_EditOn_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_TOff_BaseOff_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g><g class="edgeLabel"><g class="label" data-id="L_TOff_EditOff_0" transform="translate(0, 0)"><foreignObject width="0" height="0"><div xmlns="http://www.w3.org/1999/xhtml" class="labelBkg" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="edgeLabel"></span></div></foreignObject></g></g></g><g class="nodes"><g class="node default" id="m-69f15a42968f-flowchart-Q-0" data-look="classic" transform="translate(128, 401)"><rect class="basic label-container" style="" x="-120" y="-27" width="240" height="54"></rect><g class="label" style="" transform="translate(-90, -12)"><rect></rect><foreignObject width="180" height="24"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table-cell; white-space: nowrap; line-height: 1.5; max-width: 200px; text-align: center;"><span class="nodeLabel"><p>Math problem in dev256</p></span></div></foreignObject></g></g><g class="node default" id="m-69f15a42968f-flowchart-TOn-1" data-look="classic" transform="translate(428, 183)"><rect class="basic label-container" style="" x="-130" y="-63" width="260" height="126"></rect><g class="label" style="" transform="translate(-100, -48)"><rect></rect><foreignObject width="200" height="96"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Thinking ON (think35 preset)<br>Open-ended self-checking scratchpad</p></span></div></foreignObject></g></g><g class="node default" id="m-69f15a42968f-flowchart-TOff-3" data-look="classic" transform="translate(428, 607)"><rect class="basic label-container" style="" x="-130" y="-63" width="260" height="126"></rect><g class="label" style="" transform="translate(-100, -48)"><rect></rect><foreignObject width="200" height="96"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Thinking OFF (nothink35 preset)<br>Direct step-by-step solution</p></span></div></foreignObject></g></g><g class="node default" id="m-69f15a42968f-flowchart-BaseOn-5" data-look="classic" transform="translate(738, 83)"><rect class="basic label-container" style="" x="-130" y="-75" width="260" height="150"></rect><g class="label" style="" transform="translate(-100, -60)"><rect></rect><foreignObject width="200" height="120"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Unedited (dev/base):<br>88.7% cut (96.9% of those in repetition)<br>12.1% acc (GSM8K 15.2%, MATH 9.0%)</p></span></div></foreignObject></g></g><g class="node default" id="m-69f15a42968f-flowchart-EditOn-7" data-look="classic" transform="translate(738, 283)"><rect class="basic label-container" style="" x="-130" y="-75" width="260" height="150"></rect><g class="label" style="" transform="translate(-100, -60)"><rect></rect><foreignObject width="200" height="120"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Edited (dev/p50c):<br>loops suppressed -> 61.5% acc<br>(GSM8K 74.6%, MATH 48.4%)</p></span></div></foreignObject></g></g><g class="node default" id="m-69f15a42968f-flowchart-BaseOff-9" data-look="classic" transform="translate(738, 495)"><rect class="basic label-container" style="" x="-130" y="-87" width="260" height="174"></rect><g class="label" style="" transform="translate(-100, -72)"><rect></rect><foreignObject width="200" height="144"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Unedited (dev/base+nothink):<br>few loops (7.6% of cuts) -> 52.1% acc<br>(GSM8K 58.2%, MATH 46.1%)</p></span></div></foreignObject></g></g><g class="node default" id="m-69f15a42968f-flowchart-EditOff-11" data-look="classic" transform="translate(738, 719)"><rect class="basic label-container" style="" x="-130" y="-87" width="260" height="174"></rect><g class="label" style="" transform="translate(-100, -72)"><rect></rect><foreignObject width="200" height="144"><div xmlns="http://www.w3.org/1999/xhtml" style="display: table; white-space: break-spaces; line-height: 1.5; max-width: 200px; text-align: center; width: 200px;"><span class="nodeLabel"><p>Edited (dev/p50c+nothink):<br>+1.4 +/- 1.9 pp -> 53.5% acc<br>(GSM8K 62.1%, MATH 44.9%)</p></span></div></foreignObject></g></g></g></g></g><defs><filter id="m-69f15a42968f-drop-shadow" height="130%" width="130%"><feDropShadow dx="4" dy="4" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><defs><filter id="m-69f15a42968f-drop-shadow-small" height="150%" width="150%"><feDropShadow dx="2" dy="2" stdDeviation="0" flood-opacity="0.06" flood-color="#000000"></feDropShadow></filter></defs><linearGradient id="m-69f15a42968f-gradient" gradientUnits="objectBoundingBox" x1="0%" y1="0%" x2="100%" y2="0%"><stop offset="0%" stop-color="hsl(0, 0%, 83.3333333333%)" stop-opacity="1"></stop><stop offset="100%" stop-color="hsl(0, 0%, 88.9215686275%)" stop-opacity="1"></stop></linearGradient></svg></figure> | |
| <ol> | |
| <li><strong>The weight edit helps only when loops occur.</strong> With thinking off (<code>dev/base+nothink</code>), <code>18.0%</code> of answers hit the 8,192-token cap, and <code>7.6%</code> of those cut answers end in repetition (<code>5.4%</code> in a verbatim loop). Few answers loop, so with the weight edit and thinking off (<code>dev/p50c+nothink</code>) the paired difference against <code>dev/base+nothink</code> is <code>+1.4 +/- 1.9</code> percentage points (<code>22</code> problems rescued, <code>26</code> lost), which is indistinguishable from zero.</li> | |
| <li><strong>Once loops are repaired, the thinking preset beats the non-thinking preset, mostly on word problems.</strong> Holding the edited weights fixed, <code>dev/p50c</code> (<code>think35</code>, <code>61.5%</code>) beats <code>dev/p50c+nothink</code> (<code>nothink35</code>, <code>53.5%</code>) by <code>+8.0 +/- 2.0</code> paired points (<code>27</code> problems rescued, <code>11</code> lost). Most of that gain comes from GSM8K (<code>74.6%</code> vs <code>62.1%</code>, <code>+12.5 +/- 2.8</code> points), while the MATH gap is small (<code>48.4%</code> vs <code>44.9%</code>, <code>+3.5 +/- 2.9</code> points). The thinking preset uses about seven times as many tokens (<code>16,104</code> vs <code>2,306</code> on average). The two presets also differ in sampling settings (<code>think35</code> uses temperature <code>0.6</code>, top-p <code>0.95</code>, and a <code>32,768</code>-token cap; <code>nothink35</code> uses temperature <code>0.7</code>, top-p <code>0.8</code>, and an <code>8,192</code>-token cap), so part of the gap could come from sampling changes instead of thinking itself.</li> | |
| </ol> | |
| <h3 id="4-the-papers-gemma-3-270m-follow-up-points-the-same-way">4. The paper's Gemma 3 270M follow-up points the same way<a class="anchor" href="#4-the-papers-gemma-3-270m-follow-up-points-the-same-way" aria-hidden="true" tabindex="-1">#</a></h3> | |
| <p>In Theory Guide section 10 (and <code>external/lm_understand-logs/exp8_math/gemma_followup/README.md</code>), the authors ran the same weight-space programming setup on a different small model, Gemma 3 270M. On MATH-500, unedited Gemma 3 270M scored <code>6.8%</code> and hit the token cap on <code>8.5%</code> of answers (<code>91.5%</code> finished, yet only <code>6.8%</code> of all answers were right).</p> | |
| <p>Across multiple agent runs, no weight edit moved Gemma 3 270M's MATH-500 score outside <code>6.8%</code> to <code>7.7%</code>. As the blog puts it: <em>"Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."</em> That matches <code>dev/p50c+nothink</code>: when answers do not get trapped in loops, subtracting the loop direction <code>u</code> does not make the model better at math.</p> | |
| <h2 id="check-yourself">Check yourself<a class="anchor" href="#check-yourself" aria-hidden="true" tabindex="-1">#</a></h2> | |
| <ol> | |
| <li>Why did removing the presence penalty (<code>dev/base+pp0</code>) hurt accuracy (<code>12.1%</code> down to <code>4.5%</code>), whereas raising it from <code>1.5</code> to <code>2.0</code> (<code>dev/base+pp2</code>) barely helped (<code>12.1%</code> to <code>16.2%</code>)?</li> | |
| </ol> | |
| <details><summary>Answer</summary> | |
| <p>At <code>presence_penalty = 0</code>, no penalty discourages reusing earlier output words: <code>99.0%</code> of answers hit the cap (<code>98.4%</code> of those end in repetition), and only <code>4</code> of <code>512</code> close normally. With presence penalty <code>1.5</code>, already-used output tokens get a <code>-1.5</code> logit penalty, and <code>56</code> answers close normally (<code>62</code> correct in all, <code>12.1%</code> accuracy). Raising the flat penalty from <code>1.5</code> to <code>2.0</code> adds <code>-0.5</code> more, which lowers already-used looping tokens and already-used reasoning tokens by the same amount.</p> | |
| </details> | |
| <ol start="2"> | |
| <li>In <code>results/summary.md</code>, <code>dev/base+nothink</code> shows a <code>+40.0 +/- 2.5</code> percentage-point gain, while <code>dev/p50c+nothink</code> shows <code>+1.4 +/- 1.9</code> percentage points even though its accuracy (<code>53.5%</code>) exceeds <code>dev/base+nothink</code> (<code>52.1%</code>). Why do the two gains differ?</li> | |
| </ol> | |
| <details><summary>Answer</summary> | |
| <p><code>scripts/summarize.py</code> pairs each run against the baseline that isolates one change (<code>baseline(name)</code> in <code>scripts/summarize.py</code>). <code>dev/base+nothink</code> uses <code>dev/base</code> (<code>52.1% - 12.1% = +40.0</code> points, the effect of turning thinking off on the unedited model). <code>dev/p50c+nothink</code> changes both the model and the preset, so <code>scripts/summarize.py</code> pairs it against <code>dev/base+nothink</code> (<code>53.5% - 52.1% = +1.4</code> points, the effect of the weight edit when thinking is already off).</p> | |
| </details> | |
| <ol start="3"> | |
| <li>If a teammate suggests saving compute on the unedited thinking model by lowering <code>max_tokens</code> from <code>32,768</code> to <code>4,096</code> (<code>dev/base+max4k</code>), what do the numbers show?</li> | |
| </ol> | |
| <details><summary>Answer</summary> | |
| <p>Mean tokens drop by a factor of 7.4 (<code>29,683</code> to <code>4,019</code>), and accuracy falls in half (<code>12.1%</code> to <code>5.9%</code>, losing <code>30</code> problems and rescuing <code>3</code>). Because the unedited model's correct thinking answers have a <code>5,600</code>-token median length, a <code>4,096</code>-token cap cuts off over half of the answers that would have finished and solved the problem. For short answers without weight edits, turning <code><think></code> off (<code>dev/base+nothink</code>: <code>2,270</code> mean tokens, <code>52.1%</code> accuracy) works much better than truncating <code><think></code> at <code>4,096</code> tokens.</p> | |
| </details> | |
| <ol start="4"> | |
| <li>In vLLM's sampler (<code>wsp/protocol.py</code>), does <code>presence_penalty</code> get applied before or after dividing by <code>temperature</code>? Why does that order matter when <code>temperature = 0.6</code>?</li> | |
| </ol> | |
| <details><summary>Answer</summary> | |
| <p><code>presence_penalty</code> is subtracted before dividing by <code>temperature</code>: <code>z = (logits - presence_penalty * present) / temperature</code>. Because <code>temperature = 0.6</code> divides the penalized logits by <code>0.6</code>, a presence penalty of <code>1.5</code> subtracts <code>1.5 / 0.6 = 2.5</code> from the post-temperature logit of every token already in the answer. When <code>dev/base+temp1.0</code> raises temperature from <code>0.6</code> to <code>1.0</code>, that effective penalty drops from <code>2.5</code> to <code>1.5</code>, and every post-temperature logit gap becomes <code>0.6</code> times as large for the same raw logits and token history. The raw logits are unchanged.</p> | |
| </details> | |
| <h2 id="try-this-next">Try this next<a class="anchor" href="#try-this-next" aria-hidden="true" tabindex="-1">#</a></h2> | |
| <ol> | |
| <li><strong>Compare how <code>gsm8k_train_5628</code> ends across four decoding settings and the edited <code>dev/p50c</code> on <code>sampler</code>.</strong> Run <code>labs/peek.py</code> (CPU only, a few seconds) on the Mr. Desmond problem (<code>truth: 240</code>) across <code>dev/base</code>, <code>dev/base+temp1.0</code>, <code>dev/base+rep1.1</code>, <code>dev/base+nothink</code>, and <code>dev/p50c</code>:<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">ssh sampler | |
| . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code | |
| export CUDA_VISIBLE_DEVICES="" | |
| for r in dev/base dev/base+temp1.0 dev/base+rep1.1 dev/base+nothink dev/p50c; do | |
| echo "=== $r ===" | |
| python labs/peek.py "$r" --id gsm8k_train_5628 --chars 600 | |
| done | |
| </code></pre></div> | |
| Compare the length of <code>dev/base+nothink</code> against <code>dev/p50c</code>, and check whether <code>dev/base+rep1.1</code> solved <code>gsm8k_train_5628</code> or looped.</li> | |
| <li><strong>Test a new decoding combination on 16 problems (not run by us; write down your prediction first).</strong> Combining the two best decoding tweaks on the unedited model (<code>temperature = 1.0</code> and <code>repetition_penalty = 1.1</code>), or combining the weight edit (<code>qwen35-p50c</code>) with <code>repetition_penalty = 1.1</code>, was not in <code>plans/sampler.json</code>. You can test both in about 5 minutes on 2 GPUs of <code>sampler</code> using <code>"args": ["--limit", "16"]</code> (as in <code>plans/quick.json</code>) and a <code>"try/"</code> prefix so <code>runs/dev/</code> stays untouched:<div class="code"><div class="code-head"><span class="code-lang">json</span></div><pre><code class="language-json">{ | |
| "about": "Quick 16-problem test of combined decoding settings.", | |
| "runs": [ | |
| { | |
| "name": "try/base+temp1.0+rep1.1", | |
| "model": "base", | |
| "data": "dev256", | |
| "n": 2, | |
| "shards": 1, | |
| "est": 1, | |
| "set": ["temperature=1.0", "repetition_penalty=1.1"], | |
| "args": ["--limit", "16"] | |
| }, | |
| { | |
| "name": "try/p50c+rep1.1", | |
| "model": "qwen35-p50c", | |
| "data": "dev256", | |
| "n": 2, | |
| "shards": 1, | |
| "est": 1, | |
| "set": ["repetition_penalty=1.1"], | |
| "args": ["--limit", "16"] | |
| } | |
| ] | |
| } | |
| </code></pre></div> | |
| Save this JSON on this machine as <code>plans/try_lab5.json</code>. From the project root on this machine, sync the plan, run it on <code>sampler</code>, and pull the results:<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">bash scripts/sync_vm.sh sampler | |
| ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5.json --gpus 0,1' </dev/null | |
| bash scripts/pull_runs.sh sampler | |
| </code></pre></div> | |
| The single shard runs the first <code>16</code> rows of <code>dev256</code> (all GSM8K problems, <code>2</code> answers each). The <code>try/</code> prefix gets no paired column from <code>scripts/summarize.py</code>, so compare the shared problem IDs directly on this machine:<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">python3 - <<'PY' | |
| from wsp.summary import compare, load_run | |
| for name in ("base+temp1.0+rep1.1", "p50c+rep1.1"): | |
| trial = load_run("results/runs/try/" + name) | |
| for baseline in ("base", "p50c"): | |
| print(name, "vs dev/" + baseline, | |
| compare(load_run("results/runs/dev/" + baseline), trial)) | |
| PY | |
| </code></pre></div> | |
| <strong>What we predict:</strong> Combining temperature <code>1.0</code> and repetition penalty <code>1.1</code> may reduce repetition further, though a gain is not guaranteed. On these <code>16</code> problems, our separate runs score <code>40.6%</code> (<code>13/32</code> answers) at temperature <code>1.0</code>, <code>46.9%</code> (<code>15/32</code>) with repetition penalty <code>1.1</code>, and <code>65.6%</code> (<code>21/32</code>) with <code>p50c</code>, with standard errors across problems of about <code>11</code> to <code>12</code> percentage points, so predictions from this small check are rough. Adding repetition penalty to <code>p50c</code> might reduce cuts, but it also penalizes repeated numbers and variables in valid solutions. Because this slice has no MATH problems, it cannot test hard MATH answers.</li> | |
| <li><strong>Test <code>nothink35</code> on the three full benchmarks (not run by us; write down your prediction first).</strong> We ran <code>nothink35</code> on <code>dev256</code> (<code>52.1%</code> unedited, <code>53.5%</code> edited), not on <code>math500</code>, <code>gsm8k</code>, or <code>ifeval</code>. Scaling from <code>dev/base+nothink</code> (<code>512</code> answers in <code>2.73</code> minutes on <code>1</code> GPU) by answer count gives a rough wall-clock estimate of <code>6</code> to <code>15</code> minutes per benchmark with one shard on <code>1</code> GPU (<code>1,082</code> IFEval, <code>2,000</code> MATH-500, or <code>2,638</code> GSM8K answers); answer lengths and throughput may differ. Save this plan on this machine as <code>plans/try_lab5_nothink.json</code> (matching the benchmark files and answer counts of the <code>bench/</code> runs in <code>plans/trainer.json</code>):<div class="code"><div class="code-head"><span class="code-lang">json</span></div><pre><code class="language-json">{ | |
| "about": "Thinking off (nothink35) on the three full benchmarks, unedited and edited.", | |
| "runs": [ | |
| {"name": "try/nothink/base/math500", "model": "base", "data": "math500", "n": 4, "shards": 1, "est": 5, "preset": "nothink35"}, | |
| {"name": "try/nothink/base/gsm8k", "model": "base", "data": "gsm8k", "n": 2, "shards": 1, "est": 6, "preset": "nothink35"}, | |
| {"name": "try/nothink/base/ifeval", "model": "base", "data": "ifeval", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"}, | |
| {"name": "try/nothink/p50c/math500", "model": "qwen35-p50c", "data": "math500", "n": 4, "shards": 1, "est": 5, "preset": "nothink35"}, | |
| {"name": "try/nothink/p50c/gsm8k", "model": "qwen35-p50c", "data": "gsm8k", "n": 2, "shards": 1, "est": 6, "preset": "nothink35"}, | |
| {"name": "try/nothink/p50c/ifeval", "model": "qwen35-p50c", "data": "ifeval", "n": 2, "shards": 1, "est": 2, "preset": "nothink35"} | |
| ] | |
| } | |
| </code></pre></div> | |
| Sync, run on six free GPUs (replace <code>0-5</code> with the free GPU indices), and pull the results:<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">bash scripts/sync_vm.sh sampler | |
| ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5_nothink.json --gpus 0-5' </dev/null | |
| bash scripts/pull_runs.sh sampler | |
| </code></pre></div> | |
| Print each score and the paired gain of the edit on this machine:<div class="code"><div class="code-head"><span class="code-lang">bash</span></div><pre><code class="language-bash">python3 - <<'PY' | |
| import json | |
| from wsp.summary import compare, load_run | |
| for d in ("gsm8k", "math500", "ifeval"): | |
| for m in ("base", "p50c"): | |
| s = json.load(open(f"results/runs/try/nothink/{m}/{d}/summary.json")) | |
| print(f"{m} {d}: {100 * s['acc']:.1f} +/- {100 * s['acc_se']:.1f}%") | |
| c = compare(load_run(f"results/runs/try/nothink/base/{d}"), load_run(f"results/runs/try/nothink/p50c/{d}")) | |
| print(f" p50c minus base, paired: {100 * c['gain']:+.1f} +/- {100 * c['gain_se']:.1f} points") | |
| PY | |
| </code></pre></div> | |
| <strong>What we predict:</strong> On GSM8K test and MATH-500, <code>base</code> with <code>preset: "nothink35"</code> should mirror the <code>dev256</code> split: about <code>55%</code> to <code>60%</code> on GSM8K (above <code>14.2%</code> unedited thinking, below <code>66.9%</code> edited thinking) and <code>42%</code> to <code>46%</code> on MATH-500 (near edited thinking's <code>51.8%</code>). On IFEval, which tests formatting constraints instead of long math derivations, <code>base</code> with <code>nothink35</code> should approach <code>bench/p50c/ifeval</code> (<code>63.4%</code>), and <code>p50c</code> with <code>nothink35</code> should match <code>base</code> with <code>nothink35</code> within noise.</li> | |
| </ol> | |
| </article> | |
| <nav class="pager" aria-label="Previous and next"><a class="prev" href="lab4_dose.html" rel="prev"><span>Previous</span><b>Lab 4: Turn the knob, and the controls</b></a><a class="next" href="lab6_steering.html" rel="next"><span>Next</span><b>Lab 6: The weight edit is steering in disguise</b></a></nav> | |
| </main> | |
| <aside class="toc"><nav aria-label="On this page"><p class="toc-title">On this page</p><ul><li class="l2"><a href="#before-you-run-predict">Before you run: predict</a></li><li class="l2"><a href="#run-it">Run it</a></li><li class="l2"><a href="#what-you-should-see">What you should see</a></li><li class="l3"><a href="#1-all-nine-runs-on-dev256-512-answers-per-run">1. All nine runs on <code>dev256</code> (<code>512</code> answers per run)</a></li><li class="l3"><a href="#2-breakdown-by-problem-source-gsm8k-vs-math-and-median-length-of-correct-answers">2. Breakdown by problem source (GSM8K vs MATH) and median length of correct answers</a></li><li class="l3"><a href="#3-how-every-thinking-mode-answer-ended-resultslab5endingsjson">3. How every thinking-mode answer ended (<code>results/lab5/endings.json</code>)</a></li><li class="l2"><a href="#why-it-works">Why it works</a></li><li class="l3"><a href="#1-decoding-from-first-principles-how-vllm-turns-logits-into-a-chosen-token">1. Decoding from first principles: how vLLM turns logits into a chosen token</a></li><li class="l3"><a href="#2-a-worked-example-why-token-penalties-fail-to-break-a-strong-loop">2. A worked example: why token penalties fail to break a strong loop</a></li><li class="l3"><a href="#3-why-turning-think-off-scores-521-and-why-the-edit-adds-nothing-there">3. Why turning <code><think></code> off scores 52.1%, and why the edit adds nothing there</a></li><li class="l3"><a href="#4-the-papers-gemma-3-270m-follow-up-points-the-same-way">4. The paper's Gemma 3 270M follow-up points the same way</a></li><li class="l2"><a href="#check-yourself">Check yourself</a></li><li class="l2"><a href="#try-this-next">Try this next</a></li></ul></nav></aside> | |
| </div> | |
| <footer class="foot"> | |
| <div class="foot-in"> | |
| <p class="foot-note">Based on Experiment 8 of "<a href="https://rikiyat.github.io/blog/weight-space-programming/" target="_blank" rel="noopener">Sparks of Weight-Space Programming</a>". Logs and the agent's texts come from <a href="https://github.com/RikiyaT/lm_understand-logs" target="_blank" rel="noopener">RikiyaT/lm_understand-logs</a>, <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" rel="noopener">CC BY 4.0</a>.</p> | |
| <p class="foot-links"><a href="https://huggingface.co/spaces/thegovind/weight-space-programming/blob/main/code/labs/lab5_decoding.md" target="_blank" rel="noopener">View source</a></p> | |
| </div> | |
| </footer> | |
| </body> | |
| </html> | |