| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <meta name="color-scheme" content="only light"> |
| <meta name="theme-color" content="#f5f5f5"> |
| <title>Results · Weight-space programming lab</title> |
| <meta name="description" content=""Cut answers ending in repetition" counts two kinds of ending among the answers cut at the cap. The first is a word-for-word loop (one block of words repeated back to back at least 3 times, over at…"> |
| <link rel="icon" href="../static/favicon.svg" type="image/svg+xml"> |
| <link rel="preload" href="../static/fonts/geist-latin-wght-normal.woff2" as="font" type="font/woff2" crossorigin> |
| <link rel="stylesheet" href="../static/site.css"> |
| <script>document.documentElement.classList.add("js")</script> |
| <script defer src="../static/site.js"></script> |
| </head> |
| <body data-copy="Copy" data-copied="Copied"> |
| <a class="skip" href="#main">Skip to content</a> |
| <header class="top"> |
| <div class="top-in"> |
| <a class="brand" href="../index.html">Weight-space programming lab</a> |
| <nav class="top-links" aria-label="Site"><a href="../index.html">Home</a><a href="../quickstart.html">Quick start</a><a href="../index.html#the-nine-labs">Labs</a><a href="../reference/theory-guide.html">Theory</a><a href="summary.html" aria-current="page">Results</a><a href="https://huggingface.co/spaces/thegovind/weight-space-programming/tree/main/code/" target="_blank" rel="noopener">Code</a></nav> |
| </div> |
| </header> |
| <div class="page solo"> |
| <main id="main" class="doc"> |
| <article class="prose"> |
| <h1 id="results">Results</h1> |
| <div class="table-wrap"><table> |
| <thead> |
| <tr> |
| <th>run</th> |
| <th>problems × samples</th> |
| <th>accuracy %</th> |
| <th>cut at the cap %</th> |
| <th>mean tokens</th> |
| <th>correct when finished %</th> |
| <th>cut answers ending in repetition %</th> |
| <th>gain vs baseline (pp)</th> |
| <th>rescued / lost problems</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td>bench/base/gsm8k</td> |
| <td>1319×2</td> |
| <td>14.2 ± 0.8</td> |
| <td>87.7</td> |
| <td>29,370</td> |
| <td>97.5</td> |
| <td>98.3</td> |
| <td>-</td> |
| <td>-</td> |
| </tr> |
| <tr> |
| <td>bench/base/ifeval</td> |
| <td>541×2</td> |
| <td>21.2 ± 1.4</td> |
| <td>73.5</td> |
| <td>24,976</td> |
| <td>78.7</td> |
| <td>96.1</td> |
| <td>-</td> |
| <td>-</td> |
| </tr> |
| <tr> |
| <td>bench/base/math500</td> |
| <td>500×4</td> |
| <td>7.3 ± 0.9</td> |
| <td>93.3</td> |
| <td>30,956</td> |
| <td>100.0</td> |
| <td>96.2</td> |
| <td>-</td> |
| <td>-</td> |
| </tr> |
| <tr> |
| <td>bench/p50c/gsm8k</td> |
| <td>1319×2</td> |
| <td>66.9 ± 1.2</td> |
| <td>6.3</td> |
| <td>11,742</td> |
| <td>71.3</td> |
| <td>29.7</td> |
| <td>+52.7 ± 1.1</td> |
| <td>711 / 2</td> |
| </tr> |
| <tr> |
| <td>bench/p50c/ifeval</td> |
| <td>541×2</td> |
| <td>63.4 ± 1.8</td> |
| <td>1.4</td> |
| <td>3,984</td> |
| <td>64.3</td> |
| <td>80.0</td> |
| <td>+42.2 ± 2.0</td> |
| <td>261 / 22</td> |
| </tr> |
| <tr> |
| <td>bench/p50c/math500</td> |
| <td>500×4</td> |
| <td>51.8 ± 1.9</td> |
| <td>31.0</td> |
| <td>21,950</td> |
| <td>75.0</td> |
| <td>34.7</td> |
| <td>+44.5 ± 1.7</td> |
| <td>263 / 0</td> |
| </tr> |
| <tr> |
| <td>build/base</td> |
| <td>128×2</td> |
| <td>11.7 ± 2.5</td> |
| <td>89.8</td> |
| <td>29,872</td> |
| <td>100.0</td> |
| <td>96.5</td> |
| <td>-</td> |
| <td>-</td> |
| </tr> |
| <tr> |
| <td>dev/a-4</td> |
| <td>256×2</td> |
| <td>19.7 ± 1.9</td> |
| <td>65.8</td> |
| <td>26,063</td> |
| <td>55.4</td> |
| <td>96.4</td> |
| <td>+7.6 ± 2.2</td> |
| <td>54 / 24</td> |
| </tr> |
| <tr> |
| <td>dev/a1</td> |
| <td>256×2</td> |
| <td>22.7 ± 2.1</td> |
| <td>77.1</td> |
| <td>26,964</td> |
| <td>96.6</td> |
| <td>93.9</td> |
| <td>+10.5 ± 1.8</td> |
| <td>42 / 7</td> |
| </tr> |
| <tr> |
| <td>dev/a12</td> |
| <td>256×2</td> |
| <td>51.8 ± 2.8</td> |
| <td>12.3</td> |
| <td>13,515</td> |
| <td>59.0</td> |
| <td>9.5</td> |
| <td>+39.6 ± 2.7</td> |
| <td>102 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/a2</td> |
| <td>256×2</td> |
| <td>35.2 ± 2.6</td> |
| <td>62.7</td> |
| <td>23,911</td> |
| <td>92.1</td> |
| <td>94.7</td> |
| <td>+23.0 ± 2.3</td> |
| <td>71 / 5</td> |
| </tr> |
| <tr> |
| <td>dev/a4</td> |
| <td>256×2</td> |
| <td>59.2 ± 2.8</td> |
| <td>29.1</td> |
| <td>17,730</td> |
| <td>82.9</td> |
| <td>80.5</td> |
| <td>+47.1 ± 2.6</td> |
| <td>118 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/a8</td> |
| <td>256×2</td> |
| <td>60.7 ± 2.8</td> |
| <td>14.1</td> |
| <td>14,589</td> |
| <td>70.7</td> |
| <td>26.4</td> |
| <td>+48.6 ± 2.6</td> |
| <td>119 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/all1</td> |
| <td>256×2</td> |
| <td>28.7 ± 2.3</td> |
| <td>71.9</td> |
| <td>25,609</td> |
| <td>97.9</td> |
| <td>95.7</td> |
| <td>+16.6 ± 2.0</td> |
| <td>61 / 6</td> |
| </tr> |
| <tr> |
| <td>dev/all2</td> |
| <td>256×2</td> |
| <td>49.4 ± 2.8</td> |
| <td>46.7</td> |
| <td>20,117</td> |
| <td>91.9</td> |
| <td>87.0</td> |
| <td>+37.3 ± 2.5</td> |
| <td>97 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/base</td> |
| <td>256×2</td> |
| <td>12.1 ± 1.6</td> |
| <td>88.7</td> |
| <td>29,683</td> |
| <td>100.0</td> |
| <td>96.9</td> |
| <td>-</td> |
| <td>-</td> |
| </tr> |
| <tr> |
| <td>dev/base+max4k</td> |
| <td>256×2</td> |
| <td>5.9 ± 1.1</td> |
| <td>95.5</td> |
| <td>4,019</td> |
| <td>100.0</td> |
| <td>18.2</td> |
| <td>-6.2 ± 1.2</td> |
| <td>3 / 30</td> |
| </tr> |
| <tr> |
| <td>dev/base+nothink</td> |
| <td>256×2</td> |
| <td>52.1 ± 2.7</td> |
| <td>18.0</td> |
| <td>2,270</td> |
| <td>63.6</td> |
| <td>7.6</td> |
| <td>+40.0 ± 2.5</td> |
| <td>115 / 2</td> |
| </tr> |
| <tr> |
| <td>dev/base+pp0</td> |
| <td>256×2</td> |
| <td>4.5 ± 0.9</td> |
| <td>99.0</td> |
| <td>32,472</td> |
| <td>100.0</td> |
| <td>98.4</td> |
| <td>-7.6 ± 1.6</td> |
| <td>11 / 42</td> |
| </tr> |
| <tr> |
| <td>dev/base+pp2</td> |
| <td>256×2</td> |
| <td>16.2 ± 1.9</td> |
| <td>84.6</td> |
| <td>28,614</td> |
| <td>98.7</td> |
| <td>96.8</td> |
| <td>+4.1 ± 1.7</td> |
| <td>24 / 13</td> |
| </tr> |
| <tr> |
| <td>dev/base+rep1.1</td> |
| <td>256×2</td> |
| <td>34.4 ± 2.6</td> |
| <td>61.7</td> |
| <td>23,301</td> |
| <td>89.8</td> |
| <td>88.3</td> |
| <td>+22.3 ± 2.2</td> |
| <td>62 / 3</td> |
| </tr> |
| <tr> |
| <td>dev/base+temp1.0</td> |
| <td>256×2</td> |
| <td>27.9 ± 2.5</td> |
| <td>71.3</td> |
| <td>27,430</td> |
| <td>95.9</td> |
| <td>59.5</td> |
| <td>+15.8 ± 2.1</td> |
| <td>52 / 9</td> |
| </tr> |
| <tr> |
| <td>dev/capA-a6</td> |
| <td>256×2</td> |
| <td>43.0 ± 2.6</td> |
| <td>52.5</td> |
| <td>22,699</td> |
| <td>89.3</td> |
| <td>81.4</td> |
| <td>+30.9 ± 2.3</td> |
| <td>95 / 1</td> |
| </tr> |
| <tr> |
| <td>dev/capA-alpha-8</td> |
| <td>256×2</td> |
| <td>42.8 ± 2.8</td> |
| <td>53.5</td> |
| <td>22,772</td> |
| <td>89.9</td> |
| <td>78.5</td> |
| <td>+30.7 ± 2.5</td> |
| <td>87 / 4</td> |
| </tr> |
| <tr> |
| <td>dev/capA-max-samples-16</td> |
| <td>256×2</td> |
| <td>17.6 ± 1.9</td> |
| <td>82.6</td> |
| <td>28,525</td> |
| <td>96.6</td> |
| <td>96.0</td> |
| <td>+5.5 ± 1.9</td> |
| <td>34 / 17</td> |
| </tr> |
| <tr> |
| <td>dev/capB-a6</td> |
| <td>256×2</td> |
| <td>58.2 ± 2.9</td> |
| <td>30.7</td> |
| <td>19,899</td> |
| <td>83.7</td> |
| <td>49.0</td> |
| <td>+46.1 ± 2.6</td> |
| <td>113 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/capB-alpha-8</td> |
| <td>256×2</td> |
| <td>54.5 ± 2.9</td> |
| <td>34.0</td> |
| <td>20,870</td> |
| <td>81.7</td> |
| <td>37.9</td> |
| <td>+42.4 ± 2.6</td> |
| <td>107 / 1</td> |
| </tr> |
| <tr> |
| <td>dev/capB-max-samples-16</td> |
| <td>256×2</td> |
| <td>53.7 ± 2.8</td> |
| <td>39.6</td> |
| <td>20,725</td> |
| <td>88.7</td> |
| <td>76.8</td> |
| <td>+41.6 ± 2.6</td> |
| <td>110 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/l15only</td> |
| <td>256×2</td> |
| <td>60.4 ± 2.9</td> |
| <td>28.5</td> |
| <td>17,362</td> |
| <td>84.2</td> |
| <td>67.8</td> |
| <td>+48.2 ± 2.7</td> |
| <td>117 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/l19only</td> |
| <td>256×2</td> |
| <td>46.9 ± 2.7</td> |
| <td>49.0</td> |
| <td>20,729</td> |
| <td>91.2</td> |
| <td>90.0</td> |
| <td>+34.8 ± 2.4</td> |
| <td>100 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/p50c</td> |
| <td>256×2</td> |
| <td>61.5 ± 2.8</td> |
| <td>19.7</td> |
| <td>16,104</td> |
| <td>76.6</td> |
| <td>24.8</td> |
| <td>+49.4 ± 2.6</td> |
| <td>125 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/p50c+nothink</td> |
| <td>256×2</td> |
| <td>53.5 ± 2.8</td> |
| <td>16.4</td> |
| <td>2,306</td> |
| <td>64.0</td> |
| <td>0.0</td> |
| <td>+1.4 ± 1.9</td> |
| <td>22 / 26</td> |
| </tr> |
| <tr> |
| <td>dev/randcols</td> |
| <td>256×2</td> |
| <td>14.6 ± 1.9</td> |
| <td>86.7</td> |
| <td>29,113</td> |
| <td>100.0</td> |
| <td>97.5</td> |
| <td>+2.5 ± 1.6</td> |
| <td>23 / 20</td> |
| </tr> |
| <tr> |
| <td>dev/randdir</td> |
| <td>256×2</td> |
| <td>14.5 ± 1.8</td> |
| <td>87.3</td> |
| <td>29,486</td> |
| <td>98.5</td> |
| <td>97.3</td> |
| <td>+2.3 ± 1.6</td> |
| <td>22 / 18</td> |
| </tr> |
| <tr> |
| <td>dev/rebuilt</td> |
| <td>256×2</td> |
| <td>61.5 ± 2.8</td> |
| <td>17.6</td> |
| <td>16,350</td> |
| <td>74.6</td> |
| <td>32.2</td> |
| <td>+49.4 ± 2.6</td> |
| <td>122 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/rebuilt-layers15-19-21</td> |
| <td>256×2</td> |
| <td>61.3 ± 2.8</td> |
| <td>21.1</td> |
| <td>15,947</td> |
| <td>77.5</td> |
| <td>38.0</td> |
| <td>+49.2 ± 2.6</td> |
| <td>122 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/rebuilt-pairs2</td> |
| <td>256×2</td> |
| <td>62.3 ± 2.7</td> |
| <td>25.2</td> |
| <td>16,080</td> |
| <td>83.0</td> |
| <td>74.4</td> |
| <td>+50.2 ± 2.6</td> |
| <td>129 / 0</td> |
| </tr> |
| <tr> |
| <td>dev/rebuilt-windows36-36</td> |
| <td>256×2</td> |
| <td>61.9 ± 2.8</td> |
| <td>21.3</td> |
| <td>16,141</td> |
| <td>78.7</td> |
| <td>35.8</td> |
| <td>+49.8 ± 2.7</td> |
| <td>122 / 0</td> |
| </tr> |
| <tr> |
| <td>quick/base</td> |
| <td>16×2</td> |
| <td>18.8 ± 7.7</td> |
| <td>84.4</td> |
| <td>28,375</td> |
| <td>100.0</td> |
| <td>100.0</td> |
| <td>-</td> |
| <td>-</td> |
| </tr> |
| <tr> |
| <td>quick/p50c</td> |
| <td>16×2</td> |
| <td>65.6 ± 11.8</td> |
| <td>9.4</td> |
| <td>11,664</td> |
| <td>72.4</td> |
| <td>0.0</td> |
| <td>+46.9 ± 11.6</td> |
| <td>6 / 0</td> |
| </tr> |
| </tbody> |
| </table></div> |
| <p>"Cut answers ending in repetition" counts two kinds of ending among the answers cut at the cap. The first is a word-for-word loop (one block of words repeated back to back at least 3 times, over at least 40 words). The second is a looser circling (at least half of the 4-word phrases in the last 200 words occur 3 or more times there). See verbatim_loop_end and repetitive_end in wsp/grading.py.</p> |
| </article> |
|
|
| </main> |
|
|
| </div> |
| <footer class="foot"> |
| <div class="foot-in"> |
| <p class="foot-note">Based on Experiment 8 of "<a href="https://rikiyat.github.io/blog/weight-space-programming/" target="_blank" rel="noopener">Sparks of Weight-Space Programming</a>". Logs and the agent's texts come from <a href="https://github.com/RikiyaT/lm_understand-logs" target="_blank" rel="noopener">RikiyaT/lm_understand-logs</a>, <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" rel="noopener">CC BY 4.0</a>.</p> |
| <p class="foot-links"><a href="https://huggingface.co/spaces/thegovind/weight-space-programming/blob/main/code/results/summary.md" target="_blank" rel="noopener">View source</a></p> |
| </div> |
| </footer> |
| </body> |
| </html> |
|
|
