| <!doctype html><html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Read learning-rate changes as named, coupled settings · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">08 / Optimization</div><h1 class="lang-ja " lang="ja">学習率は部品ごとに記録し、変更の組み合わせで読む</h1><h1 class="lang-en " lang="en">Read learning-rate changes as named, coupled settings</h1><p class="lang-ja standfirst" lang="ja">「主要−10％・その他＋5％」の配分と、そこから一部を戻す試験では、得点とピーク時点が変わりました。</p><p class="lang-en standfirst" lang="en">Grouped LR changes and partial resets altered both scores and peak ages; one base-LR label was insufficient.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">「主要」と「その他」の中身</h2><h2 class="lang-en " lang="en">What the groups mean</h2><div class="lang-ja" lang="ja"><p>この384幅系列の主要グループは、Attention・FFNなどの大きな投影やMoS行列を含みます。その他には埋め込み、正規化係数、候補帰還の係数、温度、Readerなどが含まれます。実際の分類はモデルのparameter名で固定してあり、一般的な部品分類の標準ではありません。</p><p>基準から主要を10％下げ、その他を5％上げた構成では、主要実LRは0.00048→0.00042→0.00036になります。「0％へ戻す」は基準LRに戻すという意味で、学習率ゼロや凍結ではありません。</p></div><div class="lang-en" lang="en"><p>In this 384-wide series, the main group includes large Attention/FFN projections and MoS matrices. Other parameters include embeddings, normalization gains, feedback coefficients, temperature, and Reader parameters. Membership is fixed by parameter name; these labels are not a universal component taxonomy.</p><p>Reducing the main group by 10% and raising the others by 5% gives a main schedule of 0.00048 → 0.00042 → 0.00036. Resetting an item to 0% means returning to its reference LR, not freezing it or setting LR to zero.</p></div></section><section><h2 class="lang-ja " lang="ja">一部を戻した結果</h2><h2 class="lang-en " lang="en">Partial resets</h2><div class="lang-ja" lang="ja"><p>全体配分版から8種類48tensorを基準LRへ戻した設定では、最高6軸48.947を80Mで記録しました。そこから帰還係数α・候補温度τ・Attention gate biasを再び＋5％へ戻した設定は、最高6軸48.609を70Mで記録しました。</p><p>三項目を一緒に変えているので、この低下をαだけ、τだけ、biasだけの責任に分解できません。また前者と後者は70Mだけを見ると後者が上で、80Mでは前者が上です。変更の符号と最終順位を一対一で結び付けるのは危険です。</p></div><div class="lang-en" lang="en"><p>Resetting eight parameter types, 48 tensors, from the grouped recipe to their reference LRs produced a six-axis peak of 48.947 at 80M. Raising feedback α, candidate temperature τ, and Attention gate bias back to +5% then produced a peak of 48.609 at 70M.</p><p>Because all three types changed together, the decline cannot be assigned to α, τ, or bias alone. The latter configuration was better at 70M, while the former was better at 80M. The direction of an LR adjustment does not map to a fixed ranking across ages.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">記録済みの同時点比較。構造以外の設定は各IDの詳細に掲載。</span><span class="lang-en " lang="en">Recorded scores by age; each ID links to the complete recorded configuration.</span></caption><thead><tr><th>ID</th><th><span class="lang-ja " lang="ja">構造：状態幅 / FFN / 層</span><span class="lang-en " lang="en">Structure: width / FFN / layers</span></th><th>Seed</th><th><span class="lang-ja " lang="ja">学習量</span><span class="lang-en " lang="en">Exposure</span></th><th><span class="lang-ja " lang="ja">5軸</span><span class="lang-en " lang="en">Five</span></th><th><span class="lang-ja " lang="ja">6軸</span><span class="lang-en " lang="en">Six</span></th></tr></thead><tbody><tr><td><a class="model-link" href="../index.html#model-31">#31</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>40M</td><td>50.316</td><td>48.286</td></tr><tr><td><a class="model-link" href="../index.html#model-31">#31</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>70M</td><td>50.618</td><td>48.854</td></tr><tr><td><a class="model-link" href="../index.html#model-31">#31</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>80M</td><td>50.634</td><td>48.960</td></tr><tr><td><a class="model-link" href="../index.html#model-3">#3</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>40M</td><td>50.474</td><td>48.079</td></tr><tr><td><a class="model-link" href="../index.html#model-3">#3</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>70M</td><td>51.354</td><td>49.642</td></tr><tr><td><a class="model-link" href="../index.html#model-3">#3</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>80M</td><td>50.732</td><td>48.713</td></tr><tr><td><a class="model-link" href="../index.html#model-268">#268</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>40M</td><td>50.812</td><td>48.116</td></tr><tr><td><a class="model-link" href="../index.html#model-268">#268</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>70M</td><td>51.048</td><td>48.562</td></tr><tr><td><a class="model-link" href="../index.html#model-268">#268</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>80M</td><td>51.306</td><td>48.947</td></tr><tr><td><a class="model-link" href="../index.html#model-269">#269</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>70M</td><td>51.298</td><td>48.609</td></tr><tr><td><a class="model-link" href="../index.html#model-269">#269</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>80M</td><td>51.076</td><td>48.508</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">再利用できる記録</h2><h2 class="lang-en " lang="en">A reusable record</h2><div class="lang-ja" lang="ja"><p>細分化した探索の価値は、部品名に加えて実LR・WDと切替時点が残ることです。次に使う人は、「その他＋5％」という略称だけでなく、詳細にあるparameter名と数値を確認できます。</p><p>これまでの結果から、各部品を独立に少し動かせば得点を足し算で予測できる法則は確立していません。本文ではその未解決も含め、成功した配分と下がった配分を同じ表に残しています。</p></div><div class="lang-en" lang="en"><p>Fine-grained searches are reusable when component names, effective LRs, WD, and schedule boundaries are retained. Readers can inspect named parameters instead of relying on a shorthand such as “other +5%”.</p><p>The experiments have not established an additive rule that predicts performance by independently nudging each component. Both successful and unsuccessful allocations remain visible.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-31">#31</a> <a class="model-link" href="../index.html#model-3">#3</a> <a class="model-link" href="../index.html#model-268">#268</a> <a class="model-link" href="../index.html#model-269">#269</a></p></div><nav class="chapter-nav"><a href="addition.html">← <span class="lang-ja " lang="ja">FFN加算は、平均を改善しても全seedを改善しなかった</span><span class="lang-en " lang="en">FFN addition improved some averages, but not every seed</span></a><a href="memory-rank.html"><span class="lang-ja " lang="ja">記憶の次元を増やしても、性能は単調に伸びなかった</span><span class="lang-en " lang="en">More memory dimensions did not produce monotonic gains</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |
