File size: 8,783 Bytes
3897c38 | 1 | <!doctype html><html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Width comparisons also depend on the effective learning rate · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">04 / Optimization</div><h1 class="lang-ja " lang="ja">幅を変えた結果には、実際の学習率も混ざっている</h1><h1 class="lang-en " lang="en">Width comparisons also depend on the effective learning rate</h1><p class="lang-ja standfirst" lang="ja">384幅の同じ構造で主要行列のLRを下げると改善しました。ただし、幅に対する最適LRの法則が求まったわけではありません。</p><p class="lang-en standfirst" lang="en">Reducing matrix learning rates helped one 384-wide configuration, but the experiment did not identify a universal scaling law.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">なぜ幅だけでは説明できないか</h2><h2 class="lang-en " lang="en">Why width alone is not enough</h2><div class="lang-ja" lang="ja"><p>幅を増やすと表現できる量だけでなく、行列の形、初期化、学習時の更新の大きさも変わります。以前の幅探索には、主要行列の初期LRを「0.0008×256÷状態幅」で決める設定が使われていました。</p><p>この場合、幅256、272、320、384に対してLRも同時に変わっています。これらの点を並べて「太さに対する最適LRの法則を発見した」と言うことはできません。最初から設定した関係を、実験から推定した関係と取り違えるためです。</p></div><div class="lang-en" lang="en"><p>Increasing width changes matrix shapes, initialization, and update behavior as well as representational capacity. One earlier sweep assigned the initial matrix LR using 0.0008 × 256 / state width.</p><p>Width and LR therefore changed together across widths 256, 272, 320, and 384. Fitting those points cannot establish an optimal-LR scaling law: the apparent relation was imposed by the recipe.</p></div></section><section><h2 class="lang-ja " lang="ja">同じ384幅でLRを変えた結果</h2><h2 class="lang-en " lang="en">A same-width LR comparison</h2><div class="lang-ja" lang="ja"><p>384幅・FFN1408の同じ構造では、主要103行列の実LRを0.00064→0.00056→0.00048から、0.000533→0.000467→0.000400へ変更しました。初期200tensorの一致を確認し、残る97tensorのLRと全WD設定を維持した比較です。</p><p>80Mの6軸は47.118から48.960、5軸は49.206から50.634へ上がりました。ただしEWoKは下がっており、すべての課題が一緒に改善したわけではありません。</p></div><div class="lang-en" lang="en"><p>For the same 384-wide, FFN1408 architecture, the LR schedule of 103 main matrices changed from 0.00064 → 0.00056 → 0.00048 to 0.000533 → 0.000467 → 0.000400. The comparison verified 200 matching initial tensors, unchanged LRs for the other 97 tensors, and unchanged WD settings.</p><p>At 80M, the six-axis score rose from 47.118 to 48.960 and the five-axis score from 49.206 to 50.634. EWoK declined, so the gain was not uniform across tasks.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">記録済みの同時点比較。構造以外の設定は各IDの詳細に掲載。</span><span class="lang-en " lang="en">Recorded scores by age; each ID links to the complete recorded configuration.</span></caption><thead><tr><th>ID</th><th><span class="lang-ja " lang="ja">構造：状態幅 / FFN / 層</span><span class="lang-en " lang="en">Structure: width / FFN / layers</span></th><th>Seed</th><th><span class="lang-ja " lang="ja">学習量</span><span class="lang-en " lang="en">Exposure</span></th><th><span class="lang-ja " lang="ja">5軸</span><span class="lang-en " lang="en">Five</span></th><th><span class="lang-ja " lang="ja">6軸</span><span class="lang-en " lang="en">Six</span></th></tr></thead><tbody><tr><td><a class="model-link" href="../index.html#model-94">#94</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>80M</td><td>49.206</td><td>47.118</td></tr><tr><td><a class="model-link" href="../index.html#model-31">#31</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>80M</td><td>50.634</td><td>48.960</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">解釈の範囲</h2><h2 class="lang-en " lang="en">What this supports</h2><div class="lang-ja" lang="ja"><p>この構成では、試した二つのうち低いLRが有利だったと言えます。二点だけでは最適値や幅に対する指数は求まりません。またAdamWでは、同じWD係数でも一更新の減衰はおおむね「1−LR×WD」です。LR変更はこの減衰にも連動します。</p><p>再利用しやすい記録は「基本LR」という一個の数ではなく、parameter名ごとの実LR、WD、切替時点、初期化、更新単位です。比較表の詳細には、分かっている範囲の名前付き設定を載せています。</p></div><div class="lang-en" lang="en"><p>The lower of these two tested schedules helped this configuration. Two values do not locate an optimum or identify a width-scaling exponent. With AdamW, the per-update decay factor is approximately 1 − LR × WD, so changing LR also changes shrinkage despite a constant WD coefficient.</p><p>Reusable records therefore need named effective LRs, WD, schedule boundaries, initialization, and update units, not just a single base LR. The explorer exposes the recorded parameter groups.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-94">#94</a> <a class="model-link" href="../index.html#model-31">#31</a> <a class="model-link" href="../index.html#model-1">#1</a> <a class="model-link" href="../index.html#model-2">#2</a></p><a class="download" href="../data/evidence/width_lr384.json">width_lr384.json ↗</a> <a class="download" href="../data/evidence/width_lr_identifiability.json">width_lr_identifiability.json ↗</a> </div><nav class="chapter-nav"><a href="seeds.html">← <span class="lang-ja " lang="ja">単発の最高点と、seedを変えた強さは一致しない</span><span class="lang-en " lang="en">A single-run record can hide substantial seed variation</span></a><a href="depth.html"><span class="lang-ja " lang="ja">最適な深さは、組み合わせる部品によって変わる</span><span class="lang-en " lang="en">The favorable depth depends on the rest of the architecture</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |