| <!doctype html><html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>The favorable depth depends on the rest of the architecture · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">05 / Architecture</div><h1 class="lang-ja " lang="ja">最適な深さは、組み合わせる部品によって変わる</h1><h1 class="lang-en " lang="en">The favorable depth depends on the rest of the architecture</h1><p class="lang-ja standfirst" lang="ja">Fなしでは9層、F付きの試行では8層が有望でした。層数の効果は単調ではありませんでした。</p><p class="lang-en standfirst" lang="en">Nine layers led the tested non-F series, while eight layers led the F-equipped depth series.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">調べた問い</h2><h2 class="lang-en " lang="en">The question</h2><div class="lang-ja" lang="ja"><p>層を増やせば、入力を加工する段階が増えます。しかし、同じ語数とLRで学習したときに性能が単調に上がるとは限りません。384幅・FFN1408の系列で深さを変え、Fを追加した系列でも深さを変えました。出力枝は各深さの末尾1・2・3層を使う構成です。</p></div><div class="lang-en" lang="en"><p>More layers add processing stages, but performance need not improve monotonically at a fixed exposure and LR. Depth was varied for a 384-wide, FFN1408 family and again for its F-equipped counterpart. Output branches followed the trailing one, two, and three layers at each depth.</p></div></section><section><h2 class="lang-ja " lang="ja">結果の読み方</h2><h2 class="lang-en " lang="en">Reading the results</h2><div class="lang-ja" lang="ja"><p>Fなしの記録では、9層が6軸49.642、8層が48.381、10層が49.016、11層が49.115、12層が48.823でした。F付きでは7層48.022、8層49.771、9層48.981、10層48.710です。いずれも同じseedで採点した範囲の最高点です。</p><p>F付き8層は6軸で目立つ一方、5軸最高は51.660で、F付き9層の51.644との差は小さいものでした。どの指標がどの時点で伸びたかを残す必要があります。</p></div><div class="lang-en" lang="en"><p>In the non-F series, observed six-axis peaks were 49.642 at nine layers, 48.381 at eight, 49.016 at ten, 49.115 at eleven, and 48.823 at twelve. With F, the corresponding seven-, eight-, nine-, and ten-layer peaks were 48.022, 49.771, 48.981, and 48.710. These are records from the evaluated ages of one seed.</p><p>The F-equipped eight-layer model stood out on six axes, but its five-axis peak of 51.660 was only slightly above the nine-layer model’s 51.644. Metric and checkpoint age both matter.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">採点済み時点での最高点。5軸と6軸は独立に選択。</span><span class="lang-en " lang="en">Observed peaks; five- and six-axis maxima selected independently.</span></caption><thead><tr><th>ID</th><th><span class="lang-ja " lang="ja">構造：状態幅 / FFN / 層</span><span class="lang-en " lang="en">Structure: width / FFN / layers</span></th><th>Seed</th><th><span class="lang-ja " lang="ja">5軸最高</span><span class="lang-en " lang="en">Peak five</span></th><th><span class="lang-ja " lang="ja">その時点</span><span class="lang-en " lang="en">Age</span></th><th><span class="lang-ja " lang="ja">6軸最高</span><span class="lang-en " lang="en">Peak six</span></th><th><span class="lang-ja " lang="ja">その時点</span><span class="lang-en " lang="en">Age</span></th></tr></thead><tbody><tr><td><a class="model-link" href="../index.html#model-275">#275</a></td><td>384 / FFN 1408 / 8L</td><td>20260907</td><td>50.624</td><td>80.0M</td><td>48.381</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-3">#3</a></td><td>384 / FFN 1408 / 9L</td><td>20260907</td><td>51.354</td><td>70.0M</td><td>49.642</td><td>70.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-272">#272</a></td><td>384 / FFN 1408 / 10L</td><td>20260907</td><td>51.480</td><td>80.0M</td><td>49.016</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-273">#273</a></td><td>384 / FFN 1408 / 11L</td><td>20260907</td><td>50.922</td><td>80.0M</td><td>49.115</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-274">#274</a></td><td>384 / FFN 1408 / 12L</td><td>20260907</td><td>51.258</td><td>80.0M</td><td>48.823</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-280">#280</a></td><td>384 / FFN 1408 / 7L · F 16×16</td><td>20260907</td><td>50.584</td><td>80.0M</td><td>48.022</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-277">#277</a></td><td>384 / FFN 1408 / 8L · F 16×16</td><td>20260907</td><td>51.660</td><td>70.0M</td><td>49.771</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-270">#270</a></td><td>384 / FFN 1408 / 9L · F 16×16</td><td>20260907</td><td>51.644</td><td>70.0M</td><td>48.981</td><td>70.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-276">#276</a></td><td>384 / FFN 1408 / 10L · F 16×16</td><td>20260907</td><td>50.698</td><td>80.0M</td><td>48.710</td><td>100.0M</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">次の研究へ持ち出せること</h2><h2 class="lang-en " lang="en">What can be reused</h2><div class="lang-ja" lang="ja"><p>深さを変えると、部品が接続される位置や、その前後で処理される情報も変わります。「8層が普遍的な最適値」ではなく、この構造と学習条件の組み合わせで有望な点があった、と読む結果です。</p><p>初期のこの観測にはseed反復がなく、後の三seed比較でも8層F付き構成のばらつきは大きくなりました。深さの章とseedの章を合わせて読むことで、単発の成功を一般化しすぎずに済みます。</p></div><div class="lang-en" lang="en"><p>Changing depth also changes where components act and what processing precedes or follows them. The result identifies a favorable configuration under the tested conditions, not a universal eight-layer optimum.</p><p>The initial depth observation used one seed. Later three-seed evaluation showed substantial variation for the F-equipped eight-layer configuration. The depth and seed results should therefore be read together.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-3">#3</a> <a class="model-link" href="../index.html#model-272">#272</a> <a class="model-link" href="../index.html#model-273">#273</a> <a class="model-link" href="../index.html#model-274">#274</a> <a class="model-link" href="../index.html#model-275">#275</a> <a class="model-link" href="../index.html#model-270">#270</a> <a class="model-link" href="../index.html#model-276">#276</a> <a class="model-link" href="../index.html#model-277">#277</a> <a class="model-link" href="../index.html#model-280">#280</a></p></div><nav class="chapter-nav"><a href="width-lr.html">← <span class="lang-ja " lang="ja">幅を変えた結果には、実際の学習率も混ざっている</span><span class="lang-en " lang="en">Width comparisons also depend on the effective learning rate</span></a><a href="ffn-ratio.html"><span class="lang-ja " lang="ja">FFN幅は、状態幅との比率だけで決められるか</span><span class="lang-en " lang="en">Can FFN width be chosen from a single ratio?</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |
