| <html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>FFN addition improved some averages, but not every seed · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">07 / Architecture</div><h1 class="lang-ja " lang="ja">FFN加算は、平均を改善しても全seedを改善しなかった</h1><h1 class="lang-en " lang="en">FFN addition improved some averages, but not every seed</h1><p class="lang-ja standfirst" lang="ja">ゼロ初期化した加算経路は旧6軸の三seed平均を上げましたが、最高点と別の問題集では同じ結論になりませんでした。</p><p class="lang-en standfirst" lang="en">A zero-initialized additive path helped the historical three-seed mean, with different outcomes for the best seed and the filtered bank.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">何を追加したか</h2><h2 class="lang-en " lang="en">The modification</h2><div class="lang-ja" lang="ja"><p>SiLUゲート型FFNは、二つの投影から作った値を掛け合わせます。この系列ではゲート側に学習する加算bを設け、概念的に「SiLU(g)×u」を「(SiLU(g)+b)×u」へ変えました。bをゼロから始めるので、追加直後の関数を維持しながら経路を学習できます。</p><p>320幅・FFN1120の元構成と加算版を、同じ三つのseedの80Mで対応させました。</p></div><div class="lang-en" lang="en"><p>A SiLU-gated FFN multiplies two projected signals. This variant adds a learned b to the gate side, changing SiLU(g) × u into (SiLU(g) + b) × u. Initializing b to zero preserves the initial function while allowing the added route to learn.</p><p>The original and additive 320-wide, FFN1120 configurations were compared at 80M with the same three seeds.</p></div></section><section><h2 class="lang-ja " lang="ja">評価集合によって変わった結論</h2><h2 class="lang-en " lang="en">The outcome depends on the measurement</h2><div class="lang-ja" lang="ja"><p>旧6軸の平均は48.690から49.047へ上がりました。旧5軸の平均は51.123から51.187で、小さな改善です。一方、元の強いseedではSupplementやEntityを落とし、元モデルの最高6軸49.868を更新できませんでした。</p><p>後の明確回答ベンチでは、加算による同seed差が順に＋0.266、＋0.318、−0.741ポイントでした。三seed平均は元構成71.076％、加算版71.024％です。旧6軸での平均改善を、そのまま別の問題集での優位と読み替えることはできません。</p></div><div class="lang-en" lang="en"><p>The legacy six-axis mean rose from 48.690 to 49.047; the five-axis mean rose slightly from 51.123 to 51.187. Yet the strongest original seed lost Supplement and Entity performance, and addition did not beat the original six-axis record of 49.868.</p><p>On the later clear-answer bank, matched-seed changes were +0.266, +0.318, and −0.741 points. Means were 71.076% for the original and 71.024% for addition. An improvement in the legacy aggregate did not transfer directly to the new bank.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">記録済みの同時点比較。構造以外の設定は各IDの詳細に掲載。</span><span class="lang-en " lang="en">Recorded scores by age; each ID links to the complete recorded configuration.</span></caption><thead><tr><th>ID</th><th><span class="lang-ja " lang="ja">構造：状態幅 / FFN / 層</span><span class="lang-en " lang="en">Structure: width / FFN / layers</span></th><th>Seed</th><th><span class="lang-ja " lang="ja">学習量</span><span class="lang-en " lang="en">Exposure</span></th><th><span class="lang-ja " lang="ja">5軸</span><span class="lang-en " lang="en">Five</span></th><th><span class="lang-ja " lang="ja">6軸</span><span class="lang-en " lang="en">Six</span></th></tr></thead><tbody><tr><td><a class="model-link" href="../index.html#model-1">#1</a></td><td>320 / FFN 1120 / 9L</td><td>20260907</td><td>80M</td><td>52.002</td><td>49.867</td></tr><tr><td><a class="model-link" href="../index.html#model-81">#81</a></td><td>320 / FFN 1120 / 9L</td><td>731946281</td><td>80M</td><td>50.276</td><td>48.091</td></tr><tr><td><a class="model-link" href="../index.html#model-26">#26</a></td><td>320 / FFN 1120 / 9L</td><td>137</td><td>80M</td><td>51.092</td><td>48.112</td></tr><tr><td><a class="model-link" href="../index.html#model-22">#22</a></td><td>320 / FFN 1120 / 9L</td><td>20260907</td><td>80M</td><td>51.182</td><td>49.096</td></tr><tr><td><a class="model-link" href="../index.html#model-16">#16</a></td><td>320 / FFN 1120 / 9L</td><td>731946281</td><td>80M</td><td>51.016</td><td>49.205</td></tr><tr><td><a class="model-link" href="../index.html#model-40">#40</a></td><td>320 / FFN 1120 / 9L</td><td>137</td><td>80M</td><td>51.364</td><td>48.841</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">条件付きにすれば解決するか</h2><h2 class="lang-en " lang="en">Did conditional addition solve it?</h2><div class="lang-ja" lang="ja"><p>負側のゲート領域へ作用を絞る条件付き加算も試しましたが、この設定で全seedの強みを保つ解決には至りませんでした。条件付き版の明確回答ベンチ三seed平均は70Mで70.769％です。これは80Mの他系列と学習時点が違うので、同時点平均として混ぜて順位を付けません。</p><p>ここで得たのは、加算の有効性がseed・課題・条件の組み合わせに依存するという観測です。飽和を救うという設計上の説明だけで、最高点更新を予測することはできませんでした。</p></div><div class="lang-en" lang="en"><p>A conditional variant targeted the negative gate region, but the tested setting did not preserve all strengths across seeds. Its clear-answer three-seed mean was 70.769% at 70M. That age differs from the 80M means of the other families and must remain separate.</p><p>The observed effect depends on seed, task, and configuration. A plausible explanation about relieving saturation was not sufficient to predict a new performance record.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-1">#1</a> <a class="model-link" href="../index.html#model-81">#81</a> <a class="model-link" href="../index.html#model-26">#26</a> <a class="model-link" href="../index.html#model-22">#22</a> <a class="model-link" href="../index.html#model-16">#16</a> <a class="model-link" href="../index.html#model-40">#40</a> <a class="model-link" href="../index.html#model-70">#70</a> <a class="model-link" href="../index.html#model-97">#97</a> <a class="model-link" href="../index.html#model-64">#64</a></p></div><nav class="chapter-nav"><a href="ffn-ratio.html">← <span class="lang-ja " lang="ja">FFN幅は、状態幅との比率だけで決められるか</span><span class="lang-en " lang="en">Can FFN width be chosen from a single ratio?</span></a><a href="lr-groups.html"><span class="lang-ja " lang="ja">学習率は部品ごとに記録し、変更の組み合わせで読む</span><span class="lang-en " lang="en">Read learning-rate changes as named, coupled settings</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |