File size: 8,522 Bytes
3897c38 | 1 | <!doctype html><html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Storing information and using it profitably are different tests · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">10 / Memory</div><h1 class="lang-ja " lang="ja">記憶を作れたことと、予測に役立ったことを分ける</h1><h1 class="lang-en " lang="en">Storing information and using it profitably are different tests</h1><p class="lang-ja standfirst" lang="ja">D/Cモデルの経路を止める監査では、自己変化の記憶Dと候補記憶Cで寄与が大きく違いました。</p><p class="lang-en standfirst" lang="en">A frozen-weight intervention found very different contributions from processing-change memory D and candidate memory C.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">二種類の記憶</h2><h2 class="lang-en " lang="en">Two memories</h2><div class="lang-ja" lang="ja"><p>D/C系列では、DにAttention・FFNによる状態変化、Cに予測候補の情報を書き込みます。現在状態から作るQueryで行列を読み、その結果を本体の状態へ加えます。元のVISTARCより大きく構造を変更した別系列です。</p><p>10Mまで学習したDC-B1の重みを固定し、通常実行とD読み出し停止、C読み出し停止を比べました。学習済みのモデルが現に何へ依存しているかを調べる介入です。</p></div><div class="lang-en" lang="en"><p>The D/C family writes Attention/FFN state changes into D and prediction-candidate information into C. A query from the current state reads each matrix, and the result updates the model state. This is a substantial redesign of VISTARC.</p><p>For a DC-B1 checkpoint trained to 10M, a frozen-weight audit compared normal execution with D reads disabled and C reads disabled. This measures what the trained model currently relies on.</p></div></section><section><h2 class="lang-ja " lang="ja">損失の変化</h2><h2 class="lang-en " lang="en">Changes in prediction loss</h2><div class="lang-ja" lang="ja"><p>169,144予測位置を含む検証部分集合では、Dを読まないと平均負の対数尤度が約＋0.419044悪化しました。Cを読まない場合は約＋0.000508で、文書単位bootstrapの95％区間は−0.000250〜＋0.001403でした。損失は小さいほど良い指標です。</p><p>このモデルではDの利用が重要で、Cの平均寄与は小さく不確実でした。Cの効果は文章の種類や記憶を使える時刻にも依存しており、全体平均だけで常に有益と判断できませんでした。</p></div><div class="lang-en" lang="en"><p>On a validation subset with 169,144 prediction targets, disabling D increased mean negative log likelihood by about 0.419044. Disabling C increased it by about 0.000508, with a document-bootstrap 95% interval of −0.000250 to +0.001403. Lower loss is better.</p><p>D use mattered strongly in this trained model, while C’s mean contribution was small and uncertain. Effects also depended on text type and when the memory became available.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">10M重み、169,144予測対象。正の値は損失が悪化。</span><span class="lang-en " lang="en">10M weights, 169,144 target positions; positive means worse loss.</span></caption><thead><tr><th><span class="lang-ja " lang="ja">同じ重みへの介入</span><span class="lang-en " lang="en">Intervention on the same weights</span></th><th><span class="lang-ja " lang="ja">通常からのNLL変化</span><span class="lang-en " lang="en">Change in NLL</span></th></tr></thead><tbody><tr><td><span class="lang-ja " lang="ja">Dの読み出しを止める</span><span class="lang-en " lang="en">Disable D readout</span></td><td>+0.419044</td></tr><tr><td><span class="lang-ja " lang="ja">Cの読み出しを止める</span><span class="lang-en " lang="en">Disable C readout</span></td><td>+0.000508</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">介入から言える範囲</h2><h2 class="lang-en " lang="en">The intervention’s scope</h2><div class="lang-ja" lang="ja"><p>読み出しを止めたときの悪化は、学習済みモデルがその経路を使っている証拠です。しかし、最初からその部品なしで学習したモデルがどこまで到達するかは別の問いです。</p><p>後のDのみの再訓練や読み出しゲート追加、QK・Value側の変更も試しました。これらのローカルベンチは改善が一様ではありません。内部監査で経路が有効と分かることと、その経路を強化すれば総合性能が上がることは、別々に検証する必要がありました。</p></div><div class="lang-en" lang="en"><p>A loss increase when reads are disabled shows reliance on that pathway in the trained model. It does not determine the performance of a model trained without the component from the start.</p><p>Later experiments retrained D-only models and modified reading gates or QK/value handling. Benchmark gains were not uniform. Demonstrating that a pathway is used and demonstrating that strengthening it improves overall performance are distinct questions.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-296">#296</a> <a class="model-link" href="../index.html#model-297">#297</a> <a class="model-link" href="../index.html#model-298">#298</a> <a class="model-link" href="../index.html#model-299">#299</a> <a class="model-link" href="../index.html#model-300">#300</a> <a class="model-link" href="../index.html#model-301">#301</a></p><a class="download" href="../data/evidence/memory_intervention.json">memory_intervention.json ↗</a> </div><nav class="chapter-nav"><a href="memory-rank.html">← <span class="lang-ja " lang="ja">記憶の次元を増やしても、性能は単調に伸びなかった</span><span class="lang-en " lang="en">More memory dimensions did not produce monotonic gains</span></a><a href="data-boundaries.html"><span class="lang-ja " lang="ja">文章の区切り方も、学習条件の一部だった</span><span class="lang-en " lang="en">Text boundaries and update size are part of the recipe</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |