| <html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Measure the added computation before optimizing it · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">12 / Implementation</div><h1 class="lang-ja " lang="ja">速くする前に、何の計算が増えたかを測る</h1><h1 class="lang-en " lang="en">Measure the added computation before optimizing it</h1><p class="lang-ja standfirst" lang="ja">1-pass化や候補の絞り込みだけでは速度を保証できません。実際の状態更新、保存、中間値、同期まで確認します。</p><p class="lang-en standfirst" lang="en">One pass or fewer active candidates does not guarantee speed; recurrence, saved states, and synchronization still matter.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">構造名から速度は決まらない</h2><h2 class="lang-en " lang="en">Architecture labels do not determine speed</h2><div class="lang-ja" lang="ja"><p>最終予測を次トークンの入力へ返す構造には、一つの位置の処理が終わるまで次を始められない依存があります。2-passでは予備予測で代用し、学習の位置方向をまとめて計算します。ただし、これは厳密な逐次計算に対する近似です。</p><p>1-passの記憶モデルでも、時間方向の行列更新や逆伝播用の状態保存が重ければ遅くなります。しきい値で重みを0にしても全候補の投影や行列積を実行していれば、その部分の演算量は減りません。</p></div><div class="lang-en" lang="en"><p>Feeding a final prediction into the next token introduces a dependency: the current position must finish first. Two-pass training substitutes preliminary predictions so positions can be processed in parallel within a pass. This is an approximation to exact recurrence.</p><p>A one-pass memory model can still be slow if temporal matrix updates or backward-state storage dominate. Thresholding weights to zero does not reduce arithmetic when all candidate projections and matrix products still execute.</p></div></section><section><h2 class="lang-ja " lang="ja">数値を保った改善の例</h2><h2 class="lang-en " lang="en">An optimization with measured numerical agreement</h2><div class="lang-ja" lang="ja"><p>DC-B1の短い実データ試験では、文脈末尾で書き込んでも後の予測から読まれない候補処理を省略しました。同じ10M重み・Adam内部状態・乱数へ戻して比較し、四つのwindowでloss、勾配、parameter更新、optimizer状態の最大差は0でした。</p><p>記録された短期試験では速度比は約1.045倍です。計算が不要な位置を特定することで、学習の意味を保った改善が得られた例です。ただし、少数の計時から長時間学習全体の倍率までは確定しません。</p></div><div class="lang-en" lang="en"><p>A short DC-B1 test skipped candidate work at context tails where later predictions could never read the writes. Comparisons restored the same 10M weights, Adam state, and RNG. Across four real-data windows, maximum differences in loss, gradients, parameter updates, and optimizer state were zero.</p><p>The recorded short-test speed ratio was about 1.045×. This is an example of removing provably unused work while preserving the tested updates. It does not establish the end-to-end speedup of a long training run.</p></div></section><section><h2 class="lang-ja " lang="ja">他の研究で再利用できる手順</h2><h2 class="lang-en " lang="en">A transferable procedure</h2><div class="lang-ja" lang="ja"><p>速度は、同じ実データ・系列長・有効batch・精度で測ります。コンパイル初回費用と定常更新時間を分け、さらに保存と評価を含めた総時間を見ます。新しい融合や低精度化では、出力だけでなく勾配とAdam更新まで照合します。</p><p>丸めが変わる最適化と、候補数を切る・帰還を近似する構造変更は分けて記録します。どちらも試す余地はありますが、同じモデルを速くしたのか、学習する計算を変えたのかが読者に分かることが必要です。</p></div><div class="lang-en" lang="en"><p>Measure speed on the same real data, sequence lengths, effective batch, and precision. Separate first-compilation cost from steady updates, then include saving and evaluation in total runtime. For fusion or precision changes, compare gradients and Adam updates as well as outputs.</p><p>Record numerical-order changes separately from changes such as limiting candidate count or approximating feedback. Both may be useful, but readers need to know whether execution was accelerated or the learned computation was changed.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p></p><a class="download" href="../data/evidence/exact_tail_speed.json">exact_tail_speed.json ↗</a> </div><nav class="chapter-nav"><a href="data-boundaries.html">← <span class="lang-ja " lang="ja">文章の区切り方も、学習条件の一部だった</span><span class="lang-en " lang="en">Text boundaries and update size are part of the recipe</span></a><a href="early-late.html"><span class="lang-ja " lang="ja">途中の順位と、最終的な到達点を混同しない</span><span class="lang-en " lang="en">An early ranking and a later peak answer different questions</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |