| <html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Text boundaries and update size are part of the recipe · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">11 / Training</div><h1 class="lang-ja " lang="ja">文章の区切り方も、学習条件の一部だった</h1><h1 class="lang-en " lang="en">Text boundaries and update size are part of the recipe</h1><p class="lang-ja standfirst" lang="ja">保存ファイルの一行を文脈として扱うか、隣接行をまとめるかで、記憶を利用する機会と更新単位が変わります。</p><p class="lang-en standfirst" lang="en">Treating a stored line as a context or grouping adjacent lines changes both memory opportunities and optimization.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">同じコーパスでも同じ学習にはならない</h2><h2 class="lang-en " lang="en">The same corpus can define a different training problem</h2><div class="lang-ja" lang="ja"><p>記憶を持つモデルでは、どこで状態をリセットするかが、そのモデルに何を覚えさせるかを決めます。短い保存行ごとにリセットすれば、行をまたぐ情報は使えません。隣接行をまとめれば文脈は長くなりますが、そのまとまりが本当の文書境界と一致するとは限りません。</p><p>同様に、一更新で何個の予測位置をまとめるかも重要です。ファイルの行長に応じて更新回数や平均の取り方が変わると、同じLRと総語数でもoptimizerが受ける系列が変わります。</p></div><div class="lang-en" lang="en"><p>For a memory model, reset boundaries determine which information can persist. Resetting at every short stored line prevents cross-line use. Grouping adjacent lines gives longer contexts, but those groups are not necessarily true document boundaries.</p><p>The number of supervised positions in an optimizer update also matters. If line lengths change update counts or averaging, identical LR and word exposure do not imply the same optimization trajectory.</p></div></section><section><h2 class="lang-ja " lang="ja">Dのみの系列で修正したもの</h2><h2 class="lang-en " lang="en">A D-only recipe correction</h2><div class="lang-ja" lang="ja"><p>初期のDのみモデルは行単位の文脈・可変更新・LR固定でした。後のモデルは隣接行グループ、4096位置を基本にする更新、0.0004→0.00035→0.0003のscheduleへ変えています。複数条件を同時に変えた比較です。</p><p>最高5軸は50.026から50.272へ上がる一方、最高6軸は47.594から47.440へ下がりました。実装条件をそろえる修正が、そのまま全指標の改善になるとは限りません。</p></div><div class="lang-en" lang="en"><p>The initial D-only model used line-level contexts, variable-size updates, and a constant LR. A later version used adjacent-line groups, updates based on 4096 positions, and a 0.0004 → 0.00035 → 0.0003 schedule. Several conditions changed together.</p><p>The five-axis peak rose from 50.026 to 50.272, while the six-axis peak fell from 47.594 to 47.440. Correcting recipe comparability did not automatically improve every aggregate.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">採点済み時点での最高点。5軸と6軸は独立に選択。</span><span class="lang-en " lang="en">Observed peaks; five- and six-axis maxima selected independently.</span></caption><thead><tr><th>ID</th><th><span class="lang-ja " lang="ja">構造：状態幅 / FFN / 層</span><span class="lang-en " lang="en">Structure: width / FFN / layers</span></th><th>Seed</th><th><span class="lang-ja " lang="ja">5軸最高</span><span class="lang-en " lang="en">Peak five</span></th><th><span class="lang-ja " lang="ja">その時点</span><span class="lang-en " lang="en">Age</span></th><th><span class="lang-ja " lang="ja">6軸最高</span><span class="lang-en " lang="en">Peak six</span></th><th><span class="lang-ja " lang="ja">その時点</span><span class="lang-en " lang="en">Age</span></th></tr></thead><tbody><tr><td><a class="model-link" href="../index.html#model-298">#298</a></td><td>384 / FFN 1408 / 8L · D 7×(128×384)</td><td>20260907</td><td>50.026</td><td>80.0M</td><td>47.594</td><td>70.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-299">#299</a></td><td>384 / FFN 1408 / 8L · D 7×(128×384)</td><td>20260907</td><td>50.272</td><td>80.0M</td><td>47.440</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-300">#300</a></td><td>384 / FFN 1408 / 8L · D 7×(128×384)</td><td>20260907</td><td>49.930</td><td>80.0M</td><td>47.312</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-301">#301</a></td><td>384 / FFN 1408 / 8L · D 7×(128×384)</td><td>20260907</td><td>50.258</td><td>80.0M</td><td>47.722</td><td>70.0M</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">公開するときに必要な情報</h2><h2 class="lang-en " lang="en">What should accompany a result</h2><div class="lang-ja" lang="ja"><p>モデル構造と総語数だけでは再現に足りません。トークン化、文脈を切る規則、Attention mask、記憶reset、最後の端数更新、lossを平均する単位、データの次位置まで記録する必要があります。</p><p>この知見集の設定詳細は既存記録の範囲を公開しています。欠けた項目は未確認として残し、ファイル名だけから推定して埋めていません。</p></div><div class="lang-en" lang="en"><p>Architecture and exposure alone are insufficient for reproduction. Tokenization, context splitting, Attention masks, memory resets, partial final updates, loss averaging, and the next data position all belong in the recipe.</p><p>The explorer publishes the available records. Missing settings remain unknown rather than being inferred from a filename.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-298">#298</a> <a class="model-link" href="../index.html#model-299">#299</a> <a class="model-link" href="../index.html#model-301">#301</a></p></div><nav class="chapter-nav"><a href="memory-use.html">← <span class="lang-ja " lang="ja">記憶を作れたことと、予測に役立ったことを分ける</span><span class="lang-en " lang="en">Storing information and using it profitably are different tests</span></a><a href="speed.html"><span class="lang-ja " lang="ja">速くする前に、何の計算が増えたかを測る</span><span class="lang-en " lang="en">Measure the added computation before optimizing it</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |