Release phase6 staging

README.md

@@ -0,0 +1,50 @@
+---
+license: apache-2.0
+language:
+- ja
+tags:
+- historical
+- text-generation
+datasets:
+- trtd56/TimeCapsuleLLM-ja-corpus-v4
+---
+
+# TimeCapsuleLLM-ja phase6
+
+1868–1945年に限定した日本語コーパスからゼロから学習したdecoder-only GPTです。現代の知識や安全性を備えたチャットモデルではありません。
+
+
+
+## 学習データと評価
+
+対応データセットは `trtd56/TimeCapsuleLLM-ja-corpus-v4` です。青空文庫の時代・権利フィルタ通過作品と、1945年以前刊行かつPDMであるNDL OCR品質通過資料を使用しました。test作品はフェーズ間で固定しています。
+
+
+
+- Parameters: 1,046,066,688
+- Processed tokens: 11,875,123,200
+- Training iterations: 45,300
+
+- 戦前test BPB: 0.9683
+- 現代文control BPB: 1.6044
+- 自動生成rubric: 100.0%
+
+詳細な設定、checksum、固定プローブ出力は同梱の `training_config.yaml`、`artifact_manifest.json`、`evaluation.json` を参照してください。
+
+## 推論
+
+`python inference.py '文明開化とは'` を実行してください。必要パッケージは `torch`, `tokenizers`, `safetensors` です。
+
+### safetensorsの重み共有について
+
+入力埋め込みと出力射影は重みを共有しています（weight tying）。safetensorsは同一メモリを指す
+テンソルを重複して保存できないため、**`transformer.wte.weight` はファイルに含まれず
+`lm_head.weight` のみが保存されます**。同梱の `inference.py` は `safetensors.torch.load_model`
+を使っており、`GPT` が読み込み後に重みを結び直すのでそのまま動作します。
+自分でstate_dictを読む場合は `strict=False` を指定し、読み込み後に
+`model.transformer.wte.weight.data_ptr() == model.lm_head.weight.data_ptr()` を
+確認してください。これを怠ると埋め込みが未初期化のまま推論することになります。
+
+## 限界
+
+小型モデルであり、事実誤認、反復、OCR由来の誤字、歴史的な差別表現を生成し得ます。医療・法律・金融その他の意思決定には使用しないでください。

artifact_manifest.json

@@ -0,0 +1,55 @@
+{
+  "files": [
+    {
+      "path": "README.md",
+      "bytes": 2165,
+      "sha256": "6fd088824424f1744be85ee4c0ed2c987abf711b62b2c6aaf59d94dd89d2b8fc"
+    },
+    {
+      "path": "config.json",
+      "bytes": 134,
+      "sha256": "81a7b3ac292c6735a1cd0377ba43b4e7bb942ea9d81208e43b8ff87714e9b865"
+    },
+    {
+      "path": "evaluation.json",
+      "bytes": 5219,
+      "sha256": "f89830b289d82c8481828c9dd4ce91bda0ea12cd6a6f6100c1a05a2cb6b2fb81"
+    },
+    {
+      "path": "generation_config.json",
+      "bytes": 82,
+      "sha256": "c8956e566ededf5439aaee15fe39b79cd811169721ace680b1090ebf5c679b88"
+    },
+    {
+      "path": "inference.py",
+      "bytes": 609,
+      "sha256": "8c199d3991f31063f5b4cd7a21eac86b6963249fb76d9fd102ca7107cce88921"
+    },
+    {
+      "path": "model.py",
+      "bytes": 8325,
+      "sha256": "f522fc4ca93cf6a70174ed82a3fdf50622c1427585bd01b39827632d9f92878e"
+    },
+    {
+      "path": "model.safetensors",
+      "bytes": 4184290360,
+      "sha256": "40d1cf331c3b991a714cde72582e7a60eacb21aa0fab5fd873f9125dd48bb1e5"
+    },
+    {
+      "path": "tokenizer.json",
+      "bytes": 1344854,
+      "sha256": "285c9629a44871f6822453f7a5559b3f46c6b5b643310868035b640564475250"
+    },
+    {
+      "path": "training_config.yaml",
+      "bytes": 7783,
+      "sha256": "456804aa4225bddad60f3437142700711f4c4bc391700819d163d5223dc25e45"
+    },
+    {
+      "path": "training_report.json",
+      "bytes": 29482,
+      "sha256": "5f44c00f4805d37e2609f5f8adbf590d137ce5ca75ace7b73f29df9d2e3f3884"
+    }
+  ],
+  "total_bytes": 4185689013
+}

config.json

@@ -0,0 +1,9 @@
+{
+  "block_size": 1024,
+  "vocab_size": 16384,
+  "n_layer": 36,
+  "n_head": 24,
+  "n_embd": 1536,
+  "dropout": 0.0,
+  "bias": false
+}

evaluation.json

@@ -0,0 +1,63 @@
+{
+  "checkpoint_sha256": "58594d52bfec2dcbc94d8faa569579bc6d489f1a0afa91181f77a9e427e6b642",
+  "tokenizer_sha256": "285c9629a44871f6822453f7a5559b3f46c6b5b643310868035b640564475250",
+  "seed": 1945,
+  "historical_test": {
+    "tokens": 239359,
+    "nll": 3.3371582029457074,
+    "perplexity": 28.13904751256167,
+    "bytes": 1190148,
+    "bits_per_byte": 0.9682781352345962
+  },
+  "modern_control": {
+    "tokens": 306830,
+    "nll": 3.693451552249165,
+    "perplexity": 40.18330264493536,
+    "bytes": 1019052,
+    "bits_per_byte": 1.6043843611561688
+  },
+  "historical_bpb_lower_than_modern": true,
+  "generations": [
+    {
+      "prompt": "文明開化とは",
+      "text": "文明開化とは全く別物で、我國で西洋諸國の文明開化を學んだのぢやが、西洋諸國が我國より先きに開化したとは言へぬ、我れも文明開化には後れを取つた方ぢやから、西洋諸國をも凌駕したとは思はれぬが、其の後れを取つたは西洋諸國文明開化の本家本元だ、此本家本元たる英國の文明開化を學んだのぢやから、其の前には矢張り前とは別物ぢやと思ひ玉ふかも知れぬ、ナニ我れも野蠻未開の國に生れたから、日本でも野蠻未開の國の風が好きと見えると仰せらるかも知れぬ、其れは御勝手ぢや、彼の國",
+      "nonempty": true,
+      "no_mojibake": true,
+      "repetition_rate": 0.02714932126696834,
+      "finite_repetition": true
+    },
+    {
+      "prompt": "昭和十年の東京では",
+      "text": "昭和十年の東京では七百八十店で六百五十六店、二千六百五十六店が二百六十七店、三百二十四店が三十三店で、千八十店の中百三十七店が賣店になつてゐる。一店の賣上高が平均五人半として全國で東京が二千五百一一十八店、大阪が二千二百五十五店、名古屋が千二百二十六店、靑森が三百三十店、大阪が三百九十六店となり、神戶も四百六十店で靑森が百五十三店、長崎が二百二十店で、京都が二百三十店である。それから東京と大阪の賣上高を總計すると千五百二十三店で全國で千三百",
+      "nonempty": true,
+      "no_mojibake": true,
+      "repetition_rate": 0.0,
+      "finite_repetition": true
+    },
+    {
+      "prompt": "吾輩は",
+      "text": "吾輩は寧ろ斯の如きことには少しも頓着せぬ方であていつに念信相正\nそのいはれいいづこおいまたむづかいちじるが、其所由を言はゞ、孰れの國に於ても亦難しとしてある。一言に言へおいせかいへいわじんるゐぜんたいくわんかうそのなんびといなば世界の平和人類全體の輿行は、其難しとするところである。人以爲へらく、せかいけいざいけんきうじんるゐぜんたいくわんかうたゞそのかうふく世界經濟の究と人類全體の輿行とは唯其幸福を增すばかりではなくして、そのかうふくえつかうふくせいくわつえうけん",
+      "nonempty": true,
+      "no_mojibake": true,
+      "repetition_rate": 0.026431718061673992,
+      "finite_repetition": true
+    },
+    {
+      "prompt": "人工知能とは",
+      "text": "人工知能とは比較的に平易に云ひあらはせる言葉であ)86(る。この問題に關しても、マウントは人間の知能の特性は、一定の時代に於ては種々の發達段階に於て現はれるものである、と述べてゐる。人間の知能の發達を決定する諸條件は、種々である。第一は、發達の段階が必ずしも常に同一でないと云ふことである。卽ち、野蠻人の間では、本能的運動がよく發達して居り、狩獵本能が發達するに過ぎない。また文明人に於ては、本能的運動は十分に發達してゐるけれども、知能の發達は甚だ低くて、未だ成熟期に達してをらない、と說いてゐる。第二は、心的諸能力は發達の程度を異に",
+      "nonempty": true,
+      "no_mojibake": true,
+      "repetition_rate": 0.0,
+      "finite_repetition": true
+    },
+    {
+      "prompt": "國家の將來は",
+      "text": "國家の將來は恐るべき混亂に陷るだらうと痛嘆したものである。かゝる社會運動が現實に現はれるに及んでは、流石のケマル·パシアも默視し難いものがあつたと見えて一策を考案し、早速これを政府に建議したのである。それは當時世界に流布された「社會主義者の政策」と稱する宣傳文であつた。これは當時フランスに於て出版された「萬國勞働者同盟」の宣傳文集「社會主義雜誌」から轉載したもので、その一節に次のやうな文字がある。世界の平和は、諸君が平和の爲めに努力する時に保たれる。諸君は將來永遠に平和を確保する爲めに、諸君が平和を目的とする",
+      "nonempty": true,
+      "no_mojibake": true,
+      "repetition_rate": 0.0,
+      "finite_repetition": true
+    }
+  ],
+  "automatic_rubric_pass_rate": 1.0
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "temperature": 0.8,
+  "top_k": 100,
+  "max_new_tokens": 128,
+  "seed": 1945
+}

inference.py

@@ -0,0 +1,14 @@
+import json, sys, torch
+from pathlib import Path
+from safetensors.torch import load_model
+from tokenizers import Tokenizer
+from model import GPT, GPTConfig
+root = Path(__file__).parent
+model = GPT(GPTConfig(**json.loads((root/'config.json').read_text())))
+load_model(model, root/'model.safetensors')
+model.eval(); tokenizer = Tokenizer.from_file(str(root/'tokenizer.json'))
+prompt = sys.argv[1] if len(sys.argv) > 1 else '文明開化とは'
+ids = tokenizer.encode(prompt).ids
+torch.manual_seed(1945)
+out = model.generate(torch.tensor(ids).unsqueeze(0), 64, 0.8, 100)[0].tolist()
+print(tokenizer.decode(out))

model.py

@@ -0,0 +1,203 @@
+from __future__ import annotations
+
+import math
+from dataclasses import asdict, dataclass
+
+import torch
+import torch.nn as nn
+from torch.nn import functional as F
+
+
+@dataclass
+class GPTConfig:
+    block_size: int = 1024
+    vocab_size: int = 8192
+    n_layer: int = 12
+    n_head: int = 12
+    n_embd: int = 768
+    dropout: float = 0.0
+    bias: bool = False
+
+
+class CausalSelfAttention(nn.Module):
+    def __init__(self, config: GPTConfig) -> None:
+        super().__init__()
+        if config.n_embd % config.n_head:
+            raise ValueError("embedding dimension must be divisible by number of heads")
+        self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd, bias=config.bias)
+        self.c_proj = nn.Linear(config.n_embd, config.n_embd, bias=config.bias)
+        self.attn_dropout = nn.Dropout(config.dropout)
+        self.resid_dropout = nn.Dropout(config.dropout)
+        self.n_head = config.n_head
+        self.n_embd = config.n_embd
+        self.dropout = config.dropout
+
+    def forward(self, value: torch.Tensor) -> torch.Tensor:
+        batch, time, channels = value.size()
+        query, key, val = self.c_attn(value).split(self.n_embd, dim=2)
+        head_size = channels // self.n_head
+        query = query.view(batch, time, self.n_head, head_size).transpose(1, 2)
+        key = key.view(batch, time, self.n_head, head_size).transpose(1, 2)
+        val = val.view(batch, time, self.n_head, head_size).transpose(1, 2)
+        attended = F.scaled_dot_product_attention(
+            query,
+            key,
+            val,
+            attn_mask=None,
+            dropout_p=self.dropout if self.training else 0,
+            is_causal=True,
+        )
+        attended = attended.transpose(1, 2).contiguous().view(batch, time, channels)
+        return self.resid_dropout(self.c_proj(attended))
+
+
+class MLP(nn.Module):
+    def __init__(self, config: GPTConfig) -> None:
+        super().__init__()
+        self.c_fc = nn.Linear(config.n_embd, 4 * config.n_embd, bias=config.bias)
+        self.gelu = nn.GELU()
+        self.c_proj = nn.Linear(4 * config.n_embd, config.n_embd, bias=config.bias)
+        self.dropout = nn.Dropout(config.dropout)
+
+    def forward(self, value: torch.Tensor) -> torch.Tensor:
+        return self.dropout(self.c_proj(self.gelu(self.c_fc(value))))
+
+
+class Block(nn.Module):
+    def __init__(self, config: GPTConfig) -> None:
+        super().__init__()
+        self.ln_1 = nn.LayerNorm(config.n_embd, bias=config.bias)
+        self.attn = CausalSelfAttention(config)
+        self.ln_2 = nn.LayerNorm(config.n_embd, bias=config.bias)
+        self.mlp = MLP(config)
+
+    def forward(self, value: torch.Tensor) -> torch.Tensor:
+        value = value + self.attn(self.ln_1(value))
+        return value + self.mlp(self.ln_2(value))
+
+
+class GPT(nn.Module):
+    def __init__(self, config: GPTConfig) -> None:
+        super().__init__()
+        self.config = config
+        self.transformer = nn.ModuleDict(
+            {
+                "wte": nn.Embedding(config.vocab_size, config.n_embd),
+                "wpe": nn.Embedding(config.block_size, config.n_embd),
+                "drop": nn.Dropout(config.dropout),
+                "h": nn.ModuleList([Block(config) for _ in range(config.n_layer)]),
+                "ln_f": nn.LayerNorm(config.n_embd, bias=config.bias),
+            }
+        )
+        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)
+        self.transformer.wte.weight = self.lm_head.weight
+        self.apply(self._init_weights)
+        for name, parameter in self.named_parameters():
+            if name.endswith("c_proj.weight"):
+                torch.nn.init.normal_(
+                    parameter, mean=0.0, std=0.02 / math.sqrt(2 * config.n_layer)
+                )
+
+    @staticmethod
+    def _init_weights(module: nn.Module) -> None:
+        if isinstance(module, nn.Linear):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
+            if module.bias is not None:
+                torch.nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+    def forward(
+        self, index: torch.Tensor, targets: torch.Tensor | None = None
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        _, time = index.shape
+        if time > self.config.block_size:
+            raise ValueError("sequence exceeds model block size")
+        positions = torch.arange(0, time, dtype=torch.long, device=index.device)
+        value = self.transformer.drop(self.transformer.wte(index) + self.transformer.wpe(positions))
+        for block in self.transformer.h:
+            value = block(value)
+        value = self.transformer.ln_f(value)
+        logits = self.lm_head(value)
+        loss = (
+            F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-1)
+            if targets is not None
+            else None
+        )
+        return logits, loss
+
+    @torch.no_grad()
+    def generate(
+        self,
+        index: torch.Tensor,
+        max_new_tokens: int,
+        temperature: float = 0.8,
+        top_k: int | None = 200,
+        top_p: float | None = None,
+        repetition_penalty: float = 1.0,
+        no_repeat_ngram_size: int | None = None,
+    ) -> torch.Tensor:
+        """Sample a continuation.
+
+        The defaults reproduce the temperature/top-k-only sampler used for the
+        phase 1-5 evaluations, so held-out numbers stay comparable. The
+        additional knobs are opt-in: phase-5 generation samples showed the
+        low-temperature repetition loop surviving the 345M -> 730M scale-up
+        (reports/phase5_generation_samples.md), and the sampler had no
+        repetition control of any kind to blame it on.
+        """
+        for _ in range(max_new_tokens):
+            cropped = index[:, -self.config.block_size :]
+            logits, _ = self(cropped)
+            logits = logits[:, -1, :]
+            if repetition_penalty != 1.0:
+                for row, sequence in enumerate(index):
+                    seen = torch.unique(sequence)
+                    scores = logits[row, seen]
+                    logits[row, seen] = torch.where(
+                        scores > 0, scores / repetition_penalty, scores * repetition_penalty
+                    )
+            logits = logits / temperature
+            if no_repeat_ngram_size:
+                for row, sequence in enumerate(index):
+                    for token in self._banned_ngram_tokens(sequence, no_repeat_ngram_size):
+                        logits[row, token] = -float("Inf")
+            if top_k is not None:
+                values, _ = torch.topk(logits, min(top_k, logits.size(-1)))
+                logits[logits < values[:, [-1]]] = -float("Inf")
+            if top_p is not None:
+                ordered, order = torch.sort(logits, descending=True, dim=-1)
+                ranked = F.softmax(ordered, dim=-1)
+                # Drop a token once the mass ahead of it already covers top_p,
+                # which always keeps at least the most likely token.
+                remove = ranked.cumsum(dim=-1) - ranked >= top_p
+                logits = logits.masked_fill(
+                    torch.zeros_like(remove).scatter(1, order, remove), -float("Inf")
+                )
+            probabilities = F.softmax(logits, dim=-1)
+            index = torch.cat((index, torch.multinomial(probabilities, num_samples=1)), dim=1)
+        return index
+
+    @staticmethod
+    def _banned_ngram_tokens(sequence: torch.Tensor, size: int) -> list[int]:
+        """Tokens that would repeat an n-gram already present in ``sequence``."""
+        if size < 2 or len(sequence) < size:
+            return []
+        tokens = sequence.tolist()
+        prefix = tuple(tokens[-(size - 1) :])
+        banned = [
+            tokens[start + size - 1]
+            for start in range(len(tokens) - size + 1)
+            if tuple(tokens[start : start + size - 1]) == prefix
+        ]
+        return banned
+
+    def parameter_count(self, non_embedding: bool = False) -> int:
+        count = sum(parameter.numel() for parameter in self.parameters())
+        if non_embedding:
+            count -= self.transformer.wpe.weight.numel()
+        return count
+
+    def config_dict(self) -> dict[str, object]:
+        return asdict(self.config)
+

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40d1cf331c3b991a714cde72582e7a60eacb21aa0fab5fd873f9125dd48bb1e5
+size 4184290360

training_config.yaml

@@ -0,0 +1,137 @@
+extends: configs/base.yaml
+phase: phase6
+# Phase 6 reopens the corpus. Phase 5 concluded that this closed corpus was near
+# an entropy floor because two single-lever moves (data 2.5x, model 2.1x) both
+# bought ~3.5% BPB. reports/phase6_rejection_review.json shows that conclusion
+# rested on a corpus that was never exhausted: 141,521 of the frozen pool's
+# 241,430 candidates (58.6%) were dropped as short_document, and the filter
+# responsible was measuring domain novelty rather than OCR corruption -- its
+# reference 3-gram set came from the Aozora literary corpus alone (2.88M
+# 3-grams). Re-scoring 300 re-downloaded quarantined documents against a
+# reference that also includes accepted NDL text recovers 63% of them, worth an
+# estimated +1.08B prose tokens (train 2.742B -> 3.818B).
+#
+# 6.2 (this config's acquisition stage) only re-downloads. Cleaning waits for
+# the 6.1 filter redesign, so raw ZIPs are NOT pruned in between -- pruning
+# before the new filter exists would force a second five-day re-download.
+data:
+  ndl_metadata_dir: data/raw/ndl/metadata
+  ndl_metadata_glob: data/raw/ndl/metadata/books_*.zip
+  ndl_metadata_urls:
+    - https://dl.ndl.go.jp/static/files/dataset/dataset_202602_t_internet_01.zip
+    - https://dl.ndl.go.jp/static/files/dataset/dataset_202602_t_internet_02.zip
+    - https://dl.ndl.go.jp/static/files/dataset/dataset_202602_t_internet_03.zip
+    - https://dl.ndl.go.jp/static/files/dataset/dataset_202602_t_internet_04.zip
+    - https://dl.ndl.go.jp/static/files/dataset/dataset_202602_t_internet_05.zip
+  # The frozen pool stays the source of truth for order and rights evidence.
+  ndl_selection_pool_manifest: data/processed/ndl/selection.jsonl
+  # ndl-download walks ndl_selection_manifest, so phase 6 points it at the
+  # quarantined-only subset (in pool order) built by
+  # scripts/build_phase6_selection.py. Existing ZIPs are skipped by checksum, so
+  # the run stays idempotent and resumable. Do NOT run ndl-select against this
+  # config: the pool is frozen and ndl-select would overwrite this subset.
+  ndl_selection_manifest: data/processed/phase6/selection_quarantined.jsonl
+  ndl_download_manifest: data/processed/phase6/ndl_downloads.jsonl
+  ndl_raw_dir: data/raw/ndl/books
+  ndl_clean_dir: data/clean/ndl
+  ndl_clean_manifest: data/processed/phase6/ndl_clean.jsonl
+  # Phase-5's manifest is already a superset carrying phases 1-4 forward.
+  ndl_clean_reuse_manifest: data/processed/phase5/ndl_clean.jsonl
+  ndl_clean_report: phase6_ndl_cleaning.json
+  ndl_quality_sample_report: phase6_ndl_quality_sample.jsonl
+  ndl_quality_exclude_manifest: data/processed/phase5/ndl_clean.jsonl
+  ndl_target_documents: 241430
+  ndl_download_delay_seconds: 0.2
+  ndl_download_retries: 3
+  ndl_max_download_seconds: 600
+  # 6.3 froze phase-6 splits, so this now points at them (phase-4's manifest stays
+  # on disk untouched). The fixed 24-work phase-1 test set carries over unchanged.
+  split_manifest: data/processed/phase6/splits.jsonl
+  ndl_max_abnormal_ngram_rate: 0.38
+  ndl_max_latin_rate: 0.005
+  # 6.1 filter redesign. The abnormal-ngram reference was built from the Aozora
+  # literary corpus alone (2,879,238 3-grams), which made that filter a
+  # domain-novelty detector rather than an OCR-corruption one: re-scoring 300
+  # re-downloaded quarantined documents against a reference that also includes
+  # accepted NDL text recovers 63% of them (reports/phase6_rejection_review.json).
+  # Order is fixed and each manifest is capped, so the reference itself is
+  # reproducible and can be frozen alongside the thresholds.
+  ndl_reference_manifests:
+    - data/processed/aozora/clean.jsonl
+    - data/processed/phase5/ndl_clean.jsonl
+  ndl_reference_max_chars: 200000000
+  # Widening the reference lets statistics tables and directories through, since
+  # abnormal_ngram_rate had been rejecting them as a side effect. Calibrated in
+  # reports/phase6_filter_calibration.json: a 0.18 numeral-run ratio fires on
+  # 0.071% of already-accepted characters (inside 6.1's 0.5% shrink gate) and
+  # catches 100% of the pages the audit triaged as tables.
+  #
+  # No kana-run rule ships. The draft "garbled" detector was measured against the
+  # same two sets and no run length separates ruby-contaminated OCR from ordinary
+  # 文語体 (run>=8 at ratio>0.05 flags 54% of accepted characters; run>=20 at
+  # ratio>0.20 still flags 8% while recall falls to 41%). 文語体 simply contains
+  # long kana runs. mojibake_page continues to handle genuine corruption.
+  ndl_max_digit_run_rate: 0.18
+  # 6.3 の phase6-merge が書き出す、ページ規則適用後のcleanテキスト。
+  # data/clean/ndl を直接書き換えない: 記録済み clean_sha256 と公開済み v1/v2 の
+  # 基礎が変わるため（reports/phase6_cleaning_plan.json で却下した案）。
+  ndl_filtered_dir: data/clean/ndl_phase6
+  # 仮名3-gram参照（青空文庫のみ、凍結）。kana_soup の判定に使う。
+  # 生成: uv run python scripts/calibrate_phase6_kana_soup.py
+  ndl_kana_reference: artifacts/phase6/kana_reference.txt
+tokenizer:
+  # Retrained in 6.3 over the expanded corpus. BPB is per byte, so changing the
+  # tokenizer does not break comparability with phases 1-5.
+  vocab_size: 16384
+  directory: artifacts/phase6/tokenizer
+  max_train_chars: 200000000
+tokenized:
+  directory: data/processed/phase6/tokenized
+model:
+  # 6.4 co-scaling. フェーズ5は730M（36層/20head/1280）でデータ据え置き、
+  # フェーズ4は345Mでデータ2.5倍。単独レバーの改善率がほぼ一致した
+  # （+3.46% / +3.53%）ことがフェーズ6の出発点なので、ここは初めて両方を同時に
+  # 動かす: データ 2.742B → 実測3.8B級、モデル 730M → 約1.05B。
+  # head_dim = 1536/24 = 64 でフェーズ3〜5と同じ比を保つ。
+  # 実数は `tcja --config configs/phase6.yaml parameter-count --vocab-size 16384`
+  # で確認し reports/runs/parameter-count.json へ残す。
+  n_layer: 36
+  n_head: 24
+  n_embd: 1536
+  block_size: 1024
+  dropout: 0.0
+training:
+  device: cuda
+  dtype: bfloat16
+  # ユーザー判断(2026-07-30)で batch16 x accum16。tokens_per_step は 262,144 で
+  # 変わらないので max_iters もそのまま。
+  # 当初は batch8 x accum32 を第一候補にしていたが、accumulation回数が半分になる
+  # 分こちらが速いはずで、本学習163時間に対しては数%でも十数時間効く。
+  # フェーズ5は同じ形（batch16 x accum16）を730Mで走らせて peak 43.8GB・余裕48.4%
+  # だった。パラメータは1.43倍だが、支配的なoptimizer stateでも +9GB程度の見込み。
+  # **6.5のflightで実測する。VRAM余裕が10%未満なら batch8 x accum32 へ戻す。**
+  batch_size: 16
+  gradient_accumulation_steps: 16
+  # FINALIZED (2026-07-30、reports/phase6_tokenization.json): train実測
+  # 3,953,739,596 tokens × 3ep = 11,861,218,788 処理トークン ÷ 262,144 tok/step
+  # = 45,254.9 → 45,300 iters（3.003ep、処理 11,875,123,200）。
+  # tokens/param 11.35 は フェーズ5の11.3とほぼ同じで、データとモデルを同率で
+  # 伸ばした結果になっている。
+  max_iters: 45300
+  learning_rate: 0.00025
+  min_lr: 0.000025
+  warmup_iters: 300
+  lr_decay_iters: 45300
+  eval_interval: 500
+  eval_iters: 50
+  checkpoint_interval: 500
+  grad_clip: 1.0
+  out_dir: artifacts/phase6
+budget:
+  # フェーズ5実測 9.065 s/step（730M・A100 80GB・262,144 tok/step）= 104.1M tok/h。
+  # FLOPsはパラメータ数にほぼ比例するので 1.05B では約72M tok/h。
+  # 3ep（約11.4B処理）で約158h・$220（$1.39/h）。上限は余裕を見て置く。
+  minimum_train_tokens: 3500000000
+  maximum_hours: 190
+  maximum_usd: 270
+  gpu_hourly_usd: 1.39

training_report.json

@@ -0,0 +1,840 @@
+{
+  "iterations": 45300,
+  "parameter_count": 1046066688,
+  "tokens_per_step": 262144,
+  "processed_tokens": 11875123200,
+  "elapsed_seconds": 529599.8804617687,
+  "best_validation_loss": 3.3402066230773926,
+  "history": [
+    {
+      "iteration": 500,
+      "train_loss": 6.373349666595459,
+      "validation_loss": 6.335325717926025,
+      "learning_rate": 0.00024998914333562564,
+      "elapsed_seconds": 5767.518404414877,
+      "projected_total_hours": 145.1492131777744,
+      "projected_total_cost_usd": 201.75740631710642
+    },
+    {
+      "iteration": 1000,
+      "train_loss": 4.988532066345215,
+      "validation_loss": 4.971802234649658,
+      "learning_rate": 0.0002498660738422774,
+      "elapsed_seconds": 11607.223960213829,
+      "projected_total_hours": 146.05756816602403,
+      "projected_total_cost_usd": 203.02001975077337
+    },
+    {
+      "iteration": 1500,
+      "train_loss": 4.607588768005371,
+      "validation_loss": 4.590364456176758,
+      "learning_rate": 0.0002496061035966192,
+      "elapsed_seconds": 17444.452618559822,
+      "projected_total_hours": 146.33957474458518,
+      "projected_total_cost_usd": 203.41200889497338
+    },
+    {
+      "iteration": 2000,
+      "train_loss": 4.4110026359558105,
+      "validation_loss": 4.396660327911377,
+      "learning_rate": 0.00024920954933234997,
+      "elapsed_seconds": 23283.12628326984,
+      "projected_total_hours": 146.4896695322394,
+      "projected_total_cost_usd": 203.62064064981274
+    },
+    {
+      "iteration": 2500,
+      "train_loss": 4.2808990478515625,
+      "validation_loss": 4.268635272979736,
+      "learning_rate": 0.0002486768941897562,
+      "elapsed_seconds": 29123.792723594932,
+      "projected_total_hours": 146.58975670876117,
+      "projected_total_cost_usd": 203.759761825178
+    },
+    {
+      "iteration": 3000,
+      "train_loss": 4.190364837646484,
+      "validation_loss": 4.175224781036377,
+      "learning_rate": 0.00024800878712707996,
+      "elapsed_seconds": 34969.42657046695,
+      "projected_total_hours": 146.67731700390306,
+      "projected_total_cost_usd": 203.88147063542525
+    },
+    {
+      "iteration": 3500,
+      "train_loss": 4.114880561828613,
+      "validation_loss": 4.1017656326293945,
+      "learning_rate": 0.00024720604212986305,
+      "elapsed_seconds": 40825.30353703792,
+      "projected_total_hours": 146.77668652601727,
+      "projected_total_cost_usd": 204.01959427116398
+    },
+    {
+      "iteration": 4000,
+      "train_loss": 4.059500217437744,
+      "validation_loss": 4.044155120849609,
+      "learning_rate": 0.00024626963721923123,
+      "elapsed_seconds": 46672.65582661284,
+      "projected_total_hours": 146.8243964545529,
+      "projected_total_cost_usd": 204.08591107182852
+    },
+    {
+      "iteration": 4500,
+      "train_loss": 8.15172290802002,
+      "validation_loss": 8.112296104431152,
+      "learning_rate": 0.00024520071326032596,
+      "elapsed_seconds": 52528.73340205988,
+      "projected_total_hours": 146.8859026613156,
+      "projected_total_cost_usd": 204.17140469922867
+    },
+    {
+      "iteration": 5000,
+      "train_loss": 4.006167411804199,
+      "validation_loss": 3.99216365814209,
+      "learning_rate": 0.0002440005725723361,
+      "elapsed_seconds": 58357.20548674604,
+      "projected_total_hours": 146.86563380831086,
+      "projected_total_cost_usd": 204.1432309935521
+    },
+    {
+      "iteration": 5500,
+      "train_loss": 3.950077772140503,
+      "validation_loss": 3.9378490447998047,
+      "learning_rate": 0.00024267067734182245,
+      "elapsed_seconds": 64213.86831258284,
+      "projected_total_hours": 146.91354720000012,
+      "projected_total_cost_usd": 204.20983060800015
+    },
+    {
+      "iteration": 6000,
+      "train_loss": 3.9144349098205566,
+      "validation_loss": 3.9045145511627197,
+      "learning_rate": 0.0002412126478412687,
+      "elapsed_seconds": 70062.76474088593,
+      "projected_total_hours": 146.93718716491355,
+      "projected_total_cost_usd": 204.24269015922982
+    },
+    {
+      "iteration": 6500,
+      "train_loss": 3.884676933288574,
+      "validation_loss": 3.875600814819336,
+      "learning_rate": 0.00023962826045502908,
+      "elapsed_seconds": 75910.68644157494,
+      "projected_total_hours": 146.9553032394592,
+      "projected_total_cost_usd": 204.26787150284827
+    },
+    {
+      "iteration": 7000,
+      "train_loss": 3.8586838245391846,
+      "validation_loss": 3.849705219268799,
+      "learning_rate": 0.00023791944551507753,
+      "elapsed_seconds": 81744.92833834188,
+      "projected_total_hours": 146.94624022725742,
+      "projected_total_cost_usd": 204.2552739158878
+    },
+    {
+      "iteration": 7500,
+      "train_loss": 3.8353729248046875,
+      "validation_loss": 3.826043128967285,
+      "learning_rate": 0.00023608828494919563,
+      "elapsed_seconds": 87577.26776477997,
+      "projected_total_hours": 146.93519369424195,
+      "projected_total_cost_usd": 204.23991923499628
+    },
+    {
+      "iteration": 8000,
+      "train_loss": 3.810929775238037,
+      "validation_loss": 3.804882764816284,
+      "learning_rate": 0.00023413700974446423,
+      "elapsed_seconds": 93414.80510617001,
+      "projected_total_hours": 146.93370386491324,
+      "projected_total_cost_usd": 204.2378483722294
+    },
+    {
+      "iteration": 8500,
+      "train_loss": 3.7886407375335693,
+      "validation_loss": 3.783992290496826,
+      "learning_rate": 0.00023206799722914946,
+      "elapsed_seconds": 99244.98037643894,
+      "projected_total_hours": 146.92149055727725,
+      "projected_total_cost_usd": 204.22087187461537
+    },
+    {
+      "iteration": 9000,
+      "train_loss": 3.766958713531494,
+      "validation_loss": 3.7680282592773438,
+      "learning_rate": 0.0002298837681762942,
+      "elapsed_seconds": 105085.86910354486,
+      "projected_total_hours": 146.92561328365994,
+      "projected_total_cost_usd": 204.2266024642873
+    },
+    {
+      "iteration": 9500,
+      "train_loss": 3.7498161792755127,
+      "validation_loss": 3.7501485347747803,
+      "learning_rate": 0.00022758698373254472,
+      "elapsed_seconds": 110928.114993172,
+      "projected_total_hours": 146.93109968393833,
+      "projected_total_cost_usd": 204.23422856067427
+    },
+    {
+      "iteration": 10000,
+      "train_loss": 3.730133056640625,
+      "validation_loss": 3.7339835166931152,
+      "learning_rate": 0.00022518044217595314,
+      "elapsed_seconds": 116778.32068712497,
+      "projected_total_hours": 146.94605353129893,
+      "projected_total_cost_usd": 204.2550144085055
+    },
+    {
+      "iteration": 10500,
+      "train_loss": 3.7131927013397217,
+      "validation_loss": 3.7187798023223877,
+      "learning_rate": 0.00022266707550670694,
+      "elapsed_seconds": 122624.58746308996,
+      "projected_total_hours": 146.9548627533856,
+      "projected_total_cost_usd": 204.26725922720598
+    },
+    {
+      "iteration": 11000,
+      "train_loss": 3.6982581615448,
+      "validation_loss": 3.7051968574523926,
+      "learning_rate": 0.0002200499458749381,
+      "elapsed_seconds": 128481.97054895293,
+      "projected_total_hours": 146.9755875219083,
+      "projected_total_cost_usd": 204.2960666554525
+    },
+    {
+      "iteration": 11500,
+      "train_loss": 3.681401252746582,
+      "validation_loss": 3.6900885105133057,
+      "learning_rate": 0.00021733224184996498,
+      "elapsed_seconds": 134351.35519558704,
+      "projected_total_hours": 147.00764227922932,
+      "projected_total_cost_usd": 204.34062276812872
+    },
+    {
+      "iteration": 12000,
+      "train_loss": 3.667637825012207,
+      "validation_loss": 3.6769940853118896,
+      "learning_rate": 0.0002145172745355119,
+      "elapsed_seconds": 140210.84609409282,
+      "projected_total_hours": 147.02665111255567,
+      "projected_total_cost_usd": 204.36704504645238
+    },
+    {
+      "iteration": 12500,
+      "train_loss": 3.6528677940368652,
+      "validation_loss": 3.6665825843811035,
+      "learning_rate": 0.00021160847353563888,
+      "elapsed_seconds": 146071.26893039,
+      "projected_total_hours": 147.04507738992595,
+      "projected_total_cost_usd": 204.39265757199706
+    },
+    {
+      "iteration": 13000,
+      "train_loss": 3.639846086502075,
+      "validation_loss": 3.653087854385376,
+      "learning_rate": 0.000208609382776298,
+      "elapsed_seconds": 151925.76889001392,
+      "projected_total_hours": 147.05635322046217,
+      "projected_total_cost_usd": 204.40833097644241
+    },
+    {
+      "iteration": 13500,
+      "train_loss": 3.629678964614868,
+      "validation_loss": 3.6425936222076416,
+      "learning_rate": 0.0002055236561876049,
+      "elapsed_seconds": 157782.4660434397,
+      "projected_total_hours": 147.06884180592218,
+      "projected_total_cost_usd": 204.4256901102318
+    },
+    {
+      "iteration": 14000,
+      "train_loss": 3.6172244548797607,
+      "validation_loss": 3.634115219116211,
+      "learning_rate": 0.00020235505325208816,
+      "elapsed_seconds": 163627.8123262967,
+      "projected_total_hours": 147.07023607899285,
+      "projected_total_cost_usd": 204.42762814980006
+    },
+    {
+      "iteration": 14500,
+      "train_loss": 3.602062463760376,
+      "validation_loss": 3.626448154449463,
+      "learning_rate": 0.00019910743442433889,
+      "elapsed_seconds": 169474.427039932,
+      "projected_total_hours": 147.07263495994098,
+      "projected_total_cost_usd": 204.43096259431795
+    },
+    {
+      "iteration": 15000,
+      "train_loss": 3.596639394760132,
+      "validation_loss": 3.617069721221924,
+      "learning_rate": 0.00019578475642764132,
+      "elapsed_seconds": 175318.30801315093,
+      "projected_total_hours": 147.07258061103215,
+      "projected_total_cost_usd": 204.43088704933467
+    },
+    {
+      "iteration": 15500,
+      "train_loss": 3.5822103023529053,
+      "validation_loss": 3.604166030883789,
+      "learning_rate": 0.00019239106743331515,
+      "elapsed_seconds": 181170.80125680007,
+      "projected_total_hours": 147.07952145041295,
+      "projected_total_cost_usd": 204.440534816074
+    },
+    {
+      "iteration": 16000,
+      "train_loss": 3.5695128440856934,
+      "validation_loss": 3.594214677810669,
+      "learning_rate": 0.0001889305021286422,
+      "elapsed_seconds": 187022.97385376412,
+      "projected_total_hours": 147.0857763120749,
+      "projected_total_cost_usd": 204.4492290737841
+    },
+    {
+      "iteration": 16500,
+      "train_loss": 3.5584664344787598,
+      "validation_loss": 3.585881233215332,
+      "learning_rate": 0.00018540727667938712,
+      "elapsed_seconds": 192868.98684213776,
+      "projected_total_hours": 147.08695461193335,
+      "projected_total_cost_usd": 204.45086691058734
+    },
+    {
+      "iteration": 17000,
+      "train_loss": 3.5498061180114746,
+      "validation_loss": 3.5772714614868164,
+      "learning_rate": 0.0001818256835930484,
+      "elapsed_seconds": 198710.49392023915,
+      "projected_total_hours": 147.08472834292212,
+      "projected_total_cost_usd": 204.44777239666172
+    },
+    {
+      "iteration": 17500,
+      "train_loss": 3.538649797439575,
+      "validation_loss": 3.5700178146362305,
+      "learning_rate": 0.0001781900864890997,
+      "elapsed_seconds": 204562.4459545957,
+      "projected_total_hours": 147.09013971020931,
+      "projected_total_cost_usd": 204.45529419719094
+    },
+    {
+      "iteration": 18000,
+      "train_loss": 3.5280675888061523,
+      "validation_loss": 3.563185453414917,
+      "learning_rate": 0.00017450491478259136,
+      "elapsed_seconds": 210410.987407709,
+      "projected_total_hours": 147.09286619705583,
+      "projected_total_cost_usd": 204.45908401390759
+    },
+    {
+      "iteration": 18500,
+      "train_loss": 3.517913579940796,
+      "validation_loss": 3.5548534393310547,
+      "learning_rate": 0.0001707746582875906,
+      "elapsed_seconds": 216262.0165603701,
+      "projected_total_hours": 147.09713739016163,
+      "projected_total_cost_usd": 204.46502097232465
+    },
+    {
+      "iteration": 19000,
+      "train_loss": 3.50887393951416,
+      "validation_loss": 3.5479366779327393,
+      "learning_rate": 0.00016700386174703456,
+      "elapsed_seconds": 222117.43233659677,
+      "projected_total_hours": 147.10408895976366,
+      "projected_total_cost_usd": 204.47468365407147
+    },
+    {
+      "iteration": 19500,
+      "train_loss": 3.5005223751068115,
+      "validation_loss": 3.54071044921875,
+      "learning_rate": 0.00016319711929566126,
+      "elapsed_seconds": 227972.2531674751,
+      "projected_total_hours": 147.11030012089205,
+      "projected_total_cost_usd": 204.48331716803995
+    },
+    {
+      "iteration": 20000,
+      "train_loss": 3.4909307956695557,
+      "validation_loss": 3.531663179397583,
+      "learning_rate": 0.00015935906886276402,
+      "elapsed_seconds": 233830.57044673292,
+      "projected_total_hours": 147.11840057273616,
+      "projected_total_cost_usd": 204.49457679610325
+    },
+    {
+      "iteration": 20500,
+      "train_loss": 3.486023187637329,
+      "validation_loss": 3.5255727767944336,
+      "learning_rate": 0.0001554943865215889,
+      "elapsed_seconds": 239679.68671904784,
+      "projected_total_hours": 147.12045810803343,
+      "projected_total_cost_usd": 204.49743677016644
+    },
+    {
+      "iteration": 21000,
+      "train_loss": 3.4791877269744873,
+      "validation_loss": 3.5202720165252686,
+      "learning_rate": 0.00015160778079226003,
+      "elapsed_seconds": 245518.76395338494,
+      "projected_total_hours": 147.1164022101632,
+      "projected_total_cost_usd": 204.4917990721268
+    },
+    {
+      "iteration": 21500,
+      "train_loss": 3.469804048538208,
+      "validation_loss": 3.5100693702697754,
+      "learning_rate": 0.0001477039869051728,
+      "elapsed_seconds": 251360.50779337808,
+      "projected_total_hours": 147.114095646512,
+      "projected_total_cost_usd": 204.48859294865167
+    },
+    {
+      "iteration": 22000,
+      "train_loss": 3.4615044593811035,
+      "validation_loss": 3.505176067352295,
+      "learning_rate": 0.00014378776103184527,
+      "elapsed_seconds": 257201.9809560948,
+      "projected_total_hours": 147.1117391074633,
+      "projected_total_cost_usd": 204.485317359374
+    },
+    {
+      "iteration": 22500,
+      "train_loss": 3.4569902420043945,
+      "validation_loss": 3.499898672103882,
+      "learning_rate": 0.00013986387449025582,
+      "elapsed_seconds": 263036.7369214799,
+      "projected_total_hours": 147.10573064867953,
+      "projected_total_cost_usd": 204.47696560166452
+    },
+    {
+      "iteration": 23000,
+      "train_loss": 3.4522955417633057,
+      "validation_loss": 3.4949960708618164,
+      "learning_rate": 0.00013593710793172693,
+      "elapsed_seconds": 268878.98143927194,
+      "projected_total_hours": 147.10408042510892,
+      "projected_total_cost_usd": 204.47467179090137
+    },
+    {
+      "iteration": 23500,
+      "train_loss": 3.4453978538513184,
+      "validation_loss": 3.488229751586914,
+      "learning_rate": 0.0001320122455164382,
+      "elapsed_seconds": 274705.03559077205,
+      "projected_total_hours": 147.09383111420772,
+      "projected_total_cost_usd": 204.46042524874872
+    },
+    {
+      "iteration": 24000,
+      "train_loss": 3.43698787689209,
+      "validation_loss": 3.482897996902466,
+      "learning_rate": 0.00012809406908466384,
+      "elapsed_seconds": 280538.5956518408,
+      "projected_total_hours": 147.08794424801377,
+      "projected_total_cost_usd": 204.45224250473913
+    },
+    {
+      "iteration": 24500,
+      "train_loss": 3.426966667175293,
+      "validation_loss": 3.475834846496582,
+      "learning_rate": 0.00012418735233083727,
+      "elapsed_seconds": 286381.1681528948,
+      "projected_total_hours": 147.08692650029633,
+      "projected_total_cost_usd": 204.45082783541187
+    },
+    {
+      "iteration": 25000,
+      "train_loss": 3.4196131229400635,
+      "validation_loss": 3.471196174621582,
+      "learning_rate": 0.00012029685498753944,
+      "elapsed_seconds": 292214.37232685275,
+      "projected_total_hours": 147.08123407118256,
+      "projected_total_cost_usd": 204.44291535894374
+    },
+    {
+      "iteration": 25500,
+      "train_loss": 3.4136993885040283,
+      "validation_loss": 3.4657559394836426,
+      "learning_rate": 0.00011642731702649787,
+      "elapsed_seconds": 298058.5812796368,
+      "projected_total_hours": 147.08119533733714,
+      "projected_total_cost_usd": 204.4428615188986
+    },
+    {
+      "iteration": 26000,
+      "train_loss": 3.4060094356536865,
+      "validation_loss": 3.460638999938965,
+      "learning_rate": 0.0001125834528836616,
+      "elapsed_seconds": 303898.8813599739,
+      "projected_total_hours": 147.07926629921815,
+      "projected_total_cost_usd": 204.4401801559132
+    },
+    {
+      "iteration": 26500,
+      "train_loss": 3.397285223007202,
+      "validation_loss": 3.456627130508423,
+      "learning_rate": 0.00010876994571538669,
+      "elapsed_seconds": 309734.5375928958,
+      "projected_total_hours": 147.0752049576329,
+      "projected_total_cost_usd": 204.43453489110973
+    },
+    {
+      "iteration": 27000,
+      "train_loss": 3.3896384239196777,
+      "validation_loss": 3.450105667114258,
+      "learning_rate": 0.00010499144169273202,
+      "elapsed_seconds": 315585.65890582977,
+      "projected_total_hours": 147.0785015270997,
+      "projected_total_cost_usd": 204.43911712266856
+    },
+    {
+      "iteration": 27500,
+      "train_loss": 3.3839428424835205,
+      "validation_loss": 3.444591999053955,
+      "learning_rate": 0.0001012525443408152,
+      "elapsed_seconds": 321438.789508109,
+      "projected_total_hours": 147.08259762340742,
+      "projected_total_cost_usd": 204.4448106965363
+    },
+    {
+      "iteration": 28000,
+      "train_loss": 3.3783419132232666,
+      "validation_loss": 3.4397692680358887,
+      "learning_rate": 9.755780893012659e-05,
+      "elapsed_seconds": 327273.82601866405,
+      "projected_total_hours": 147.0784158595782,
+      "projected_total_cost_usd": 204.43899804481367
+    },
+    {
+      "iteration": 28500,
+      "train_loss": 3.3702282905578613,
+      "validation_loss": 3.4354019165039062,
+      "learning_rate": 9.391173692663375e-05,
+      "elapsed_seconds": 333119.06978819985,
+      "projected_total_hours": 147.07888753806483,
+      "projected_total_cost_usd": 204.43965367791012
+    },
+    {
+      "iteration": 29000,
+      "train_loss": 3.364657402038574,
+      "validation_loss": 3.432204246520996,
+      "learning_rate": 9.03187705074386e-05,
+      "elapsed_seconds": 338967.86247838195,
+      "projected_total_hours": 147.08088285699907,
+      "projected_total_cost_usd": 204.4424271712287
+    },
+    {
+      "iteration": 29500,
+      "train_loss": 3.361586332321167,
+      "validation_loss": 3.4272539615631104,
+      "learning_rate": 8.678328714866887e-05,
+      "elapsed_seconds": 344816.71290178783,
+      "projected_total_hours": 147.08283516432195,
+      "projected_total_cost_usd": 204.4451408784075
+    },
+    {
+      "iteration": 30000,
+      "train_loss": 3.3544201850891113,
+      "validation_loss": 3.4213171005249023,
+      "learning_rate": 8.330959429219775e-05,
+      "elapsed_seconds": 350665.20799918985,
+      "projected_total_hours": 147.08457335521572,
+      "projected_total_cost_usd": 204.44755696374983
+    },
+    {
+      "iteration": 30500,
+      "train_loss": 3.3491179943084717,
+      "validation_loss": 3.4158291816711426,
+      "learning_rate": 7.99019240976895e-05,
+      "elapsed_seconds": 356519.57499028975,
+      "projected_total_hours": 147.0886771134802,
+      "projected_total_cost_usd": 204.45326118773747
+    },
+    {
+      "iteration": 31000,
+      "train_loss": 3.34259033203125,
+      "validation_loss": 3.414461135864258,
+      "learning_rate": 7.656442828636477e-05,
+      "elapsed_seconds": 362371.10408779606,
+      "projected_total_hours": 147.09149655176665,
+      "projected_total_cost_usd": 204.45718020695563
+    },
+    {
+      "iteration": 31500,
+      "train_loss": 3.336934804916382,
+      "validation_loss": 3.4084677696228027,
+      "learning_rate": 7.330117308276784e-05,
+      "elapsed_seconds": 368224.406660873,
+      "projected_total_hours": 147.0949349359572,
+      "projected_total_cost_usd": 204.46195956098052
+    },
+    {
+      "iteration": 32000,
+      "train_loss": 3.332810878753662,
+      "validation_loss": 3.4053311347961426,
+      "learning_rate": 7.011613426069842e-05,
+      "elapsed_seconds": 374078.46307888115,
+      "projected_total_hours": 147.09856230445587,
+      "projected_total_cost_usd": 204.46700160319364
+    },
+    {
+      "iteration": 32500,
+      "train_loss": 3.325847864151001,
+      "validation_loss": 3.3993771076202393,
+      "learning_rate": 6.701319229934343e-05,
+      "elapsed_seconds": 379934.33091358515,
+      "projected_total_hours": 147.1027794050035,
+      "projected_total_cost_usd": 204.47286337295486
+    },
+    {
+      "iteration": 33000,
+      "train_loss": 3.3214898109436035,
+      "validation_loss": 3.3968589305877686,
+      "learning_rate": 6.39961276555114e-05,
+      "elapsed_seconds": 385785.82466378994,
+      "projected_total_hours": 147.1052008187684,
+      "projected_total_cost_usd": 204.47622913808806
+    },
+    {
+      "iteration": 33500,
+      "train_loss": 3.3171088695526123,
+      "validation_loss": 3.393434762954712,
+      "learning_rate": 6.106861615772748e-05,
+      "elapsed_seconds": 391644.3146269447,
+      "projected_total_hours": 147.11017788226033,
+      "projected_total_cost_usd": 204.48314725634185
+    },
+    {
+      "iteration": 34000,
+      "train_loss": 3.312814712524414,
+      "validation_loss": 3.388615131378174,
+      "learning_rate": 5.823422452780332e-05,
+      "elapsed_seconds": 397495.1632606508,
+      "projected_total_hours": 147.11218052048596,
+      "projected_total_cost_usd": 204.48593092347548
+    },
+    {
+      "iteration": 34500,
+      "train_loss": 3.305443048477173,
+      "validation_loss": 3.3842084407806396,
+      "learning_rate": 5.549640603533529e-05,
+      "elapsed_seconds": 403351.415478304,
+      "projected_total_hours": 147.11609598363265,
+      "projected_total_cost_usd": 204.49137341724938
+    },
+    {
+      "iteration": 35000,
+      "train_loss": 3.299060583114624,
+      "validation_loss": 3.3820013999938965,
+      "learning_rate": 5.285849629042789e-05,
+      "elapsed_seconds": 409202.1039497829,
+      "projected_total_hours": 147.11789927718385,
+      "projected_total_cost_usd": 204.49387999528554
+    },
+    {
+      "iteration": 35500,
+      "train_loss": 3.2942333221435547,
+      "validation_loss": 3.3802318572998047,
+      "learning_rate": 5.032370917976645e-05,
+      "elapsed_seconds": 415038.40922521707,
+      "projected_total_hours": 147.11455350471311,
+      "projected_total_cost_usd": 204.4892293715512
+    },
+    {
+      "iteration": 36000,
+      "train_loss": 3.2915239334106445,
+      "validation_loss": 3.375929594039917,
+      "learning_rate": 4.7895132950991336e-05,
+      "elapsed_seconds": 420878.96743322304,
+      "projected_total_hours": 147.11278722781637,
+      "projected_total_cost_usd": 204.48677424666474
+    },
+    {
+      "iteration": 36500,
+      "train_loss": 3.284877300262451,
+      "validation_loss": 3.3734710216522217,
+      "learning_rate": 4.55757264501437e-05,
+      "elapsed_seconds": 426712.58293662686,
+      "projected_total_hours": 147.1086758525814,
+      "projected_total_cost_usd": 204.48105943508813
+    },
+    {
+      "iteration": 37000,
+      "train_loss": 3.2819366455078125,
+      "validation_loss": 3.370877742767334,
+      "learning_rate": 4.336831551676778e-05,
+      "elapsed_seconds": 432558.64357199613,
+      "projected_total_hours": 147.10890806164736,
+      "projected_total_cost_usd": 204.4813822056898
+    },
+    {
+      "iteration": 37500,
+      "train_loss": 3.2778007984161377,
+      "validation_loss": 3.368626594543457,
+      "learning_rate": 4.12755895410602e-05,
+      "elapsed_seconds": 438408.5406104899,
+      "projected_total_hours": 147.11042140485327,
+      "projected_total_cost_usd": 204.48348575274602
+    },
+    {
+      "iteration": 38000,
+      "train_loss": 3.2740094661712646,
+      "validation_loss": 3.3664190769195557,
+      "learning_rate": 3.930009818726269e-05,
+      "elapsed_seconds": 444255.0420406908,
+      "projected_total_hours": 147.11077050031648,
+      "projected_total_cost_usd": 204.48397099543988
+    },
+    {
+      "iteration": 38500,
+      "train_loss": 3.270355224609375,
+      "validation_loss": 3.36319637298584,
+      "learning_rate": 3.744424828728872e-05,
+      "elapsed_seconds": 450094.42088113073,
+      "projected_total_hours": 147.10878258236093,
+      "projected_total_cost_usd": 204.4812077894817
+    },
+    {
+      "iteration": 39000,
+      "train_loss": 3.267413854598999,
+      "validation_loss": 3.3608386516571045,
+      "learning_rate": 3.571030090836964e-05,
+      "elapsed_seconds": 455934.32394408295,
+      "projected_total_hours": 147.1070147768302,
+      "projected_total_cost_usd": 204.47875053979396
+    },
+    {
+      "iteration": 39500,
+      "train_loss": 3.2651331424713135,
+      "validation_loss": 3.3594908714294434,
+      "learning_rate": 3.410036859829256e-05,
+      "elapsed_seconds": 461770.6794984429,
+      "projected_total_hours": 147.10416161237316,
+      "projected_total_cost_usd": 204.47478464119868
+    },
+    {
+      "iteration": 40000,
+      "train_loss": 3.261547803878784,
+      "validation_loss": 3.3563365936279297,
+      "learning_rate": 3.261641281158625e-05,
+      "elapsed_seconds": 467615.53966924874,
+      "projected_total_hours": 147.1040551876178,
+      "projected_total_cost_usd": 204.47463671078873
+    },
+    {
+      "iteration": 40500,
+      "train_loss": 3.2590126991271973,
+      "validation_loss": 3.3551130294799805,
+      "learning_rate": 3.126024151979095e-05,
+      "elapsed_seconds": 473461.22736253403,
+      "projected_total_hours": 147.10420850152806,
+      "projected_total_cost_usd": 204.47484981712398
+    },
+    {
+      "iteration": 41000,
+      "train_loss": 3.255312919616699,
+      "validation_loss": 3.3526666164398193,
+      "learning_rate": 3.0033507008723506e-05,
+      "elapsed_seconds": 479312.32710030675,
+      "projected_total_hours": 147.1060190897283,
+      "projected_total_cost_usd": 204.47736653472234
+    },
+    {
+      "iteration": 41500,
+      "train_loss": 3.2539491653442383,
+      "validation_loss": 3.350898504257202,
+      "learning_rate": 2.8937703865421725e-05,
+      "elapsed_seconds": 485160.5302867908,
+      "projected_total_hours": 147.10690777772172,
+      "projected_total_cost_usd": 204.47860181103317
+    },
+    {
+      "iteration": 42000,
+      "train_loss": 3.2522432804107666,
+      "validation_loss": 3.349637746810913,
+      "learning_rate": 2.797416715722014e-05,
+      "elapsed_seconds": 491000.89962377073,
+      "projected_total_hours": 147.1054282602964,
+      "projected_total_cost_usd": 204.47654528181198
+    },
+    {
+      "iteration": 42500,
+      "train_loss": 3.2494194507598877,
+      "validation_loss": 3.347707509994507,
+      "learning_rate": 2.7144070805176307e-05,
+      "elapsed_seconds": 496843.76710513374,
+      "projected_total_hours": 147.10472320171607,
+      "projected_total_cost_usd": 204.47556525038533
+    },
+    {
+      "iteration": 43000,
+      "train_loss": 3.247154474258423,
+      "validation_loss": 3.3457424640655518,
+      "learning_rate": 2.6448426153828607e-05,
+      "elapsed_seconds": 502679.92955206474,
+      "projected_total_hours": 147.10207240767787,
+      "projected_total_cost_usd": 204.47188064667222
+    },
+    {
+      "iteration": 43500,
+      "train_loss": 3.2443184852600098,
+      "validation_loss": 3.3455331325531006,
+      "learning_rate": 2.588808073902886e-05,
+      "elapsed_seconds": 508520.75622208,
+      "projected_total_hours": 147.10083178071662,
+      "projected_total_cost_usd": 204.47015617519608
+    },
+    {
+      "iteration": 44000,
+      "train_loss": 3.2425286769866943,
+      "validation_loss": 3.344048500061035,
+      "learning_rate": 2.5463717255350436e-05,
+      "elapsed_seconds": 514355.92856104113,
+      "projected_total_hours": 147.0980022968129,
+      "projected_total_cost_usd": 204.4662231925699
+    },
+    {
+      "iteration": 44500,
+      "train_loss": 3.2407639026641846,
+      "validation_loss": 3.343557834625244,
+      "learning_rate": 2.5175852724329982e-05,
+      "elapsed_seconds": 520196.1437763069,
+      "projected_total_hours": 147.0966623786935,
+      "projected_total_cost_usd": 204.46436070638396
+    },
+    {
+      "iteration": 45000,
+      "train_loss": 3.239764928817749,
+      "validation_loss": 3.3421542644500732,
+      "learning_rate": 2.5024837864556432e-05,
+      "elapsed_seconds": 526028.3676926447,
+      "projected_total_hours": 147.09311763257287,
+      "projected_total_cost_usd": 204.45943350927627
+    },
+    {
+      "iteration": 45300,
+      "train_loss": 3.2387826442718506,
+      "validation_loss": 3.3402066230773926,
+      "learning_rate": 2.5000000274155676e-05,
+      "elapsed_seconds": 529567.9590760078,
+      "projected_total_hours": 147.1022108544466,
+      "projected_total_cost_usd": 204.47207308768077
+    }
+  ],
+  "environment": {
+    "python": "3.11.10",
+    "platform": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
+    "torch": "2.5.1+cu124",
+    "cuda": "12.4",
+    "gpu": "NVIDIA A100 80GB PCIe"
+  },
+  "data_manifest_sha256": "c8cb248a827a40f241297474b8e1c2e64589ecd3dbaebe860474aa401697bf98",
+  "peak_vram_bytes": 54941026304,
+  "gpu_total_bytes": 85093777408,
+  "estimated_training_cost_usd": 204.4843982894051
+}