Initial dropout decay research pipeline

.gitignore

@@ -0,0 +1,5 @@
+.DS_Store
+__pycache__/
+*.py[cod]
+.cache/
+*.npy

LICENSE

@@ -0,0 +1,26 @@
+MIT License
+
+Copyright (c) 2025 Andrej Karpathy
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Additional project attribution:
+This repository contains research code derived from Andrej Karpathy's
+nanochat project (https://github.com/karpathy/nanochat), which is distributed
+under the MIT License above.

README.md

@@ -0,0 +1,89 @@
+# Dropout Decay Streaming Experiments
+
+This project tests dropout decay only after first finding a model/data regime
+where static dropout has a real nonzero validation optimum.
+
+The implementation is derived from Andrej Karpathy's `nanochat` repository:
+https://github.com/karpathy/nanochat. Only the core tokenizer ideas and
+foundational causal Transformer architecture are retained. Chat interfaces,
+deployment scripts, distributed training code, and inference services are not
+included. The original nanochat MIT copyright and permission notice are retained
+in derived source files and in `LICENSE`.
+
+## Compliance
+
+All Torch experiment runs are MPS-only. The runner exits before model creation if
+MPS is unavailable, if PyTorch was not built with MPS, or if
+`PYTORCH_ENABLE_MPS_FALLBACK=1` is set.
+
+## Workflow
+
+1. Screen candidate model sizes with cheap static dropout sweeps.
+2. Select candidate models whose validation curve has an interior nonzero
+   dropout optimum.
+3. Confirm the winner with a 3-seed static sweep.
+4. Lock the model and run static-vs-decay streaming comparisons from scratch.
+
+Every run writes:
+
+- `config.json`: command, model specs, data paths, environment, attribution.
+- `metrics.jsonl`: one row per seed/model/dropout/stage.
+- `trace.jsonl`: optional training and intermediate evaluation trace.
+- `summary.csv` / `summary.json`: mean/std train loss, validation loss, and gap.
+- `model_selection.csv` / `model_selection.json`: static-sweep optimum and
+  plateau diagnostics for screen and confirm runs.
+
+Old exploratory outputs are archived under `archive/`.
+
+## Step 1: Cheap Static Screen
+
+Use one or two seeds. The output tells us, for each model, where the static
+dropout curve bottoms out and which dropout range is within the configured
+plateau delta.
+
+```bash
+PYTHONPATH=src /Users/mandeepsidhu/Desktop/code/nanochat/.venv/bin/python scripts/run_experiments.py \
+  --mode screen_static \
+  --corpus-glob "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet" \
+  --models 8x8x256 12x8x384 16x8x384 \
+  --seeds 1 2 \
+  --token-limits 5000000 \
+  --dropout-rates 0.0 0.02 0.05 0.08 0.10 0.14 0.20 0.30 0.50 \
+  --steps 2000 \
+  --eval-batches 64
+```
+
+## Step 2: Confirm Winner
+
+After selecting a promising model, rerun the static dropout curve with exactly
+three seeds.
+
+```bash
+PYTHONPATH=src /Users/mandeepsidhu/Desktop/code/nanochat/.venv/bin/python scripts/run_experiments.py \
+  --mode confirm_static \
+  --corpus-glob "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet" \
+  --models winner=12x8x384 \
+  --seeds 1 2 3 \
+  --token-limits 5000000 \
+  --dropout-rates 0.0 0.02 0.05 0.08 0.10 0.14 0.20 0.30 0.50 \
+  --steps 2000 \
+  --eval-batches 64
+```
+
+## Step 3: Locked Streaming Comparison
+
+Only after the model is locked, compare static dropout and decay schedules from
+fresh initialization.
+
+```bash
+PYTHONPATH=src /Users/mandeepsidhu/Desktop/code/nanochat/.venv/bin/python scripts/run_experiments.py \
+  --mode locked_stream \
+  --corpus-glob "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet" \
+  --models winner=12x8x384 \
+  --seeds 1 2 3 \
+  --stream-token-caps 5000000 10000000 20000000 40000000 \
+  --dropout-rates 0.0 0.10 0.14 0.20 \
+  --decays decay_030_to_014:0.30:0.14:cosine decay_020_to_010:0.20:0.10:cosine \
+  --stage-steps 1000 \
+  --eval-batches 64
+```

archive/legacy_docs/control_results.md

@@ -0,0 +1,48 @@
+# Control Results
+
+These controls were run after the initial core experiment to isolate whether
+dropout decay caused the streaming improvement.
+
+Both runs used CLIMB-mix parquet data from the local nanochat cache, MPS-only
+Torch execution, five seeds, a 4096-token BPE vocabulary, 500k validation tokens,
+and the same 8.39M-parameter large model.
+
+## Run Artifacts
+
+- Core run: `runs/publishable/20260522-132351`
+- Control run: `runs/controls/20260522-141120`
+
+## Matched Streaming Comparison
+
+| Condition | 5M | 10M | 20M | 40M |
+| --- | ---: | ---: | ---: | ---: |
+| Dropout decay streaming | 6.9213 +/- 0.0434 | 6.2689 +/- 0.0539 | 5.4262 +/- 0.0722 | 4.9090 +/- 0.0452 |
+| Static 0.1 dropout streaming | 5.6310 +/- 0.0438 | 5.1018 +/- 0.0306 | 4.8497 +/- 0.0338 | 4.6743 +/- 0.0449 |
+| Static 0.8 dropout streaming | 6.9898 +/- 0.0505 | 6.7637 +/- 0.0762 | 6.4835 +/- 0.0408 | 6.2390 +/- 0.0543 |
+
+## Other Controls
+
+| Condition | Token limit | Mean eval loss | Std |
+| --- | ---: | ---: | ---: |
+| Large static baseline, fixed initial data | 5M | 5.2036 | 0.0258 |
+| Large dropout decay, fixed initial data | 5M | 6.7389 | 0.0484 |
+| Large high static dropout, fixed initial data | 5M | 6.8355 | 0.0613 |
+| Large static full data from start | 40M | 4.6915 | 0.0430 |
+
+## Conclusion
+
+The original strong hypothesis is falsified in this configuration. A high-capacity
+model does absorb the expanding data stream, but dropout decay from 0.8 to 0.1 is
+not the causal reason for the improvement. The matched static 0.1 dropout
+streaming control beats dropout decay at every token cap, and high static dropout
+performs poorly.
+
+The narrower supported claim is:
+
+> An 8.39M-parameter causal Transformer can continuously improve as the simulated
+> stream grows from 5M to 40M tokens, but for this setup static 0.1 dropout is
+> stronger than a high-initial-dropout decay schedule.
+
+Extra tuning could search for a better schedule, such as a lower initial dropout
+or a faster decay horizon, but that would be a new hypothesis rather than support
+for the original one.

archive/legacy_docs/stage_dropout_search.md

@@ -0,0 +1,82 @@
+# Stage Dropout Search
+
+This search tested whether the original failure came from the `0.8` initial
+dropout rather than from the decay idea itself.
+
+All runs used the same CLIMB-mix corpus, 8.39M-parameter large model, 5M/10M/20M/
+40M stream caps, MPS-only execution, and 64 validation batches.
+
+## Single-Seed Probe
+
+Seed: 1.
+
+| Condition | 5M | 10M | 20M | 40M |
+| --- | ---: | ---: | ---: | ---: |
+| Static 0.1 dropout | 5.5670 | 5.0850 | 4.8676 | 4.7417 |
+| Old 0.8 cosine decay | 6.9460 | 6.2013 | 5.3635 | 4.9677 |
+| Mild: 0.30 / 0.22 / 0.14 / 0.10 | 5.7511 | 5.1962 | 4.9055 | 4.7524 |
+| Medium: 0.50 / 0.32 / 0.16 / 0.10 | 6.1544 | 5.3261 | 4.9566 | 4.8095 |
+| Fast: 0.50 / 0.10 / 0.10 / 0.10 | 6.1544 | 5.1626 | 4.8824 | 4.7583 |
+
+The medium schedule was clearly weaker. Mild and fast were close enough to static
+0.1 at 40M to justify multi-seed runs.
+
+## Multi-Seed Follow-Up
+
+Seeds: 1, 2, 3, 4, 5.
+
+| Condition | 5M | 10M | 20M | 40M |
+| --- | ---: | ---: | ---: | ---: |
+| Static 0.0 dropout | 5.5694 +/- 0.0465 | 5.0527 +/- 0.0596 | 4.7800 +/- 0.0423 | 4.5998 +/- 0.0391 |
+| Static 0.01 dropout | 5.5807 +/- 0.0503 | 5.0627 +/- 0.0630 | 4.7904 +/- 0.0405 | 4.6152 +/- 0.0525 |
+| Static 0.02 dropout | 5.5873 +/- 0.0436 | 5.0575 +/- 0.0396 | 4.8009 +/- 0.0406 | 4.6192 +/- 0.0455 |
+| Static 0.03 dropout | 5.5920 +/- 0.0436 | 5.0734 +/- 0.0505 | 4.8110 +/- 0.0410 | 4.6281 +/- 0.0448 |
+| Static 0.04 dropout | 5.5964 +/- 0.0435 | 5.0752 +/- 0.0490 | 4.8165 +/- 0.0444 | 4.6416 +/- 0.0475 |
+| Static 0.06 dropout | 5.6075 +/- 0.0476 | 5.0797 +/- 0.0394 | 4.8324 +/- 0.0399 | 4.6499 +/- 0.0430 |
+| Static 0.08 dropout | 5.6146 +/- 0.0425 | 5.0849 +/- 0.0363 | 4.8358 +/- 0.0350 | 4.6541 +/- 0.0387 |
+| Static 0.1 dropout | 5.6310 +/- 0.0438 | 5.1018 +/- 0.0306 | 4.8497 +/- 0.0338 | 4.6743 +/- 0.0449 |
+| Static 0.14 dropout | 5.6544 +/- 0.0361 | 5.1239 +/- 0.0318 | 4.8742 +/- 0.0323 | 4.7003 +/- 0.0365 |
+| Static 0.20 dropout | 5.6960 +/- 0.0453 | 5.1722 +/- 0.0320 | 4.9164 +/- 0.0286 | 4.7492 +/- 0.0379 |
+| Sub-0.1: 0.10 / 0.08 / 0.06 / 0.05 | 5.6310 +/- 0.0438 | 5.0922 +/- 0.0312 | 4.8311 +/- 0.0352 | 4.6496 +/- 0.0499 |
+| Sub-0.08: 0.08 / 0.06 / 0.04 / 0.03 | 5.6146 +/- 0.0425 | 5.0768 +/- 0.0386 | 4.8163 +/- 0.0372 | 4.6322 +/- 0.0405 |
+| Mild: 0.30 / 0.22 / 0.14 / 0.10 | 5.8129 +/- 0.0528 | 5.2047 +/- 0.0351 | 4.8991 +/- 0.0294 | 4.7034 +/- 0.0415 |
+| Fast: 0.50 / 0.10 / 0.10 / 0.10 | 6.2006 +/- 0.0451 | 5.1861 +/- 0.0352 | 4.8875 +/- 0.0350 | 4.7025 +/- 0.0414 |
+| Old 0.8 cosine decay | 6.9213 +/- 0.0434 | 6.2689 +/- 0.0539 | 5.4262 +/- 0.0722 | 4.9090 +/- 0.0452 |
+
+## Conclusion
+
+The original initial dropout was too large, and the old schedule kept dropout too
+large for too long. Lower and faster schedules recover most of the lost
+performance, which supports the diagnosis that `0.8` dropout was the main
+problem.
+
+However, the revised decay schedules still do not beat static `0.1` dropout.
+The best revised schedule at 40M was fast decay at `4.7025 +/- 0.0414`, compared
+with static `0.1` at `4.6743 +/- 0.0449`. The gap is small, but not favorable.
+
+The sub-0.1 schedules changed the result relative to static `0.1`, but the
+static sub-0.1 plateau sweep changed the interpretation. Static dropout kept
+improving as dropout was lowered across the tested grid, and the best tested
+condition is now static `0.0`, not a decay schedule.
+
+Updated supported claim:
+
+> High initial dropout is harmful in this setup. Dropout values above 0.1 lose
+> monotonically in the tested static sweeps. Sub-0.1 decay improves over static
+> 0.1, but static sub-0.1 controls show that this improvement is likely caused
+> by using lower dropout values, not by decay itself. On the current 8.39M
+> parameter model and CLIMB-mix stream, the static-dropout plateau was not found
+> above zero.
+
+Run artifacts:
+
+- Single mild: `runs/stage-dropout-single/20260523-075308`
+- Single medium: `runs/stage-dropout-single/20260523-075636`
+- Single fast: `runs/stage-dropout-single/20260523-080008`
+- Multi mild: `runs/stage-dropout-multiseed/20260523-080428`
+- Multi fast: `runs/stage-dropout-multiseed/20260523-082216`
+- Static 0.14: `runs/static-dropout-sweep/20260523-084122`
+- Static 0.20: `runs/static-dropout-sweep/20260523-085817`
+- Static sub-0.1 sweep: `runs/static-sub01-sweep/20260525-074602`
+- Sub-0.1: `runs/sub01-dropout-decay/20260525-070308`
+- Sub-0.08: `runs/sub01-dropout-decay/20260525-071923`

archive/runs_legacy_20260525/control-smoke/20260522-141107/config.json

@@ -0,0 +1,57 @@
+{
+  "args": {
+    "corpus": "/Users/mandeepsidhu/Desktop/code/transformer/data/tinystories_train.txt",
+    "corpus_glob": null,
+    "text_column": "text",
+    "output_dir": "runs/control-smoke",
+    "suite": "controls",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 20000,
+    "stream_token_caps": [
+      20000,
+      40000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": true,
+    "force_retokenize": false,
+    "vocab_size": 512,
+    "tokenizer_train_chars": 200000,
+    "block_size": 32,
+    "batch_size": 4,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 1,
+    "stream_steps_per_stage": 1,
+    "eval_batches": 1,
+    "log_every": 0,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/control-smoke/20260522-141107/tokenizer.json",
+  "encoded_path": "runs/control-smoke/20260522-141107/tokens-v512-uint16.npy",
+  "train_tokens": 777983,
+  "val_tokens": 86442,
+  "effective_initial_tokens": 20000,
+  "effective_stream_token_caps": [
+    20000,
+    40000
+  ]
+}

archive/runs_legacy_20260525/control-smoke/20260522-141107/metrics.jsonl

@@ -0,0 +1,30 @@
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 1, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.2400970458984375, "eval_loss": 6.237194538116455, "elapsed_sec": 0.12676668167114258, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 1, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.240777015686035, "eval_loss": 6.2333245277404785, "elapsed_sec": 0.0425410270690918, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 1, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.22697639465332, "eval_loss": 6.224180698394775, "elapsed_sec": 0.03090190887451172, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 1, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.2400970458984375, "eval_loss": 6.237194538116455, "elapsed_sec": 0.03940081596374512, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 1, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.8, "train_loss_last": 6.236742973327637, "eval_loss": 6.234339714050293, "elapsed_sec": 0.029629945755004883, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 1, "stage": null, "token_limit": 40000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.221075057983398, "eval_loss": 6.228199481964111, "elapsed_sec": 0.06530499458312988, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 2, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.236819267272949, "eval_loss": 6.234230995178223, "elapsed_sec": 0.03899407386779785, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 2, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.238691329956055, "eval_loss": 6.229793548583984, "elapsed_sec": 0.03891491889953613, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 2, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.2327423095703125, "eval_loss": 6.218461513519287, "elapsed_sec": 0.029666900634765625, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 2, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.236819267272949, "eval_loss": 6.234230995178223, "elapsed_sec": 0.03844118118286133, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 2, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.8, "train_loss_last": 6.236791133880615, "eval_loss": 6.230033874511719, "elapsed_sec": 0.030471086502075195, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 2, "stage": null, "token_limit": 40000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.2310051918029785, "eval_loss": 6.225197792053223, "elapsed_sec": 0.06515097618103027, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 3, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.238975524902344, "eval_loss": 6.2339982986450195, "elapsed_sec": 0.03981590270996094, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 3, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.237975120544434, "eval_loss": 6.229854583740234, "elapsed_sec": 0.04071402549743652, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 3, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.23427677154541, "eval_loss": 6.21778678894043, "elapsed_sec": 0.03224611282348633, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 3, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.238975524902344, "eval_loss": 6.2339982986450195, "elapsed_sec": 0.040772199630737305, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 3, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.8, "train_loss_last": 6.238644599914551, "eval_loss": 6.229787826538086, "elapsed_sec": 0.030426025390625, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 3, "stage": null, "token_limit": 40000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.2319865226745605, "eval_loss": 6.2150068283081055, "elapsed_sec": 0.06477165222167969, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 4, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.239019393920898, "eval_loss": 6.237016677856445, "elapsed_sec": 0.03902721405029297, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 4, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.239620208740234, "eval_loss": 6.232203960418701, "elapsed_sec": 0.03930306434631348, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 4, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.233333110809326, "eval_loss": 6.221055507659912, "elapsed_sec": 0.030991077423095703, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 4, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.239019393920898, "eval_loss": 6.237016677856445, "elapsed_sec": 0.0409388542175293, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 4, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.8, "train_loss_last": 6.235775947570801, "eval_loss": 6.23379373550415, "elapsed_sec": 0.030637741088867188, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 4, "stage": null, "token_limit": 40000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.2377543449401855, "eval_loss": 6.226580619812012, "elapsed_sec": 0.06628823280334473, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 5, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.239316463470459, "eval_loss": 6.237186431884766, "elapsed_sec": 0.039865732192993164, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 5, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.23555850982666, "eval_loss": 6.231681823730469, "elapsed_sec": 0.039572954177856445, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 5, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.229386329650879, "eval_loss": 6.219687461853027, "elapsed_sec": 0.03072190284729004, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 5, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.239316463470459, "eval_loss": 6.237186431884766, "elapsed_sec": 0.04007697105407715, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 5, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.8, "train_loss_last": 6.239014148712158, "eval_loss": 6.233704566955566, "elapsed_sec": 0.031112194061279297, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 5, "stage": null, "token_limit": 40000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.230358123779297, "eval_loss": 6.227848052978516, "elapsed_sec": 0.06935000419616699, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}

archive/runs_legacy_20260525/control-smoke/20260522-141107/summary.csv

@@ -0,0 +1,7 @@
+phase,condition,stage,token_limit,parameters,n,mean_eval_loss,std_eval_loss
+phase2_fixed_data_dropout_optimization,large_high_static_fixed_data,,20000,6553600,5,6.235925388336182,0.0016565421564777797
+phase3_streaming_controls,large_high_dropout_streaming,0,20000,6553600,5,6.235925388336182,0.0016565421564777797
+phase3_streaming_controls,large_high_dropout_streaming,1,40000,6553600,5,6.232331943511963,0.0022251812166423465
+phase3_streaming_controls,large_static_dropout_streaming,0,20000,6553600,5,6.231371688842773,0.0015325284625100665
+phase3_streaming_controls,large_static_dropout_streaming,1,40000,6553600,5,6.220234394073486,0.0025337993467635157
+phase4_full_data_controls,large_static_full_data_from_start,,40000,6553600,5,6.224566555023193,0.005472880389192584

archive/runs_legacy_20260525/control-smoke/20260522-141107/summary.json

@@ -0,0 +1,62 @@
+[
+  {
+    "phase": "phase2_fixed_data_dropout_optimization",
+    "condition": "large_high_static_fixed_data",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.235925388336182,
+    "std_eval_loss": 0.0016565421564777797
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_high_dropout_streaming",
+    "stage": 0,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.235925388336182,
+    "std_eval_loss": 0.0016565421564777797
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_high_dropout_streaming",
+    "stage": 1,
+    "token_limit": 40000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.232331943511963,
+    "std_eval_loss": 0.0022251812166423465
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_static_dropout_streaming",
+    "stage": 0,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.231371688842773,
+    "std_eval_loss": 0.0015325284625100665
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_static_dropout_streaming",
+    "stage": 1,
+    "token_limit": 40000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.220234394073486,
+    "std_eval_loss": 0.0025337993467635157
+  },
+  {
+    "phase": "phase4_full_data_controls",
+    "condition": "large_static_full_data_from_start",
+    "stage": null,
+    "token_limit": 40000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.224566555023193,
+    "std_eval_loss": 0.005472880389192584
+  }
+]

archive/runs_legacy_20260525/control-smoke/20260522-141107/tokenizer.json

@@ -0,0 +1,1629 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<|bos|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<|user_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<|user_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<|assistant_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<|assistant_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "<|python_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 6,
+      "content": "<|python_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 7,
+      "content": "<|output_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 8,
+      "content": "<|output_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Sequence",
+    "pretokenizers": [
+      {
+        "type": "Split",
+        "pattern": {
+          "Regex": "'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?+\\p{L}+|\\p{N}{1,2}| ?[^\\s\\p{L}\\p{N}]++[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+"
+        },
+        "behavior": "Isolated",
+        "invert": false
+      },
+      {
+        "type": "ByteLevel",
+        "add_prefix_space": false,
+        "trim_offsets": true,
+        "use_regex": false
+      }
+    ]
+  },
+  "post_processor": null,
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": true,
+    "ignore_merges": false,
+    "vocab": {
+      "<|bos|>": 0,
+      "<|user_start|>": 1,
+      "<|user_end|>": 2,
+      "<|assistant_start|>": 3,
+      "<|assistant_end|>": 4,
+      "<|python_start|>": 5,
+      "<|python_end|>": 6,
+      "<|output_start|>": 7,
+      "<|output_end|>": 8,
+      "!": 9,
+      "\"": 10,
+      "#": 11,
+      "$": 12,
+      "%": 13,
+      "&": 14,
+      "'": 15,
+      "(": 16,
+      ")": 17,
+      "*": 18,
+      "+": 19,
+      ",": 20,
+      "-": 21,
+      ".": 22,
+      "/": 23,
+      "0": 24,
+      "1": 25,
+      "2": 26,
+      "3": 27,
+      "4": 28,
+      "5": 29,
+      "6": 30,
+      "7": 31,
+      "8": 32,
+      "9": 33,
+      ":": 34,
+      ";": 35,
+      "<": 36,
+      "=": 37,
+      ">": 38,
+      "?": 39,
+      "@": 40,
+      "A": 41,
+      "B": 42,
+      "C": 43,
+      "D": 44,
+      "E": 45,
+      "F": 46,
+      "G": 47,
+      "H": 48,
+      "I": 49,
+      "J": 50,
+      "K": 51,
+      "L": 52,
+      "M": 53,
+      "N": 54,
+      "O": 55,
+      "P": 56,
+      "Q": 57,
+      "R": 58,
+      "S": 59,
+      "T": 60,
+      "U": 61,
+      "V": 62,
+      "W": 63,
+      "X": 64,
+      "Y": 65,
+      "Z": 66,
+      "[": 67,
+      "\\": 68,
+      "]": 69,
+      "^": 70,
+      "_": 71,
+      "`": 72,
+      "a": 73,
+      "b": 74,
+      "c": 75,
+      "d": 76,
+      "e": 77,
+      "f": 78,
+      "g": 79,
+      "h": 80,
+      "i": 81,
+      "j": 82,
+      "k": 83,
+      "l": 84,
+      "m": 85,
+      "n": 86,
+      "o": 87,
+      "p": 88,
+      "q": 89,
+      "r": 90,
+      "s": 91,
+      "t": 92,
+      "u": 93,
+      "v": 94,
+      "w": 95,
+      "x": 96,
+      "y": 97,
+      "z": 98,
+      "{": 99,
+      "|": 100,
+      "}": 101,
+      "~": 102,
+      "¡": 103,
+      "¢": 104,
+      "£": 105,
+      "¤": 106,
+      "¥": 107,
+      "¦": 108,
+      "§": 109,
+      "¨": 110,
+      "©": 111,
+      "ª": 112,
+      "«": 113,
+      "¬": 114,
+      "®": 115,
+      "¯": 116,
+      "°": 117,
+      "±": 118,
+      "²": 119,
+      "³": 120,
+      "´": 121,
+      "µ": 122,
+      "¶": 123,
+      "·": 124,
+      "¸": 125,
+      "¹": 126,
+      "º": 127,
+      "»": 128,
+      "¼": 129,
+      "½": 130,
+      "¾": 131,
+      "¿": 132,
+      "À": 133,
+      "Á": 134,
+      "Â": 135,
+      "Ã": 136,
+      "Ä": 137,
+      "Å": 138,
+      "Æ": 139,
+      "Ç": 140,
+      "È": 141,
+      "É": 142,
+      "Ê": 143,
+      "Ë": 144,
+      "Ì": 145,
+      "Í": 146,
+      "Î": 147,
+      "Ï": 148,
+      "Ð": 149,
+      "Ñ": 150,
+      "Ò": 151,
+      "Ó": 152,
+      "Ô": 153,
+      "Õ": 154,
+      "Ö": 155,
+      "×": 156,
+      "Ø": 157,
+      "Ù": 158,
+      "Ú": 159,
+      "Û": 160,
+      "Ü": 161,
+      "Ý": 162,
+      "Þ": 163,
+      "ß": 164,
+      "à": 165,
+      "á": 166,
+      "â": 167,
+      "ã": 168,
+      "ä": 169,
+      "å": 170,
+      "æ": 171,
+      "ç": 172,
+      "è": 173,
+      "é": 174,
+      "ê": 175,
+      "ë": 176,
+      "ì": 177,
+      "í": 178,
+      "î": 179,
+      "ï": 180,
+      "ð": 181,
+      "ñ": 182,
+      "ò": 183,
+      "ó": 184,
+      "ô": 185,
+      "õ": 186,
+      "ö": 187,
+      "÷": 188,
+      "ø": 189,
+      "ù": 190,
+      "ú": 191,
+      "û": 192,
+      "ü": 193,
+      "ý": 194,
+      "þ": 195,
+      "ÿ": 196,
+      "Ā": 197,
+      "ā": 198,
+      "Ă": 199,
+      "ă": 200,
+      "Ą": 201,
+      "ą": 202,
+      "Ć": 203,
+      "ć": 204,
+      "Ĉ": 205,
+      "ĉ": 206,
+      "Ċ": 207,
+      "ċ": 208,
+      "Č": 209,
+      "č": 210,
+      "Ď": 211,
+      "ď": 212,
+      "Đ": 213,
+      "đ": 214,
+      "Ē": 215,
+      "ē": 216,
+      "Ĕ": 217,
+      "ĕ": 218,
+      "Ė": 219,
+      "ė": 220,
+      "Ę": 221,
+      "ę": 222,
+      "Ě": 223,
+      "ě": 224,
+      "Ĝ": 225,
+      "ĝ": 226,
+      "Ğ": 227,
+      "ğ": 228,
+      "Ġ": 229,
+      "ġ": 230,
+      "Ģ": 231,
+      "ģ": 232,
+      "Ĥ": 233,
+      "ĥ": 234,
+      "Ħ": 235,
+      "ħ": 236,
+      "Ĩ": 237,
+      "ĩ": 238,
+      "Ī": 239,
+      "ī": 240,
+      "Ĭ": 241,
+      "ĭ": 242,
+      "Į": 243,
+      "į": 244,
+      "İ": 245,
+      "ı": 246,
+      "IJ": 247,
+      "ij": 248,
+      "Ĵ": 249,
+      "ĵ": 250,
+      "Ķ": 251,
+      "ķ": 252,
+      "ĸ": 253,
+      "Ĺ": 254,
+      "ĺ": 255,
+      "Ļ": 256,
+      "ļ": 257,
+      "Ľ": 258,
+      "ľ": 259,
+      "Ŀ": 260,
+      "ŀ": 261,
+      "Ł": 262,
+      "ł": 263,
+      "Ń": 264,
+      "he": 265,
+      "Ġt": 266,
+      "Ġa": 267,
+      "Ġs": 268,
+      "Ġw": 269,
+      "Ġthe": 270,
+      "nd": 271,
+      "ed": 272,
+      "in": 273,
+      "Ġand": 274,
+      "Ġwa": 275,
+      "Ġb": 276,
+      "Ġto": 277,
+      "re": 278,
+      "Ġh": 279,
+      "ou": 280,
+      "it": 281,
+      "Ġf": 282,
+      "er": 283,
+      "ĊĊ": 284,
+      "Ġwas": 285,
+      "Ġl": 286,
+      "Ġc": 287,
+      "Ġhe": 288,
+      "Ġp": 289,
+      "ing": 290,
+      "Ġd": 291,
+      "Ġm": 292,
+      "Ġo": 293,
+      "Ġg": 294,
+      "ar": 295,
+      "is": 296,
+      "id": 297,
+      "ay": 298,
+      "om": 299,
+      "at": 300,
+      "ll": 301,
+      "en": 302,
+      "Ġsa": 303,
+      "ne": 304,
+      "The": 305,
+      ".ĊĊ": 306,
+      "le": 307,
+      "Ġth": 308,
+      "im": 309,
+      "an": 310,
+      "Ġha": 311,
+      "or": 312,
+      "Ġit": 313,
+      "et": 314,
+      "ver": 315,
+      "ld": 316,
+      "Ġin": 317,
+      "ĠS": 318,
+      "on": 319,
+      "Ġe": 320,
+      "ce": 321,
+      "Ġbe": 322,
+      "Ġher": 323,
+      "ir": 324,
+      "Ġ\"": 325,
+      "ĠH": 326,
+      "Ġu": 327,
+      "Ġsaid": 328,
+      "Ġn": 329,
+      "ck": 330,
+      "ow": 331,
+      "ri": 332,
+      "ĠThe": 333,
+      "Ġshe": 334,
+      "Ġso": 335,
+      "st": 336,
+      "Ġy": 337,
+      "ot": 338,
+      "ĠHe": 339,
+      "Ġof": 340,
+      "il": 341,
+      "Ġst": 342,
+      "ut": 343,
+      "ke": 344,
+      "am": 345,
+      "ked": 346,
+      "oo": 347,
+      "pp": 348,
+      "Ġr": 349,
+      "ĠShe": 350,
+      "very": 351,
+      "ĠI": 352,
+      "ve": 353,
+      "Ġthat": 354,
+      "ig": 355,
+      "ith": 356,
+      "Ġhis": 357,
+      "Ġup": 358,
+      "ĠĊĊ": 359,
+      "Ġday": 360,
+      "Ġwith": 361,
+      "Ġpl": 362,
+      "Ġyou": 363,
+      "itt": 364,
+      "ould": 365,
+      "el": 366,
+      "ted": 367,
+      "ent": 368,
+      "ad": 369,
+      "Ġhad": 370,
+      "ound": 371,
+      "al": 372,
+      "ĠJ": 373,
+      "Ġwe": 374,
+      "her": 375,
+      "ittle": 376,
+      "'s": 377,
+      "Ġsm": 378,
+      "Ġplay": 379,
+      "end": 380,
+      "Ġthey": 381,
+      "ack": 382,
+      "Ġthere": 383,
+      "ime": 384,
+      "ly": 385,
+      "Ġsh": 386,
+      "Ġlittle": 387,
+      "Ġre": 388,
+      "Ġne": 389,
+      "Ġtime": 390,
+      "out": 391,
+      "Ġfor": 392,
+      "un": 393,
+      "ch": 394,
+      "se": 395,
+      "Ġhapp": 396,
+      "Ġwh": 397,
+      "my": 398,
+      "ome": 399,
+      "ht": 400,
+      "um": 401,
+      "Ġfri": 402,
+      "Ġas": 403,
+      "Ġfriend": 404,
+      "Ġvery": 405,
+      "all": 406,
+      "ter": 407,
+      "â": 408,
+      "âĤ": 409,
+      "âĤ¬": 410,
+      "On": 411,
+      "Ġk": 412,
+      "ved": 413,
+      "ĠT": 414,
+      "Ġon": 415,
+      "irl": 416,
+      "Once": 417,
+      "ug": 418,
+      "\"ĊĊ": 419,
+      "ill": 420,
+      "Ġgirl": 421,
+      "Ġan": 422,
+      "es": 423,
+      "Ġex": 424,
+      "'t": 425,
+      "ec": 426,
+      "Ġbut": 427,
+      "Ġloo": 428,
+      "Ġli": 429,
+      "Ġbo": 430,
+      "Ġwere": 431,
+      "One": 432,
+      "Ġwan": 433,
+      "Ġhappy": 434,
+      "ake": 435,
+      "ore": 436,
+      "Ġbig": 437,
+      "fu": 438,
+      "Ġsp": 439,
+      "ide": 440,
+      "Ġsaw": 441,
+      "ĠB": 442,
+      "hing": 443,
+      "Ġupon": 444,
+      "ard": 445,
+      "Ġcould": 446,
+      "ic": 447,
+      "Ġout": 448,
+      "iled": 449,
+      "one": 450,
+      "round": 451,
+      "ra": 452,
+      "ry": 453,
+      "Ġsmiled": 454,
+      "Ġhim": 455,
+      "ĠA": 456,
+      "Ġmom": 457,
+      "hen": 458,
+      "way": 459,
+      "ur": 460,
+      "ĠIt": 461,
+      "ful": 462,
+      "ain": 463,
+      "Ġwent": 464,
+      "Ġhel": 465,
+      "Ġnot": 466,
+      "ĠThey": 467,
+      "Ġwanted": 468,
+      "ind": 469,
+      "are": 470,
+      "ear": 471,
+      "ĠM": 472,
+      "Ġall": 473,
+      "Ġfriends": 474,
+      "Ġtoo": 475,
+      "Ġgo": 476,
+      "ily": 477,
+      "ame": 478,
+      "ĠTim": 479,
+      "Ġhelp": 480,
+      "omet": 481,
+      "ĠL": 482,
+      "Ġlo": 483,
+      "ght": 484,
+      "Ġsomet": 485,
+      "Ġat": 486,
+      "Ġdo": 487,
+      "Ġasked": 488,
+      "!\"": 489,
+      "Ġaround": 490,
+      "Ġj": 491,
+      "ree": 492,
+      "Ġlooked": 493,
+      "Ġsomething": 494,
+      "Ġse": 495,
+      "Ġwor": 496,
+      "dd": 497,
+      "hed": 498,
+      "ood": 499,
+      "Ġcl": 500,
+      "amed": 501,
+      "ro": 502,
+      "Ġcan": 503,
+      "ark": 504,
+      "king": 505,
+      "ĠE": 506,
+      "rom": 507,
+      "Ġback": 508,
+      "Ġexc": 509,
+      "ab": 510,
+      "ick": 511
+    },
+    "merges": [
+      [
+        "h",
+        "e"
+      ],
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "n",
+        "d"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "Ġa",
+        "nd"
+      ],
+      [
+        "Ġw",
+        "a"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "Ċ",
+        "Ċ"
+      ],
+      [
+        "Ġwa",
+        "s"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "Ġ",
+        "g"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "a",
+        "y"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "l",
+        "l"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġs",
+        "a"
+      ],
+      [
+        "n",
+        "e"
+      ],
+      [
+        "T",
+        "he"
+      ],
+      [
+        ".",
+        "ĊĊ"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "Ġh",
+        "a"
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "Ġ",
+        "S"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġ",
+        "e"
+      ],
+      [
+        "c",
+        "e"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "Ġhe",
+        "r"
+      ],
+      [
+        "i",
+        "r"
+      ],
+      [
+        "Ġ",
+        "\""
+      ],
+      [
+        "Ġ",
+        "H"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "Ġsa",
+        "id"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "c",
+        "k"
+      ],
+      [
+        "o",
+        "w"
+      ],
+      [
+        "r",
+        "i"
+      ],
+      [
+        "Ġ",
+        "The"
+      ],
+      [
+        "Ġs",
+        "he"
+      ],
+      [
+        "Ġs",
+        "o"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "Ġ",
+        "y"
+      ],
+      [
+        "o",
+        "t"
+      ],
+      [
+        "ĠH",
+        "e"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "Ġs",
+        "t"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "k",
+        "e"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "k",
+        "ed"
+      ],
+      [
+        "o",
+        "o"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "ĠS",
+        "he"
+      ],
+      [
+        "ver",
+        "y"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "i",
+        "g"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "Ġh",
+        "is"
+      ],
+      [
+        "Ġu",
+        "p"
+      ],
+      [
+        "Ġ",
+        "ĊĊ"
+      ],
+      [
+        "Ġd",
+        "ay"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "Ġp",
+        "l"
+      ],
+      [
+        "Ġy",
+        "ou"
+      ],
+      [
+        "it",
+        "t"
+      ],
+      [
+        "ou",
+        "ld"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "t",
+        "ed"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "Ġha",
+        "d"
+      ],
+      [
+        "ou",
+        "nd"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "Ġ",
+        "J"
+      ],
+      [
+        "Ġw",
+        "e"
+      ],
+      [
+        "he",
+        "r"
+      ],
+      [
+        "itt",
+        "le"
+      ],
+      [
+        "'",
+        "s"
+      ],
+      [
+        "Ġs",
+        "m"
+      ],
+      [
+        "Ġpl",
+        "ay"
+      ],
+      [
+        "e",
+        "nd"
+      ],
+      [
+        "Ġthe",
+        "y"
+      ],
+      [
+        "a",
+        "ck"
+      ],
+      [
+        "Ġthe",
+        "re"
+      ],
+      [
+        "im",
+        "e"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "Ġs",
+        "h"
+      ],
+      [
+        "Ġl",
+        "ittle"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġ",
+        "ne"
+      ],
+      [
+        "Ġt",
+        "ime"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "u",
+        "n"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "Ġha",
+        "pp"
+      ],
+      [
+        "Ġw",
+        "h"
+      ],
+      [
+        "m",
+        "y"
+      ],
+      [
+        "om",
+        "e"
+      ],
+      [
+        "h",
+        "t"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "Ġf",
+        "ri"
+      ],
+      [
+        "Ġa",
+        "s"
+      ],
+      [
+        "Ġfri",
+        "end"
+      ],
+      [
+        "Ġ",
+        "very"
+      ],
+      [
+        "a",
+        "ll"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ã",
+        "¢"
+      ],
+      [
+        "â",
+        "Ĥ"
+      ],
+      [
+        "âĤ",
+        "¬"
+      ],
+      [
+        "O",
+        "n"
+      ],
+      [
+        "Ġ",
+        "k"
+      ],
+      [
+        "v",
+        "ed"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "Ġo",
+        "n"
+      ],
+      [
+        "ir",
+        "l"
+      ],
+      [
+        "On",
+        "ce"
+      ],
+      [
+        "u",
+        "g"
+      ],
+      [
+        "\"",
+        "ĊĊ"
+      ],
+      [
+        "i",
+        "ll"
+      ],
+      [
+        "Ġg",
+        "irl"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "Ġe",
+        "x"
+      ],
+      [
+        "'",
+        "t"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "Ġb",
+        "ut"
+      ],
+      [
+        "Ġl",
+        "oo"
+      ],
+      [
+        "Ġl",
+        "i"
+      ],
+      [
+        "Ġb",
+        "o"
+      ],
+      [
+        "Ġwe",
+        "re"
+      ],
+      [
+        "O",
+        "ne"
+      ],
+      [
+        "Ġwa",
+        "n"
+      ],
+      [
+        "Ġhapp",
+        "y"
+      ],
+      [
+        "a",
+        "ke"
+      ],
+      [
+        "o",
+        "re"
+      ],
+      [
+        "Ġb",
+        "ig"
+      ],
+      [
+        "f",
+        "u"
+      ],
+      [
+        "Ġs",
+        "p"
+      ],
+      [
+        "id",
+        "e"
+      ],
+      [
+        "Ġsa",
+        "w"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "h",
+        "ing"
+      ],
+      [
+        "Ġup",
+        "on"
+      ],
+      [
+        "ar",
+        "d"
+      ],
+      [
+        "Ġc",
+        "ould"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "Ġ",
+        "out"
+      ],
+      [
+        "il",
+        "ed"
+      ],
+      [
+        "o",
+        "ne"
+      ],
+      [
+        "r",
+        "ound"
+      ],
+      [
+        "r",
+        "a"
+      ],
+      [
+        "r",
+        "y"
+      ],
+      [
+        "Ġsm",
+        "iled"
+      ],
+      [
+        "Ġh",
+        "im"
+      ],
+      [
+        "Ġ",
+        "A"
+      ],
+      [
+        "Ġm",
+        "om"
+      ],
+      [
+        "he",
+        "n"
+      ],
+      [
+        "w",
+        "ay"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "ĠI",
+        "t"
+      ],
+      [
+        "fu",
+        "l"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "Ġw",
+        "ent"
+      ],
+      [
+        "Ġhe",
+        "l"
+      ],
+      [
+        "Ġn",
+        "ot"
+      ],
+      [
+        "ĠThe",
+        "y"
+      ],
+      [
+        "Ġwan",
+        "ted"
+      ],
+      [
+        "i",
+        "nd"
+      ],
+      [
+        "a",
+        "re"
+      ],
+      [
+        "e",
+        "ar"
+      ],
+      [
+        "Ġ",
+        "M"
+      ],
+      [
+        "Ġa",
+        "ll"
+      ],
+      [
+        "Ġfriend",
+        "s"
+      ],
+      [
+        "Ġto",
+        "o"
+      ],
+      [
+        "Ġg",
+        "o"
+      ],
+      [
+        "il",
+        "y"
+      ],
+      [
+        "am",
+        "e"
+      ],
+      [
+        "ĠT",
+        "im"
+      ],
+      [
+        "Ġhel",
+        "p"
+      ],
+      [
+        "om",
+        "et"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "Ġl",
+        "o"
+      ],
+      [
+        "g",
+        "ht"
+      ],
+      [
+        "Ġs",
+        "omet"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "Ġd",
+        "o"
+      ],
+      [
+        "Ġas",
+        "ked"
+      ],
+      [
+        "!",
+        "\""
+      ],
+      [
+        "Ġa",
+        "round"
+      ],
+      [
+        "Ġ",
+        "j"
+      ],
+      [
+        "re",
+        "e"
+      ],
+      [
+        "Ġloo",
+        "ked"
+      ],
+      [
+        "Ġsomet",
+        "hing"
+      ],
+      [
+        "Ġs",
+        "e"
+      ],
+      [
+        "Ġw",
+        "or"
+      ],
+      [
+        "d",
+        "d"
+      ],
+      [
+        "he",
+        "d"
+      ],
+      [
+        "oo",
+        "d"
+      ],
+      [
+        "Ġc",
+        "l"
+      ],
+      [
+        "am",
+        "ed"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "ar",
+        "k"
+      ],
+      [
+        "k",
+        "ing"
+      ],
+      [
+        "Ġ",
+        "E"
+      ],
+      [
+        "r",
+        "om"
+      ],
+      [
+        "Ġb",
+        "ack"
+      ],
+      [
+        "Ġex",
+        "c"
+      ],
+      [
+        "a",
+        "b"
+      ],
+      [
+        "i",
+        "ck"
+      ]
+    ]
+  }
+}

archive/runs_legacy_20260525/controls/20260522-141120/config.json

@@ -0,0 +1,61 @@
+{
+  "args": {
+    "corpus": null,
+    "corpus_glob": "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet",
+    "text_column": "text",
+    "output_dir": "runs/controls",
+    "suite": "controls",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 5000000,
+    "stream_token_caps": [
+      5000000,
+      10000000,
+      20000000,
+      40000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 2000,
+    "stream_steps_per_stage": 1000,
+    "eval_batches": 64,
+    "log_every": 500,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/controls/20260522-141120/tokenizer.json",
+  "encoded_path": "runs/controls/20260522-141120/tokens-v4096-uint16.npy",
+  "train_tokens": 40000160,
+  "val_tokens": 500000,
+  "effective_initial_tokens": 5000000,
+  "effective_stream_token_caps": [
+    5000000,
+    10000000,
+    20000000,
+    40000000
+  ]
+}

archive/runs_legacy_20260525/controls/20260522-141120/metrics.jsonl

@@ -0,0 +1,50 @@
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 1, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.082937240600586, "eval_loss": 6.814276188611984, "elapsed_sec": 75.75013089179993, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 1, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.1, "train_loss_last": 5.391843318939209, "eval_loss": 5.567012831568718, "elapsed_sec": 38.50700092315674, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 1, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 4.914791584014893, "eval_loss": 5.084997855126858, "elapsed_sec": 38.48884105682373, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 1, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.900063514709473, "eval_loss": 4.867573603987694, "elapsed_sec": 38.46290588378906, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 1, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.553297519683838, "eval_loss": 4.741723917424679, "elapsed_sec": 38.33492302894592, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 1, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.8, "train_loss_last": 6.440576076507568, "eval_loss": 6.986421458423138, "elapsed_sec": 38.296988010406494, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 1, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.248085975646973, "eval_loss": 6.673301495611668, "elapsed_sec": 39.15381717681885, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 1, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.8, "train_loss_last": 6.112507343292236, "eval_loss": 6.5272732228040695, "elapsed_sec": 40.44030404090881, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 1, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.8, "train_loss_last": 5.754788398742676, "eval_loss": 6.267793655395508, "elapsed_sec": 41.09470295906067, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 1, "stage": null, "token_limit": 40000000, "steps": 4000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.643326759338379, "eval_loss": 4.662323743104935, "elapsed_sec": 166.3614981174469, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 2, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.124882698059082, "eval_loss": 6.84135477244854, "elapsed_sec": 83.92461490631104, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 2, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.1, "train_loss_last": 5.3705644607543945, "eval_loss": 5.689733423292637, "elapsed_sec": 42.08622622489929, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 2, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.033831596374512, "eval_loss": 5.1538038700819016, "elapsed_sec": 42.04193305969238, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 2, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.8757171630859375, "eval_loss": 4.890640087425709, "elapsed_sec": 42.02943682670593, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 2, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.631913185119629, "eval_loss": 4.67994287610054, "elapsed_sec": 42.03091287612915, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 2, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.8, "train_loss_last": 6.397376537322998, "eval_loss": 7.076408430933952, "elapsed_sec": 42.1185507774353, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 2, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.197352409362793, "eval_loss": 6.774318270385265, "elapsed_sec": 42.08896017074585, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 2, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.8, "train_loss_last": 6.076539993286133, "eval_loss": 6.475470535457134, "elapsed_sec": 42.058411836624146, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 2, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.8, "train_loss_last": 5.836711883544922, "eval_loss": 6.320266917347908, "elapsed_sec": 42.04223084449768, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 2, "stage": null, "token_limit": 40000000, "steps": 4000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.5983476638793945, "eval_loss": 4.709539167582989, "elapsed_sec": 166.29632186889648, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 3, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.094411849975586, "eval_loss": 6.762092016637325, "elapsed_sec": 83.63494491577148, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 3, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.1, "train_loss_last": 5.442709922790527, "eval_loss": 5.6250176429748535, "elapsed_sec": 41.97037887573242, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 3, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 4.9339494705200195, "eval_loss": 5.091545574367046, "elapsed_sec": 41.93742275238037, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 3, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.820074081420898, "eval_loss": 4.85930198431015, "elapsed_sec": 42.00108504295349, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 3, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.612112522125244, "eval_loss": 4.680890738964081, "elapsed_sec": 42.07970404624939, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 3, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.8, "train_loss_last": 6.521695613861084, "eval_loss": 6.976697988808155, "elapsed_sec": 42.128258228302, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 3, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.109576225280762, "eval_loss": 6.698994763195515, "elapsed_sec": 42.083462953567505, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 3, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.8, "train_loss_last": 6.1710710525512695, "eval_loss": 6.4792612716555595, "elapsed_sec": 42.04126191139221, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 3, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.8, "train_loss_last": 5.799557685852051, "eval_loss": 6.188394881784916, "elapsed_sec": 41.99322700500488, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 3, "stage": null, "token_limit": 40000000, "steps": 4000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.611385345458984, "eval_loss": 4.758806340396404, "elapsed_sec": 165.85685300827026, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 4, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.14980411529541, "eval_loss": 6.930917248129845, "elapsed_sec": 83.3329451084137, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 4, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.1, "train_loss_last": 5.4042534828186035, "eval_loss": 5.632189579308033, "elapsed_sec": 41.89421987533569, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 4, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.005560874938965, "eval_loss": 5.1024143025279045, "elapsed_sec": 41.94953107833862, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 4, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.75225830078125, "eval_loss": 4.807056985795498, "elapsed_sec": 41.98645496368408, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 4, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.604494094848633, "eval_loss": 4.6223960518836975, "elapsed_sec": 42.013452768325806, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 4, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.8, "train_loss_last": 6.446765899658203, "eval_loss": 6.958507925271988, "elapsed_sec": 42.097583055496216, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 4, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.149364471435547, "eval_loss": 6.848812915384769, "elapsed_sec": 42.05741786956787, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 4, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.8, "train_loss_last": 6.284854888916016, "eval_loss": 6.4219686314463615, "elapsed_sec": 42.04438519477844, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 4, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.8, "train_loss_last": 5.919014930725098, "eval_loss": 6.207054376602173, "elapsed_sec": 41.94203591346741, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 4, "stage": null, "token_limit": 40000000, "steps": 4000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.628224849700928, "eval_loss": 4.670411393046379, "elapsed_sec": 166.0933392047882, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_high_static_fixed_data", "seed": 5, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.167638778686523, "eval_loss": 6.828722849488258, "elapsed_sec": 83.42469191551208, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 5, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.1, "train_loss_last": 5.504278182983398, "eval_loss": 5.640969134867191, "elapsed_sec": 41.92028093338013, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 5, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.050647735595703, "eval_loss": 5.076318271458149, "elapsed_sec": 41.88388991355896, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 5, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.690304756164551, "eval_loss": 4.823811210691929, "elapsed_sec": 41.971457719802856, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_static_dropout_streaming", "seed": 5, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.762080669403076, "eval_loss": 4.646689280867577, "elapsed_sec": 41.99556493759155, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 5, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.8, "train_loss_last": 6.486720085144043, "eval_loss": 6.950839214026928, "elapsed_sec": 42.06647801399231, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 5, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.8, "train_loss_last": 6.209081649780273, "eval_loss": 6.823136441409588, "elapsed_sec": 42.13153791427612, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 5, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.8, "train_loss_last": 5.975912094116211, "eval_loss": 6.513336889445782, "elapsed_sec": 42.0465452671051, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_streaming_controls", "condition": "large_high_dropout_streaming", "seed": 5, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.8, "train_loss_last": 5.925265789031982, "eval_loss": 6.211368426680565, "elapsed_sec": 41.90367317199707, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase4_full_data_controls", "condition": "large_static_full_data_from_start", "seed": 5, "stage": null, "token_limit": 40000000, "steps": 4000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.792789459228516, "eval_loss": 4.656416833400726, "elapsed_sec": 166.22296500205994, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}

archive/runs_legacy_20260525/controls/20260522-141120/summary.csv

@@ -0,0 +1,11 @@
+phase,condition,stage,token_limit,parameters,n,mean_eval_loss,std_eval_loss
+phase2_fixed_data_dropout_optimization,large_high_static_fixed_data,,5000000,8388608,5,6.835472615063191,0.06128588298231963
+phase3_streaming_controls,large_high_dropout_streaming,0,5000000,8388608,5,6.9897750034928325,0.0504510436205119
+phase3_streaming_controls,large_high_dropout_streaming,1,10000000,8388608,5,6.763712777197361,0.07623697109669446
+phase3_streaming_controls,large_high_dropout_streaming,2,20000000,8388608,5,6.483462110161781,0.040849957965590676
+phase3_streaming_controls,large_high_dropout_streaming,3,40000000,8388608,5,6.238975651562214,0.054263911050486884
+phase3_streaming_controls,large_static_dropout_streaming,0,5000000,8388608,5,5.6309845224022865,0.0438193989069106
+phase3_streaming_controls,large_static_dropout_streaming,1,10000000,8388608,5,5.1018159747123715,0.0305841560371311
+phase3_streaming_controls,large_static_dropout_streaming,2,20000000,8388608,5,4.849676774442196,0.03382457670014637
+phase3_streaming_controls,large_static_dropout_streaming,3,40000000,8388608,5,4.674328573048115,0.0449378239644584
+phase4_full_data_controls,large_static_full_data_from_start,,40000000,8388608,5,4.691499495506287,0.04296035064933353

archive/runs_legacy_20260525/controls/20260522-141120/summary.json

@@ -0,0 +1,102 @@
+[
+  {
+    "phase": "phase2_fixed_data_dropout_optimization",
+    "condition": "large_high_static_fixed_data",
+    "stage": null,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.835472615063191,
+    "std_eval_loss": 0.06128588298231963
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_high_dropout_streaming",
+    "stage": 0,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.9897750034928325,
+    "std_eval_loss": 0.0504510436205119
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_high_dropout_streaming",
+    "stage": 1,
+    "token_limit": 10000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.763712777197361,
+    "std_eval_loss": 0.07623697109669446
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_high_dropout_streaming",
+    "stage": 2,
+    "token_limit": 20000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.483462110161781,
+    "std_eval_loss": 0.040849957965590676
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_high_dropout_streaming",
+    "stage": 3,
+    "token_limit": 40000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.238975651562214,
+    "std_eval_loss": 0.054263911050486884
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_static_dropout_streaming",
+    "stage": 0,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 5.6309845224022865,
+    "std_eval_loss": 0.0438193989069106
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_static_dropout_streaming",
+    "stage": 1,
+    "token_limit": 10000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 5.1018159747123715,
+    "std_eval_loss": 0.0305841560371311
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_static_dropout_streaming",
+    "stage": 2,
+    "token_limit": 20000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.849676774442196,
+    "std_eval_loss": 0.03382457670014637
+  },
+  {
+    "phase": "phase3_streaming_controls",
+    "condition": "large_static_dropout_streaming",
+    "stage": 3,
+    "token_limit": 40000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.674328573048115,
+    "std_eval_loss": 0.0449378239644584
+  },
+  {
+    "phase": "phase4_full_data_controls",
+    "condition": "large_static_full_data_from_start",
+    "stage": null,
+    "token_limit": 40000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.691499495506287,
+    "std_eval_loss": 0.04296035064933353
+  }
+]

archive/runs_legacy_20260525/publishable/20260522-132351/config.json

@@ -0,0 +1,60 @@
+{
+  "args": {
+    "corpus": null,
+    "corpus_glob": "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet",
+    "text_column": "text",
+    "output_dir": "runs/publishable",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 5000000,
+    "stream_token_caps": [
+      5000000,
+      10000000,
+      20000000,
+      40000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 2000,
+    "stream_steps_per_stage": 1000,
+    "eval_batches": 64,
+    "log_every": 500,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/publishable/20260522-132351/tokenizer.json",
+  "encoded_path": "runs/publishable/20260522-132351/tokens-v4096-uint16.npy",
+  "train_tokens": 40000160,
+  "val_tokens": 500000,
+  "effective_initial_tokens": 5000000,
+  "effective_stream_token_caps": [
+    5000000,
+    10000000,
+    20000000,
+    40000000
+  ]
+}

archive/runs_legacy_20260525/publishable/20260522-132351/metrics.jsonl

@@ -0,0 +1,35 @@
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 1, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.450620174407959, "eval_loss": 5.772196859121323, "elapsed_sec": 31.631267070770264, "parameters": 1835008, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 1, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 4.913602828979492, "eval_loss": 5.207479499280453, "elapsed_sec": 80.3967981338501, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 1, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.7820626459322383, "train_loss_last": 6.046256065368652, "eval_loss": 6.741213291883469, "elapsed_sec": 85.33950591087341, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.7733578363789504, "train_loss_last": 6.342434883117676, "eval_loss": 6.946014620363712, "elapsed_sec": 38.47059369087219, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.6974873734152917, "train_loss_last": 5.969331741333008, "eval_loss": 6.201295383274555, "elapsed_sec": 39.12462592124939, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.45000000000000007, "train_loss_last": 5.437297821044922, "eval_loss": 5.363524094223976, "elapsed_sec": 40.467276096343994, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.748871803283691, "eval_loss": 4.967737302184105, "elapsed_sec": 41.435073137283325, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 2, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.485232830047607, "eval_loss": 5.843787379562855, "elapsed_sec": 30.04755687713623, "parameters": 1835008, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 2, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 4.922041416168213, "eval_loss": 5.218974679708481, "elapsed_sec": 84.12733793258667, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 2, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.7820626459322383, "train_loss_last": 6.0871381759643555, "eval_loss": 6.744259864091873, "elapsed_sec": 84.6592960357666, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.7733578363789504, "train_loss_last": 6.306346893310547, "eval_loss": 6.984690964221954, "elapsed_sec": 42.66448211669922, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.6974873734152917, "train_loss_last": 5.959338188171387, "eval_loss": 6.249792195856571, "elapsed_sec": 42.512439250946045, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.45000000000000007, "train_loss_last": 5.383289813995361, "eval_loss": 5.421286106109619, "elapsed_sec": 42.43638801574707, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.842014312744141, "eval_loss": 4.901044279336929, "elapsed_sec": 42.34708309173584, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 3, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.422863960266113, "eval_loss": 5.829333141446114, "elapsed_sec": 29.95318865776062, "parameters": 1835008, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 3, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 4.909119606018066, "eval_loss": 5.191086798906326, "elapsed_sec": 81.83940720558167, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 3, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.7820626459322383, "train_loss_last": 6.057084083557129, "eval_loss": 6.660528361797333, "elapsed_sec": 81.67780780792236, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.7733578363789504, "train_loss_last": 6.432514190673828, "eval_loss": 6.905187286436558, "elapsed_sec": 41.10620903968811, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.6974873734152917, "train_loss_last": 5.846698760986328, "eval_loss": 6.245104171335697, "elapsed_sec": 41.87897992134094, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.45000000000000007, "train_loss_last": 5.401530742645264, "eval_loss": 5.512500010430813, "elapsed_sec": 42.506834983825684, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.839996337890625, "eval_loss": 4.933420121669769, "elapsed_sec": 42.479960203170776, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 4, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.519951820373535, "eval_loss": 5.800006277859211, "elapsed_sec": 29.89086604118347, "parameters": 1835008, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 4, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 4.959898471832275, "eval_loss": 5.166869513690472, "elapsed_sec": 84.1205141544342, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 4, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.7820626459322383, "train_loss_last": 6.105772018432617, "eval_loss": 6.792966462671757, "elapsed_sec": 84.61292290687561, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.7733578363789504, "train_loss_last": 6.376546859741211, "eval_loss": 6.8808704018592834, "elapsed_sec": 42.68510293960571, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.6974873734152917, "train_loss_last": 5.919679641723633, "eval_loss": 6.322697900235653, "elapsed_sec": 42.60045385360718, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.45000000000000007, "train_loss_last": 5.4210944175720215, "eval_loss": 5.348750911653042, "elapsed_sec": 42.2938597202301, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.828921318054199, "eval_loss": 4.846531391143799, "elapsed_sec": 42.36263298988342, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 5, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.451421737670898, "eval_loss": 5.83368007838726, "elapsed_sec": 29.718762159347534, "parameters": 1835008, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 5, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 4.846748352050781, "eval_loss": 5.233553200960159, "elapsed_sec": 83.74996209144592, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 5, "stage": null, "token_limit": 5000000, "steps": 2000, "tokens_seen": 4096000, "dropout": 0.7820626459322383, "train_loss_last": 6.124170780181885, "eval_loss": 6.755315415561199, "elapsed_sec": 84.20570206642151, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.7733578363789504, "train_loss_last": 6.408639907836914, "eval_loss": 6.889496520161629, "elapsed_sec": 42.334360122680664, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.6974873734152917, "train_loss_last": 5.972659111022949, "eval_loss": 6.325718097388744, "elapsed_sec": 42.13765597343445, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.45000000000000007, "train_loss_last": 5.278019428253174, "eval_loss": 5.48484568297863, "elapsed_sec": 42.261926889419556, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.95810079574585, "eval_loss": 4.896161742508411, "elapsed_sec": 42.27780485153198, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}

archive/runs_legacy_20260525/publishable/20260522-132351/summary.csv

@@ -0,0 +1,8 @@
+phase,condition,stage,token_limit,parameters,n,mean_eval_loss,std_eval_loss
+phase1_baseline_comparison,large_static,,5000000,8388608,5,5.203592738509178,0.02576011500878837
+phase1_baseline_comparison,small_static,,5000000,1835008,5,5.815800747275352,0.029312165077338317
+phase2_fixed_data_dropout_optimization,large_dropout_decay_fixed_data,,5000000,8388608,5,6.738856679201126,0.04839636351890629
+phase3_simulated_streaming_scale_up,large_dropout_decay_streaming,0,5000000,8388608,5,6.921251958608627,0.0434017526687061
+phase3_simulated_streaming_scale_up,large_dropout_decay_streaming,1,10000000,8388608,5,6.268921549618244,0.05390788852828186
+phase3_simulated_streaming_scale_up,large_dropout_decay_streaming,2,20000000,8388608,5,5.426181361079216,0.0721764902844538
+phase3_simulated_streaming_scale_up,large_dropout_decay_streaming,3,40000000,8388608,5,4.908978967368602,0.045212974458371656

archive/runs_legacy_20260525/publishable/20260522-132351/summary.json

@@ -0,0 +1,72 @@
+[
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "large_static",
+    "stage": null,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 5.203592738509178,
+    "std_eval_loss": 0.02576011500878837
+  },
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "small_static",
+    "stage": null,
+    "token_limit": 5000000,
+    "parameters": 1835008,
+    "n": 5,
+    "mean_eval_loss": 5.815800747275352,
+    "std_eval_loss": 0.029312165077338317
+  },
+  {
+    "phase": "phase2_fixed_data_dropout_optimization",
+    "condition": "large_dropout_decay_fixed_data",
+    "stage": null,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.738856679201126,
+    "std_eval_loss": 0.04839636351890629
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 0,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.921251958608627,
+    "std_eval_loss": 0.0434017526687061
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 1,
+    "token_limit": 10000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.268921549618244,
+    "std_eval_loss": 0.05390788852828186
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 2,
+    "token_limit": 20000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 5.426181361079216,
+    "std_eval_loss": 0.0721764902844538
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 3,
+    "token_limit": 40000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.908978967368602,
+    "std_eval_loss": 0.045212974458371656
+  }
+]

archive/runs_legacy_20260525/smoke/20260522-132106/config.json

@@ -0,0 +1,51 @@
+{
+  "args": {
+    "corpus": "/Users/mandeepsidhu/Desktop/code/transformer/data/tinystories_train.txt",
+    "corpus_glob": null,
+    "text_column": "text",
+    "output_dir": "runs/smoke",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 20000,
+    "stream_token_caps": [
+      20000,
+      40000,
+      80000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": true,
+    "force_retokenize": false,
+    "vocab_size": 512,
+    "tokenizer_train_chars": 200000,
+    "block_size": 32,
+    "batch_size": 4,
+    "steps_per_run": 2,
+    "stream_steps_per_stage": 1,
+    "eval_batches": 1,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/smoke/20260522-132106/tokenizer.json",
+  "encoded_path": "runs/smoke/20260522-132106/tokens-v512-uint16.npy",
+  "train_tokens": 777983,
+  "val_tokens": 86442,
+  "effective_initial_tokens": 20000,
+  "effective_stream_token_caps": [
+    20000,
+    40000,
+    80000
+  ]
+}

archive/runs_legacy_20260525/smoke/20260522-132106/metrics.jsonl

@@ -0,0 +1,30 @@
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 1, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.231201171875, "eval_loss": 6.233233451843262, "elapsed_sec": 2.1697847843170166, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 1, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.222594738006592, "eval_loss": 6.22459602355957, "elapsed_sec": 0.2784230709075928, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 1, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.238912582397461, "eval_loss": 6.234484672546387, "elapsed_sec": 0.0704340934753418, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.240694999694824, "eval_loss": 6.236194610595703, "elapsed_sec": 0.04119396209716797, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.233834266662598, "eval_loss": 6.2317962646484375, "elapsed_sec": 0.032363176345825195, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231166839599609, "eval_loss": 6.221035957336426, "elapsed_sec": 0.030178070068359375, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 2, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.234216690063477, "eval_loss": 6.230881214141846, "elapsed_sec": 0.03483176231384277, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 2, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.228965759277344, "eval_loss": 6.218391418457031, "elapsed_sec": 0.06615877151489258, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 2, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.237312316894531, "eval_loss": 6.230140209197998, "elapsed_sec": 0.0667262077331543, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.23726749420166, "eval_loss": 6.232943534851074, "elapsed_sec": 0.03887295722961426, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.2372612953186035, "eval_loss": 6.226807594299316, "elapsed_sec": 0.02962803840637207, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.228798866271973, "eval_loss": 6.214606285095215, "elapsed_sec": 0.0299530029296875, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 3, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.236220359802246, "eval_loss": 6.231494903564453, "elapsed_sec": 0.032794952392578125, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 3, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.231418132781982, "eval_loss": 6.218476295471191, "elapsed_sec": 0.06430721282958984, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 3, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.236932754516602, "eval_loss": 6.230288505554199, "elapsed_sec": 0.06323909759521484, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.238128662109375, "eval_loss": 6.2329301834106445, "elapsed_sec": 0.03883004188537598, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.236502170562744, "eval_loss": 6.225698471069336, "elapsed_sec": 0.03026604652404785, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231315612792969, "eval_loss": 6.213197708129883, "elapsed_sec": 0.030527830123901367, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 4, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.235054016113281, "eval_loss": 6.234989166259766, "elapsed_sec": 0.03532004356384277, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 4, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.231119632720947, "eval_loss": 6.223240852355957, "elapsed_sec": 0.065032958984375, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 4, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.237499237060547, "eval_loss": 6.234406471252441, "elapsed_sec": 0.06367802619934082, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.239429950714111, "eval_loss": 6.235345363616943, "elapsed_sec": 0.03851008415222168, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.23697566986084, "eval_loss": 6.229433536529541, "elapsed_sec": 0.030126094818115234, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231019020080566, "eval_loss": 6.218218803405762, "elapsed_sec": 0.02992105484008789, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 5, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.236649513244629, "eval_loss": 6.2315874099731445, "elapsed_sec": 0.034050703048706055, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 5, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.230384826660156, "eval_loss": 6.221922874450684, "elapsed_sec": 0.06528592109680176, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 5, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.236981391906738, "eval_loss": 6.234584331512451, "elapsed_sec": 0.06582903861999512, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.239423751831055, "eval_loss": 6.236317157745361, "elapsed_sec": 0.04046988487243652, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.234818458557129, "eval_loss": 6.230718612670898, "elapsed_sec": 0.0317072868347168, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.232948303222656, "eval_loss": 6.2211151123046875, "elapsed_sec": 0.030757665634155273, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}

archive/runs_legacy_20260525/smoke/20260522-132106/summary.json

@@ -0,0 +1,62 @@
+[
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "large_static",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.221325492858886,
+    "std_eval_loss": 0.0028039506140008267
+  },
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "small_static",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 917504,
+    "n": 5,
+    "mean_eval_loss": 6.232437229156494,
+    "std_eval_loss": 0.001671653854539933
+  },
+  {
+    "phase": "phase2_fixed_data_dropout_optimization",
+    "condition": "large_dropout_decay_fixed_data",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.232780838012696,
+    "std_eval_loss": 0.0023442997873944736
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 0,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.2347461700439455,
+    "std_eval_loss": 0.0016935404053936482
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 1,
+    "token_limit": 40000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.228890895843506,
+    "std_eval_loss": 0.0025791421476287733
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 2,
+    "token_limit": 80000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.217634773254394,
+    "std_eval_loss": 0.0036359727629622267
+  }
+]

archive/runs_legacy_20260525/smoke/20260522-132106/tokenizer.json

@@ -0,0 +1,1629 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<|bos|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<|user_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<|user_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<|assistant_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<|assistant_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "<|python_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 6,
+      "content": "<|python_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 7,
+      "content": "<|output_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 8,
+      "content": "<|output_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Sequence",
+    "pretokenizers": [
+      {
+        "type": "Split",
+        "pattern": {
+          "Regex": "'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?+\\p{L}+|\\p{N}{1,2}| ?[^\\s\\p{L}\\p{N}]++[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+"
+        },
+        "behavior": "Isolated",
+        "invert": false
+      },
+      {
+        "type": "ByteLevel",
+        "add_prefix_space": false,
+        "trim_offsets": true,
+        "use_regex": false
+      }
+    ]
+  },
+  "post_processor": null,
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": true,
+    "ignore_merges": false,
+    "vocab": {
+      "<|bos|>": 0,
+      "<|user_start|>": 1,
+      "<|user_end|>": 2,
+      "<|assistant_start|>": 3,
+      "<|assistant_end|>": 4,
+      "<|python_start|>": 5,
+      "<|python_end|>": 6,
+      "<|output_start|>": 7,
+      "<|output_end|>": 8,
+      "!": 9,
+      "\"": 10,
+      "#": 11,
+      "$": 12,
+      "%": 13,
+      "&": 14,
+      "'": 15,
+      "(": 16,
+      ")": 17,
+      "*": 18,
+      "+": 19,
+      ",": 20,
+      "-": 21,
+      ".": 22,
+      "/": 23,
+      "0": 24,
+      "1": 25,
+      "2": 26,
+      "3": 27,
+      "4": 28,
+      "5": 29,
+      "6": 30,
+      "7": 31,
+      "8": 32,
+      "9": 33,
+      ":": 34,
+      ";": 35,
+      "<": 36,
+      "=": 37,
+      ">": 38,
+      "?": 39,
+      "@": 40,
+      "A": 41,
+      "B": 42,
+      "C": 43,
+      "D": 44,
+      "E": 45,
+      "F": 46,
+      "G": 47,
+      "H": 48,
+      "I": 49,
+      "J": 50,
+      "K": 51,
+      "L": 52,
+      "M": 53,
+      "N": 54,
+      "O": 55,
+      "P": 56,
+      "Q": 57,
+      "R": 58,
+      "S": 59,
+      "T": 60,
+      "U": 61,
+      "V": 62,
+      "W": 63,
+      "X": 64,
+      "Y": 65,
+      "Z": 66,
+      "[": 67,
+      "\\": 68,
+      "]": 69,
+      "^": 70,
+      "_": 71,
+      "`": 72,
+      "a": 73,
+      "b": 74,
+      "c": 75,
+      "d": 76,
+      "e": 77,
+      "f": 78,
+      "g": 79,
+      "h": 80,
+      "i": 81,
+      "j": 82,
+      "k": 83,
+      "l": 84,
+      "m": 85,
+      "n": 86,
+      "o": 87,
+      "p": 88,
+      "q": 89,
+      "r": 90,
+      "s": 91,
+      "t": 92,
+      "u": 93,
+      "v": 94,
+      "w": 95,
+      "x": 96,
+      "y": 97,
+      "z": 98,
+      "{": 99,
+      "|": 100,
+      "}": 101,
+      "~": 102,
+      "¡": 103,
+      "¢": 104,
+      "£": 105,
+      "¤": 106,
+      "¥": 107,
+      "¦": 108,
+      "§": 109,
+      "¨": 110,
+      "©": 111,
+      "ª": 112,
+      "«": 113,
+      "¬": 114,
+      "®": 115,
+      "¯": 116,
+      "°": 117,
+      "±": 118,
+      "²": 119,
+      "³": 120,
+      "´": 121,
+      "µ": 122,
+      "¶": 123,
+      "·": 124,
+      "¸": 125,
+      "¹": 126,
+      "º": 127,
+      "»": 128,
+      "¼": 129,
+      "½": 130,
+      "¾": 131,
+      "¿": 132,
+      "À": 133,
+      "Á": 134,
+      "Â": 135,
+      "Ã": 136,
+      "Ä": 137,
+      "Å": 138,
+      "Æ": 139,
+      "Ç": 140,
+      "È": 141,
+      "É": 142,
+      "Ê": 143,
+      "Ë": 144,
+      "Ì": 145,
+      "Í": 146,
+      "Î": 147,
+      "Ï": 148,
+      "Ð": 149,
+      "Ñ": 150,
+      "Ò": 151,
+      "Ó": 152,
+      "Ô": 153,
+      "Õ": 154,
+      "Ö": 155,
+      "×": 156,
+      "Ø": 157,
+      "Ù": 158,
+      "Ú": 159,
+      "Û": 160,
+      "Ü": 161,
+      "Ý": 162,
+      "Þ": 163,
+      "ß": 164,
+      "à": 165,
+      "á": 166,
+      "â": 167,
+      "ã": 168,
+      "ä": 169,
+      "å": 170,
+      "æ": 171,
+      "ç": 172,
+      "è": 173,
+      "é": 174,
+      "ê": 175,
+      "ë": 176,
+      "ì": 177,
+      "í": 178,
+      "î": 179,
+      "ï": 180,
+      "ð": 181,
+      "ñ": 182,
+      "ò": 183,
+      "ó": 184,
+      "ô": 185,
+      "õ": 186,
+      "ö": 187,
+      "÷": 188,
+      "ø": 189,
+      "ù": 190,
+      "ú": 191,
+      "û": 192,
+      "ü": 193,
+      "ý": 194,
+      "þ": 195,
+      "ÿ": 196,
+      "Ā": 197,
+      "ā": 198,
+      "Ă": 199,
+      "ă": 200,
+      "Ą": 201,
+      "ą": 202,
+      "Ć": 203,
+      "ć": 204,
+      "Ĉ": 205,
+      "ĉ": 206,
+      "Ċ": 207,
+      "ċ": 208,
+      "Č": 209,
+      "č": 210,
+      "Ď": 211,
+      "ď": 212,
+      "Đ": 213,
+      "đ": 214,
+      "Ē": 215,
+      "ē": 216,
+      "Ĕ": 217,
+      "ĕ": 218,
+      "Ė": 219,
+      "ė": 220,
+      "Ę": 221,
+      "ę": 222,
+      "Ě": 223,
+      "ě": 224,
+      "Ĝ": 225,
+      "ĝ": 226,
+      "Ğ": 227,
+      "ğ": 228,
+      "Ġ": 229,
+      "ġ": 230,
+      "Ģ": 231,
+      "ģ": 232,
+      "Ĥ": 233,
+      "ĥ": 234,
+      "Ħ": 235,
+      "ħ": 236,
+      "Ĩ": 237,
+      "ĩ": 238,
+      "Ī": 239,
+      "ī": 240,
+      "Ĭ": 241,
+      "ĭ": 242,
+      "Į": 243,
+      "į": 244,
+      "İ": 245,
+      "ı": 246,
+      "IJ": 247,
+      "ij": 248,
+      "Ĵ": 249,
+      "ĵ": 250,
+      "Ķ": 251,
+      "ķ": 252,
+      "ĸ": 253,
+      "Ĺ": 254,
+      "ĺ": 255,
+      "Ļ": 256,
+      "ļ": 257,
+      "Ľ": 258,
+      "ľ": 259,
+      "Ŀ": 260,
+      "ŀ": 261,
+      "Ł": 262,
+      "ł": 263,
+      "Ń": 264,
+      "he": 265,
+      "Ġt": 266,
+      "Ġa": 267,
+      "Ġs": 268,
+      "Ġw": 269,
+      "Ġthe": 270,
+      "nd": 271,
+      "ed": 272,
+      "in": 273,
+      "Ġand": 274,
+      "Ġwa": 275,
+      "Ġb": 276,
+      "Ġto": 277,
+      "re": 278,
+      "Ġh": 279,
+      "ou": 280,
+      "it": 281,
+      "Ġf": 282,
+      "er": 283,
+      "ĊĊ": 284,
+      "Ġwas": 285,
+      "Ġl": 286,
+      "Ġc": 287,
+      "Ġhe": 288,
+      "Ġp": 289,
+      "ing": 290,
+      "Ġd": 291,
+      "Ġm": 292,
+      "Ġo": 293,
+      "Ġg": 294,
+      "ar": 295,
+      "is": 296,
+      "id": 297,
+      "ay": 298,
+      "om": 299,
+      "at": 300,
+      "ll": 301,
+      "en": 302,
+      "Ġsa": 303,
+      "ne": 304,
+      "The": 305,
+      ".ĊĊ": 306,
+      "le": 307,
+      "Ġth": 308,
+      "im": 309,
+      "an": 310,
+      "Ġha": 311,
+      "or": 312,
+      "Ġit": 313,
+      "et": 314,
+      "ver": 315,
+      "ld": 316,
+      "Ġin": 317,
+      "ĠS": 318,
+      "on": 319,
+      "Ġe": 320,
+      "ce": 321,
+      "Ġbe": 322,
+      "Ġher": 323,
+      "ir": 324,
+      "Ġ\"": 325,
+      "ĠH": 326,
+      "Ġu": 327,
+      "Ġsaid": 328,
+      "Ġn": 329,
+      "ck": 330,
+      "ow": 331,
+      "ri": 332,
+      "ĠThe": 333,
+      "Ġshe": 334,
+      "Ġso": 335,
+      "st": 336,
+      "Ġy": 337,
+      "ot": 338,
+      "ĠHe": 339,
+      "Ġof": 340,
+      "il": 341,
+      "Ġst": 342,
+      "ut": 343,
+      "ke": 344,
+      "am": 345,
+      "ked": 346,
+      "oo": 347,
+      "pp": 348,
+      "Ġr": 349,
+      "ĠShe": 350,
+      "very": 351,
+      "ĠI": 352,
+      "ve": 353,
+      "Ġthat": 354,
+      "ig": 355,
+      "ith": 356,
+      "Ġhis": 357,
+      "Ġup": 358,
+      "ĠĊĊ": 359,
+      "Ġday": 360,
+      "Ġwith": 361,
+      "Ġpl": 362,
+      "Ġyou": 363,
+      "itt": 364,
+      "ould": 365,
+      "el": 366,
+      "ted": 367,
+      "ent": 368,
+      "ad": 369,
+      "Ġhad": 370,
+      "ound": 371,
+      "al": 372,
+      "ĠJ": 373,
+      "Ġwe": 374,
+      "her": 375,
+      "ittle": 376,
+      "'s": 377,
+      "Ġsm": 378,
+      "Ġplay": 379,
+      "end": 380,
+      "Ġthey": 381,
+      "ack": 382,
+      "Ġthere": 383,
+      "ime": 384,
+      "ly": 385,
+      "Ġsh": 386,
+      "Ġlittle": 387,
+      "Ġre": 388,
+      "Ġne": 389,
+      "Ġtime": 390,
+      "out": 391,
+      "Ġfor": 392,
+      "un": 393,
+      "ch": 394,
+      "se": 395,
+      "Ġhapp": 396,
+      "Ġwh": 397,
+      "my": 398,
+      "ome": 399,
+      "ht": 400,
+      "um": 401,
+      "Ġfri": 402,
+      "Ġas": 403,
+      "Ġfriend": 404,
+      "Ġvery": 405,
+      "all": 406,
+      "ter": 407,
+      "â": 408,
+      "âĤ": 409,
+      "âĤ¬": 410,
+      "On": 411,
+      "Ġk": 412,
+      "ved": 413,
+      "ĠT": 414,
+      "Ġon": 415,
+      "irl": 416,
+      "Once": 417,
+      "ug": 418,
+      "\"ĊĊ": 419,
+      "ill": 420,
+      "Ġgirl": 421,
+      "Ġan": 422,
+      "es": 423,
+      "Ġex": 424,
+      "'t": 425,
+      "ec": 426,
+      "Ġbut": 427,
+      "Ġloo": 428,
+      "Ġli": 429,
+      "Ġbo": 430,
+      "Ġwere": 431,
+      "One": 432,
+      "Ġwan": 433,
+      "Ġhappy": 434,
+      "ake": 435,
+      "ore": 436,
+      "Ġbig": 437,
+      "fu": 438,
+      "Ġsp": 439,
+      "ide": 440,
+      "Ġsaw": 441,
+      "ĠB": 442,
+      "hing": 443,
+      "Ġupon": 444,
+      "ard": 445,
+      "Ġcould": 446,
+      "ic": 447,
+      "Ġout": 448,
+      "iled": 449,
+      "one": 450,
+      "round": 451,
+      "ra": 452,
+      "ry": 453,
+      "Ġsmiled": 454,
+      "Ġhim": 455,
+      "ĠA": 456,
+      "Ġmom": 457,
+      "hen": 458,
+      "way": 459,
+      "ur": 460,
+      "ĠIt": 461,
+      "ful": 462,
+      "ain": 463,
+      "Ġwent": 464,
+      "Ġhel": 465,
+      "Ġnot": 466,
+      "ĠThey": 467,
+      "Ġwanted": 468,
+      "ind": 469,
+      "are": 470,
+      "ear": 471,
+      "ĠM": 472,
+      "Ġall": 473,
+      "Ġfriends": 474,
+      "Ġtoo": 475,
+      "Ġgo": 476,
+      "ily": 477,
+      "ame": 478,
+      "ĠTim": 479,
+      "Ġhelp": 480,
+      "omet": 481,
+      "ĠL": 482,
+      "Ġlo": 483,
+      "ght": 484,
+      "Ġsomet": 485,
+      "Ġat": 486,
+      "Ġdo": 487,
+      "Ġasked": 488,
+      "!\"": 489,
+      "Ġaround": 490,
+      "Ġj": 491,
+      "ree": 492,
+      "Ġlooked": 493,
+      "Ġsomething": 494,
+      "Ġse": 495,
+      "Ġwor": 496,
+      "dd": 497,
+      "hed": 498,
+      "ood": 499,
+      "Ġcl": 500,
+      "amed": 501,
+      "ro": 502,
+      "Ġcan": 503,
+      "ark": 504,
+      "king": 505,
+      "ĠE": 506,
+      "rom": 507,
+      "Ġback": 508,
+      "Ġexc": 509,
+      "ab": 510,
+      "ick": 511
+    },
+    "merges": [
+      [
+        "h",
+        "e"
+      ],
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "n",
+        "d"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "Ġa",
+        "nd"
+      ],
+      [
+        "Ġw",
+        "a"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "Ċ",
+        "Ċ"
+      ],
+      [
+        "Ġwa",
+        "s"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "Ġ",
+        "g"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "a",
+        "y"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "l",
+        "l"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġs",
+        "a"
+      ],
+      [
+        "n",
+        "e"
+      ],
+      [
+        "T",
+        "he"
+      ],
+      [
+        ".",
+        "ĊĊ"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "Ġh",
+        "a"
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "Ġ",
+        "S"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġ",
+        "e"
+      ],
+      [
+        "c",
+        "e"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "Ġhe",
+        "r"
+      ],
+      [
+        "i",
+        "r"
+      ],
+      [
+        "Ġ",
+        "\""
+      ],
+      [
+        "Ġ",
+        "H"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "Ġsa",
+        "id"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "c",
+        "k"
+      ],
+      [
+        "o",
+        "w"
+      ],
+      [
+        "r",
+        "i"
+      ],
+      [
+        "Ġ",
+        "The"
+      ],
+      [
+        "Ġs",
+        "he"
+      ],
+      [
+        "Ġs",
+        "o"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "Ġ",
+        "y"
+      ],
+      [
+        "o",
+        "t"
+      ],
+      [
+        "ĠH",
+        "e"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "Ġs",
+        "t"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "k",
+        "e"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "k",
+        "ed"
+      ],
+      [
+        "o",
+        "o"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "ĠS",
+        "he"
+      ],
+      [
+        "ver",
+        "y"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "i",
+        "g"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "Ġh",
+        "is"
+      ],
+      [
+        "Ġu",
+        "p"
+      ],
+      [
+        "Ġ",
+        "ĊĊ"
+      ],
+      [
+        "Ġd",
+        "ay"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "Ġp",
+        "l"
+      ],
+      [
+        "Ġy",
+        "ou"
+      ],
+      [
+        "it",
+        "t"
+      ],
+      [
+        "ou",
+        "ld"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "t",
+        "ed"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "Ġha",
+        "d"
+      ],
+      [
+        "ou",
+        "nd"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "Ġ",
+        "J"
+      ],
+      [
+        "Ġw",
+        "e"
+      ],
+      [
+        "he",
+        "r"
+      ],
+      [
+        "itt",
+        "le"
+      ],
+      [
+        "'",
+        "s"
+      ],
+      [
+        "Ġs",
+        "m"
+      ],
+      [
+        "Ġpl",
+        "ay"
+      ],
+      [
+        "e",
+        "nd"
+      ],
+      [
+        "Ġthe",
+        "y"
+      ],
+      [
+        "a",
+        "ck"
+      ],
+      [
+        "Ġthe",
+        "re"
+      ],
+      [
+        "im",
+        "e"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "Ġs",
+        "h"
+      ],
+      [
+        "Ġl",
+        "ittle"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġ",
+        "ne"
+      ],
+      [
+        "Ġt",
+        "ime"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "u",
+        "n"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "Ġha",
+        "pp"
+      ],
+      [
+        "Ġw",
+        "h"
+      ],
+      [
+        "m",
+        "y"
+      ],
+      [
+        "om",
+        "e"
+      ],
+      [
+        "h",
+        "t"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "Ġf",
+        "ri"
+      ],
+      [
+        "Ġa",
+        "s"
+      ],
+      [
+        "Ġfri",
+        "end"
+      ],
+      [
+        "Ġ",
+        "very"
+      ],
+      [
+        "a",
+        "ll"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ã",
+        "¢"
+      ],
+      [
+        "â",
+        "Ĥ"
+      ],
+      [
+        "âĤ",
+        "¬"
+      ],
+      [
+        "O",
+        "n"
+      ],
+      [
+        "Ġ",
+        "k"
+      ],
+      [
+        "v",
+        "ed"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "Ġo",
+        "n"
+      ],
+      [
+        "ir",
+        "l"
+      ],
+      [
+        "On",
+        "ce"
+      ],
+      [
+        "u",
+        "g"
+      ],
+      [
+        "\"",
+        "ĊĊ"
+      ],
+      [
+        "i",
+        "ll"
+      ],
+      [
+        "Ġg",
+        "irl"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "Ġe",
+        "x"
+      ],
+      [
+        "'",
+        "t"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "Ġb",
+        "ut"
+      ],
+      [
+        "Ġl",
+        "oo"
+      ],
+      [
+        "Ġl",
+        "i"
+      ],
+      [
+        "Ġb",
+        "o"
+      ],
+      [
+        "Ġwe",
+        "re"
+      ],
+      [
+        "O",
+        "ne"
+      ],
+      [
+        "Ġwa",
+        "n"
+      ],
+      [
+        "Ġhapp",
+        "y"
+      ],
+      [
+        "a",
+        "ke"
+      ],
+      [
+        "o",
+        "re"
+      ],
+      [
+        "Ġb",
+        "ig"
+      ],
+      [
+        "f",
+        "u"
+      ],
+      [
+        "Ġs",
+        "p"
+      ],
+      [
+        "id",
+        "e"
+      ],
+      [
+        "Ġsa",
+        "w"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "h",
+        "ing"
+      ],
+      [
+        "Ġup",
+        "on"
+      ],
+      [
+        "ar",
+        "d"
+      ],
+      [
+        "Ġc",
+        "ould"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "Ġ",
+        "out"
+      ],
+      [
+        "il",
+        "ed"
+      ],
+      [
+        "o",
+        "ne"
+      ],
+      [
+        "r",
+        "ound"
+      ],
+      [
+        "r",
+        "a"
+      ],
+      [
+        "r",
+        "y"
+      ],
+      [
+        "Ġsm",
+        "iled"
+      ],
+      [
+        "Ġh",
+        "im"
+      ],
+      [
+        "Ġ",
+        "A"
+      ],
+      [
+        "Ġm",
+        "om"
+      ],
+      [
+        "he",
+        "n"
+      ],
+      [
+        "w",
+        "ay"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "ĠI",
+        "t"
+      ],
+      [
+        "fu",
+        "l"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "Ġw",
+        "ent"
+      ],
+      [
+        "Ġhe",
+        "l"
+      ],
+      [
+        "Ġn",
+        "ot"
+      ],
+      [
+        "ĠThe",
+        "y"
+      ],
+      [
+        "Ġwan",
+        "ted"
+      ],
+      [
+        "i",
+        "nd"
+      ],
+      [
+        "a",
+        "re"
+      ],
+      [
+        "e",
+        "ar"
+      ],
+      [
+        "Ġ",
+        "M"
+      ],
+      [
+        "Ġa",
+        "ll"
+      ],
+      [
+        "Ġfriend",
+        "s"
+      ],
+      [
+        "Ġto",
+        "o"
+      ],
+      [
+        "Ġg",
+        "o"
+      ],
+      [
+        "il",
+        "y"
+      ],
+      [
+        "am",
+        "e"
+      ],
+      [
+        "ĠT",
+        "im"
+      ],
+      [
+        "Ġhel",
+        "p"
+      ],
+      [
+        "om",
+        "et"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "Ġl",
+        "o"
+      ],
+      [
+        "g",
+        "ht"
+      ],
+      [
+        "Ġs",
+        "omet"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "Ġd",
+        "o"
+      ],
+      [
+        "Ġas",
+        "ked"
+      ],
+      [
+        "!",
+        "\""
+      ],
+      [
+        "Ġa",
+        "round"
+      ],
+      [
+        "Ġ",
+        "j"
+      ],
+      [
+        "re",
+        "e"
+      ],
+      [
+        "Ġloo",
+        "ked"
+      ],
+      [
+        "Ġsomet",
+        "hing"
+      ],
+      [
+        "Ġs",
+        "e"
+      ],
+      [
+        "Ġw",
+        "or"
+      ],
+      [
+        "d",
+        "d"
+      ],
+      [
+        "he",
+        "d"
+      ],
+      [
+        "oo",
+        "d"
+      ],
+      [
+        "Ġc",
+        "l"
+      ],
+      [
+        "am",
+        "ed"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "ar",
+        "k"
+      ],
+      [
+        "k",
+        "ing"
+      ],
+      [
+        "Ġ",
+        "E"
+      ],
+      [
+        "r",
+        "om"
+      ],
+      [
+        "Ġb",
+        "ack"
+      ],
+      [
+        "Ġex",
+        "c"
+      ],
+      [
+        "a",
+        "b"
+      ],
+      [
+        "i",
+        "ck"
+      ]
+    ]
+  }
+}

archive/runs_legacy_20260525/smoke/20260522-132219/config.json

@@ -0,0 +1,57 @@
+{
+  "args": {
+    "corpus": "/Users/mandeepsidhu/Desktop/code/transformer/data/tinystories_train.txt",
+    "corpus_glob": null,
+    "text_column": "text",
+    "output_dir": "runs/smoke",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 20000,
+    "stream_token_caps": [
+      20000,
+      40000,
+      80000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": true,
+    "force_retokenize": false,
+    "vocab_size": 512,
+    "tokenizer_train_chars": 200000,
+    "block_size": 32,
+    "batch_size": 4,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 2,
+    "stream_steps_per_stage": 1,
+    "eval_batches": 1,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/smoke/20260522-132219/tokenizer.json",
+  "encoded_path": "runs/smoke/20260522-132219/tokens-v512-uint16.npy",
+  "train_tokens": 777983,
+  "val_tokens": 86442,
+  "effective_initial_tokens": 20000,
+  "effective_stream_token_caps": [
+    20000,
+    40000,
+    80000
+  ]
+}

archive/runs_legacy_20260525/smoke/20260522-132219/metrics.jsonl

@@ -0,0 +1,30 @@
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 1, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.231201171875, "eval_loss": 6.233233451843262, "elapsed_sec": 0.16947412490844727, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 1, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.222594738006592, "eval_loss": 6.22459602355957, "elapsed_sec": 0.14142203330993652, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 1, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.238912582397461, "eval_loss": 6.234484672546387, "elapsed_sec": 0.07789421081542969, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.240694999694824, "eval_loss": 6.236194610595703, "elapsed_sec": 0.04669904708862305, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.233834266662598, "eval_loss": 6.2317962646484375, "elapsed_sec": 0.03601408004760742, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231166839599609, "eval_loss": 6.221035957336426, "elapsed_sec": 0.03204011917114258, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 2, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.234216690063477, "eval_loss": 6.230881214141846, "elapsed_sec": 0.03834128379821777, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 2, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.228965759277344, "eval_loss": 6.218391418457031, "elapsed_sec": 0.07362222671508789, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 2, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.237312316894531, "eval_loss": 6.230140209197998, "elapsed_sec": 0.07616114616394043, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.23726749420166, "eval_loss": 6.232943534851074, "elapsed_sec": 0.04496598243713379, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.2372612953186035, "eval_loss": 6.226807594299316, "elapsed_sec": 0.03384113311767578, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.228798866271973, "eval_loss": 6.214606285095215, "elapsed_sec": 0.03626108169555664, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 3, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.236220359802246, "eval_loss": 6.231494903564453, "elapsed_sec": 0.04115915298461914, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 3, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.231418132781982, "eval_loss": 6.218476295471191, "elapsed_sec": 0.07224202156066895, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 3, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.236932754516602, "eval_loss": 6.230288505554199, "elapsed_sec": 0.06969714164733887, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.238128662109375, "eval_loss": 6.2329301834106445, "elapsed_sec": 0.04061102867126465, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.236502170562744, "eval_loss": 6.225698471069336, "elapsed_sec": 0.0318758487701416, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231315612792969, "eval_loss": 6.213197708129883, "elapsed_sec": 0.032286882400512695, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 4, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.235054016113281, "eval_loss": 6.234989166259766, "elapsed_sec": 0.0366358757019043, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 4, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.231119632720947, "eval_loss": 6.223240852355957, "elapsed_sec": 0.06865692138671875, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 4, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.237499237060547, "eval_loss": 6.234406471252441, "elapsed_sec": 0.07039403915405273, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.239429950714111, "eval_loss": 6.235345363616943, "elapsed_sec": 0.04240703582763672, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.23697566986084, "eval_loss": 6.229433536529541, "elapsed_sec": 0.0330810546875, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231019020080566, "eval_loss": 6.218218803405762, "elapsed_sec": 0.033483028411865234, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 5, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.236649513244629, "eval_loss": 6.2315874099731445, "elapsed_sec": 0.03715205192565918, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 5, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.1, "train_loss_last": 6.230384826660156, "eval_loss": 6.221922874450684, "elapsed_sec": 0.06780505180358887, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 5, "stage": null, "token_limit": 20000, "steps": 2, "tokens_seen": 256, "dropout": 0.799995578426538, "train_loss_last": 6.236981391906738, "eval_loss": 6.234584331512451, "elapsed_sec": 0.06941580772399902, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.239423751831055, "eval_loss": 6.236317157745361, "elapsed_sec": 0.04212617874145508, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.234818458557129, "eval_loss": 6.230718612670898, "elapsed_sec": 0.03323984146118164, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.232948303222656, "eval_loss": 6.2211151123046875, "elapsed_sec": 0.033397674560546875, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}

archive/runs_legacy_20260525/smoke/20260522-132219/summary.json

@@ -0,0 +1,62 @@
+[
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "large_static",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.221325492858886,
+    "std_eval_loss": 0.0028039506140008267
+  },
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "small_static",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 917504,
+    "n": 5,
+    "mean_eval_loss": 6.232437229156494,
+    "std_eval_loss": 0.001671653854539933
+  },
+  {
+    "phase": "phase2_fixed_data_dropout_optimization",
+    "condition": "large_dropout_decay_fixed_data",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.232780838012696,
+    "std_eval_loss": 0.0023442997873944736
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 0,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.2347461700439455,
+    "std_eval_loss": 0.0016935404053936482
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 1,
+    "token_limit": 40000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.228890895843506,
+    "std_eval_loss": 0.0025791421476287733
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 2,
+    "token_limit": 80000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.217634773254394,
+    "std_eval_loss": 0.0036359727629622267
+  }
+]

archive/runs_legacy_20260525/smoke/20260522-132219/tokenizer.json

@@ -0,0 +1,1629 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<|bos|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<|user_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<|user_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<|assistant_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<|assistant_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "<|python_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 6,
+      "content": "<|python_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 7,
+      "content": "<|output_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 8,
+      "content": "<|output_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Sequence",
+    "pretokenizers": [
+      {
+        "type": "Split",
+        "pattern": {
+          "Regex": "'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?+\\p{L}+|\\p{N}{1,2}| ?[^\\s\\p{L}\\p{N}]++[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+"
+        },
+        "behavior": "Isolated",
+        "invert": false
+      },
+      {
+        "type": "ByteLevel",
+        "add_prefix_space": false,
+        "trim_offsets": true,
+        "use_regex": false
+      }
+    ]
+  },
+  "post_processor": null,
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": true,
+    "ignore_merges": false,
+    "vocab": {
+      "<|bos|>": 0,
+      "<|user_start|>": 1,
+      "<|user_end|>": 2,
+      "<|assistant_start|>": 3,
+      "<|assistant_end|>": 4,
+      "<|python_start|>": 5,
+      "<|python_end|>": 6,
+      "<|output_start|>": 7,
+      "<|output_end|>": 8,
+      "!": 9,
+      "\"": 10,
+      "#": 11,
+      "$": 12,
+      "%": 13,
+      "&": 14,
+      "'": 15,
+      "(": 16,
+      ")": 17,
+      "*": 18,
+      "+": 19,
+      ",": 20,
+      "-": 21,
+      ".": 22,
+      "/": 23,
+      "0": 24,
+      "1": 25,
+      "2": 26,
+      "3": 27,
+      "4": 28,
+      "5": 29,
+      "6": 30,
+      "7": 31,
+      "8": 32,
+      "9": 33,
+      ":": 34,
+      ";": 35,
+      "<": 36,
+      "=": 37,
+      ">": 38,
+      "?": 39,
+      "@": 40,
+      "A": 41,
+      "B": 42,
+      "C": 43,
+      "D": 44,
+      "E": 45,
+      "F": 46,
+      "G": 47,
+      "H": 48,
+      "I": 49,
+      "J": 50,
+      "K": 51,
+      "L": 52,
+      "M": 53,
+      "N": 54,
+      "O": 55,
+      "P": 56,
+      "Q": 57,
+      "R": 58,
+      "S": 59,
+      "T": 60,
+      "U": 61,
+      "V": 62,
+      "W": 63,
+      "X": 64,
+      "Y": 65,
+      "Z": 66,
+      "[": 67,
+      "\\": 68,
+      "]": 69,
+      "^": 70,
+      "_": 71,
+      "`": 72,
+      "a": 73,
+      "b": 74,
+      "c": 75,
+      "d": 76,
+      "e": 77,
+      "f": 78,
+      "g": 79,
+      "h": 80,
+      "i": 81,
+      "j": 82,
+      "k": 83,
+      "l": 84,
+      "m": 85,
+      "n": 86,
+      "o": 87,
+      "p": 88,
+      "q": 89,
+      "r": 90,
+      "s": 91,
+      "t": 92,
+      "u": 93,
+      "v": 94,
+      "w": 95,
+      "x": 96,
+      "y": 97,
+      "z": 98,
+      "{": 99,
+      "|": 100,
+      "}": 101,
+      "~": 102,
+      "¡": 103,
+      "¢": 104,
+      "£": 105,
+      "¤": 106,
+      "¥": 107,
+      "¦": 108,
+      "§": 109,
+      "¨": 110,
+      "©": 111,
+      "ª": 112,
+      "«": 113,
+      "¬": 114,
+      "®": 115,
+      "¯": 116,
+      "°": 117,
+      "±": 118,
+      "²": 119,
+      "³": 120,
+      "´": 121,
+      "µ": 122,
+      "¶": 123,
+      "·": 124,
+      "¸": 125,
+      "¹": 126,
+      "º": 127,
+      "»": 128,
+      "¼": 129,
+      "½": 130,
+      "¾": 131,
+      "¿": 132,
+      "À": 133,
+      "Á": 134,
+      "Â": 135,
+      "Ã": 136,
+      "Ä": 137,
+      "Å": 138,
+      "Æ": 139,
+      "Ç": 140,
+      "È": 141,
+      "É": 142,
+      "Ê": 143,
+      "Ë": 144,
+      "Ì": 145,
+      "Í": 146,
+      "Î": 147,
+      "Ï": 148,
+      "Ð": 149,
+      "Ñ": 150,
+      "Ò": 151,
+      "Ó": 152,
+      "Ô": 153,
+      "Õ": 154,
+      "Ö": 155,
+      "×": 156,
+      "Ø": 157,
+      "Ù": 158,
+      "Ú": 159,
+      "Û": 160,
+      "Ü": 161,
+      "Ý": 162,
+      "Þ": 163,
+      "ß": 164,
+      "à": 165,
+      "á": 166,
+      "â": 167,
+      "ã": 168,
+      "ä": 169,
+      "å": 170,
+      "æ": 171,
+      "ç": 172,
+      "è": 173,
+      "é": 174,
+      "ê": 175,
+      "ë": 176,
+      "ì": 177,
+      "í": 178,
+      "î": 179,
+      "ï": 180,
+      "ð": 181,
+      "ñ": 182,
+      "ò": 183,
+      "ó": 184,
+      "ô": 185,
+      "õ": 186,
+      "ö": 187,
+      "÷": 188,
+      "ø": 189,
+      "ù": 190,
+      "ú": 191,
+      "û": 192,
+      "ü": 193,
+      "ý": 194,
+      "þ": 195,
+      "ÿ": 196,
+      "Ā": 197,
+      "ā": 198,
+      "Ă": 199,
+      "ă": 200,
+      "Ą": 201,
+      "ą": 202,
+      "Ć": 203,
+      "ć": 204,
+      "Ĉ": 205,
+      "ĉ": 206,
+      "Ċ": 207,
+      "ċ": 208,
+      "Č": 209,
+      "č": 210,
+      "Ď": 211,
+      "ď": 212,
+      "Đ": 213,
+      "đ": 214,
+      "Ē": 215,
+      "ē": 216,
+      "Ĕ": 217,
+      "ĕ": 218,
+      "Ė": 219,
+      "ė": 220,
+      "Ę": 221,
+      "ę": 222,
+      "Ě": 223,
+      "ě": 224,
+      "Ĝ": 225,
+      "ĝ": 226,
+      "Ğ": 227,
+      "ğ": 228,
+      "Ġ": 229,
+      "ġ": 230,
+      "Ģ": 231,
+      "ģ": 232,
+      "Ĥ": 233,
+      "ĥ": 234,
+      "Ħ": 235,
+      "ħ": 236,
+      "Ĩ": 237,
+      "ĩ": 238,
+      "Ī": 239,
+      "ī": 240,
+      "Ĭ": 241,
+      "ĭ": 242,
+      "Į": 243,
+      "į": 244,
+      "İ": 245,
+      "ı": 246,
+      "IJ": 247,
+      "ij": 248,
+      "Ĵ": 249,
+      "ĵ": 250,
+      "Ķ": 251,
+      "ķ": 252,
+      "ĸ": 253,
+      "Ĺ": 254,
+      "ĺ": 255,
+      "Ļ": 256,
+      "ļ": 257,
+      "Ľ": 258,
+      "ľ": 259,
+      "Ŀ": 260,
+      "ŀ": 261,
+      "Ł": 262,
+      "ł": 263,
+      "Ń": 264,
+      "he": 265,
+      "Ġt": 266,
+      "Ġa": 267,
+      "Ġs": 268,
+      "Ġw": 269,
+      "Ġthe": 270,
+      "nd": 271,
+      "ed": 272,
+      "in": 273,
+      "Ġand": 274,
+      "Ġwa": 275,
+      "Ġb": 276,
+      "Ġto": 277,
+      "re": 278,
+      "Ġh": 279,
+      "ou": 280,
+      "it": 281,
+      "Ġf": 282,
+      "er": 283,
+      "ĊĊ": 284,
+      "Ġwas": 285,
+      "Ġl": 286,
+      "Ġc": 287,
+      "Ġhe": 288,
+      "Ġp": 289,
+      "ing": 290,
+      "Ġd": 291,
+      "Ġm": 292,
+      "Ġo": 293,
+      "Ġg": 294,
+      "ar": 295,
+      "is": 296,
+      "id": 297,
+      "ay": 298,
+      "om": 299,
+      "at": 300,
+      "ll": 301,
+      "en": 302,
+      "Ġsa": 303,
+      "ne": 304,
+      "The": 305,
+      ".ĊĊ": 306,
+      "le": 307,
+      "Ġth": 308,
+      "im": 309,
+      "an": 310,
+      "Ġha": 311,
+      "or": 312,
+      "Ġit": 313,
+      "et": 314,
+      "ver": 315,
+      "ld": 316,
+      "Ġin": 317,
+      "ĠS": 318,
+      "on": 319,
+      "Ġe": 320,
+      "ce": 321,
+      "Ġbe": 322,
+      "Ġher": 323,
+      "ir": 324,
+      "Ġ\"": 325,
+      "ĠH": 326,
+      "Ġu": 327,
+      "Ġsaid": 328,
+      "Ġn": 329,
+      "ck": 330,
+      "ow": 331,
+      "ri": 332,
+      "ĠThe": 333,
+      "Ġshe": 334,
+      "Ġso": 335,
+      "st": 336,
+      "Ġy": 337,
+      "ot": 338,
+      "ĠHe": 339,
+      "Ġof": 340,
+      "il": 341,
+      "Ġst": 342,
+      "ut": 343,
+      "ke": 344,
+      "am": 345,
+      "ked": 346,
+      "oo": 347,
+      "pp": 348,
+      "Ġr": 349,
+      "ĠShe": 350,
+      "very": 351,
+      "ĠI": 352,
+      "ve": 353,
+      "Ġthat": 354,
+      "ig": 355,
+      "ith": 356,
+      "Ġhis": 357,
+      "Ġup": 358,
+      "ĠĊĊ": 359,
+      "Ġday": 360,
+      "Ġwith": 361,
+      "Ġpl": 362,
+      "Ġyou": 363,
+      "itt": 364,
+      "ould": 365,
+      "el": 366,
+      "ted": 367,
+      "ent": 368,
+      "ad": 369,
+      "Ġhad": 370,
+      "ound": 371,
+      "al": 372,
+      "ĠJ": 373,
+      "Ġwe": 374,
+      "her": 375,
+      "ittle": 376,
+      "'s": 377,
+      "Ġsm": 378,
+      "Ġplay": 379,
+      "end": 380,
+      "Ġthey": 381,
+      "ack": 382,
+      "Ġthere": 383,
+      "ime": 384,
+      "ly": 385,
+      "Ġsh": 386,
+      "Ġlittle": 387,
+      "Ġre": 388,
+      "Ġne": 389,
+      "Ġtime": 390,
+      "out": 391,
+      "Ġfor": 392,
+      "un": 393,
+      "ch": 394,
+      "se": 395,
+      "Ġhapp": 396,
+      "Ġwh": 397,
+      "my": 398,
+      "ome": 399,
+      "ht": 400,
+      "um": 401,
+      "Ġfri": 402,
+      "Ġas": 403,
+      "Ġfriend": 404,
+      "Ġvery": 405,
+      "all": 406,
+      "ter": 407,
+      "â": 408,
+      "âĤ": 409,
+      "âĤ¬": 410,
+      "On": 411,
+      "Ġk": 412,
+      "ved": 413,
+      "ĠT": 414,
+      "Ġon": 415,
+      "irl": 416,
+      "Once": 417,
+      "ug": 418,
+      "\"ĊĊ": 419,
+      "ill": 420,
+      "Ġgirl": 421,
+      "Ġan": 422,
+      "es": 423,
+      "Ġex": 424,
+      "'t": 425,
+      "ec": 426,
+      "Ġbut": 427,
+      "Ġloo": 428,
+      "Ġli": 429,
+      "Ġbo": 430,
+      "Ġwere": 431,
+      "One": 432,
+      "Ġwan": 433,
+      "Ġhappy": 434,
+      "ake": 435,
+      "ore": 436,
+      "Ġbig": 437,
+      "fu": 438,
+      "Ġsp": 439,
+      "ide": 440,
+      "Ġsaw": 441,
+      "ĠB": 442,
+      "hing": 443,
+      "Ġupon": 444,
+      "ard": 445,
+      "Ġcould": 446,
+      "ic": 447,
+      "Ġout": 448,
+      "iled": 449,
+      "one": 450,
+      "round": 451,
+      "ra": 452,
+      "ry": 453,
+      "Ġsmiled": 454,
+      "Ġhim": 455,
+      "ĠA": 456,
+      "Ġmom": 457,
+      "hen": 458,
+      "way": 459,
+      "ur": 460,
+      "ĠIt": 461,
+      "ful": 462,
+      "ain": 463,
+      "Ġwent": 464,
+      "Ġhel": 465,
+      "Ġnot": 466,
+      "ĠThey": 467,
+      "Ġwanted": 468,
+      "ind": 469,
+      "are": 470,
+      "ear": 471,
+      "ĠM": 472,
+      "Ġall": 473,
+      "Ġfriends": 474,
+      "Ġtoo": 475,
+      "Ġgo": 476,
+      "ily": 477,
+      "ame": 478,
+      "ĠTim": 479,
+      "Ġhelp": 480,
+      "omet": 481,
+      "ĠL": 482,
+      "Ġlo": 483,
+      "ght": 484,
+      "Ġsomet": 485,
+      "Ġat": 486,
+      "Ġdo": 487,
+      "Ġasked": 488,
+      "!\"": 489,
+      "Ġaround": 490,
+      "Ġj": 491,
+      "ree": 492,
+      "Ġlooked": 493,
+      "Ġsomething": 494,
+      "Ġse": 495,
+      "Ġwor": 496,
+      "dd": 497,
+      "hed": 498,
+      "ood": 499,
+      "Ġcl": 500,
+      "amed": 501,
+      "ro": 502,
+      "Ġcan": 503,
+      "ark": 504,
+      "king": 505,
+      "ĠE": 506,
+      "rom": 507,
+      "Ġback": 508,
+      "Ġexc": 509,
+      "ab": 510,
+      "ick": 511
+    },
+    "merges": [
+      [
+        "h",
+        "e"
+      ],
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "n",
+        "d"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "Ġa",
+        "nd"
+      ],
+      [
+        "Ġw",
+        "a"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "Ċ",
+        "Ċ"
+      ],
+      [
+        "Ġwa",
+        "s"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "Ġ",
+        "g"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "a",
+        "y"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "l",
+        "l"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġs",
+        "a"
+      ],
+      [
+        "n",
+        "e"
+      ],
+      [
+        "T",
+        "he"
+      ],
+      [
+        ".",
+        "ĊĊ"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "Ġh",
+        "a"
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "Ġ",
+        "S"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġ",
+        "e"
+      ],
+      [
+        "c",
+        "e"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "Ġhe",
+        "r"
+      ],
+      [
+        "i",
+        "r"
+      ],
+      [
+        "Ġ",
+        "\""
+      ],
+      [
+        "Ġ",
+        "H"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "Ġsa",
+        "id"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "c",
+        "k"
+      ],
+      [
+        "o",
+        "w"
+      ],
+      [
+        "r",
+        "i"
+      ],
+      [
+        "Ġ",
+        "The"
+      ],
+      [
+        "Ġs",
+        "he"
+      ],
+      [
+        "Ġs",
+        "o"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "Ġ",
+        "y"
+      ],
+      [
+        "o",
+        "t"
+      ],
+      [
+        "ĠH",
+        "e"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "Ġs",
+        "t"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "k",
+        "e"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "k",
+        "ed"
+      ],
+      [
+        "o",
+        "o"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "ĠS",
+        "he"
+      ],
+      [
+        "ver",
+        "y"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "i",
+        "g"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "Ġh",
+        "is"
+      ],
+      [
+        "Ġu",
+        "p"
+      ],
+      [
+        "Ġ",
+        "ĊĊ"
+      ],
+      [
+        "Ġd",
+        "ay"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "Ġp",
+        "l"
+      ],
+      [
+        "Ġy",
+        "ou"
+      ],
+      [
+        "it",
+        "t"
+      ],
+      [
+        "ou",
+        "ld"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "t",
+        "ed"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "Ġha",
+        "d"
+      ],
+      [
+        "ou",
+        "nd"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "Ġ",
+        "J"
+      ],
+      [
+        "Ġw",
+        "e"
+      ],
+      [
+        "he",
+        "r"
+      ],
+      [
+        "itt",
+        "le"
+      ],
+      [
+        "'",
+        "s"
+      ],
+      [
+        "Ġs",
+        "m"
+      ],
+      [
+        "Ġpl",
+        "ay"
+      ],
+      [
+        "e",
+        "nd"
+      ],
+      [
+        "Ġthe",
+        "y"
+      ],
+      [
+        "a",
+        "ck"
+      ],
+      [
+        "Ġthe",
+        "re"
+      ],
+      [
+        "im",
+        "e"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "Ġs",
+        "h"
+      ],
+      [
+        "Ġl",
+        "ittle"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġ",
+        "ne"
+      ],
+      [
+        "Ġt",
+        "ime"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "u",
+        "n"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "Ġha",
+        "pp"
+      ],
+      [
+        "Ġw",
+        "h"
+      ],
+      [
+        "m",
+        "y"
+      ],
+      [
+        "om",
+        "e"
+      ],
+      [
+        "h",
+        "t"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "Ġf",
+        "ri"
+      ],
+      [
+        "Ġa",
+        "s"
+      ],
+      [
+        "Ġfri",
+        "end"
+      ],
+      [
+        "Ġ",
+        "very"
+      ],
+      [
+        "a",
+        "ll"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ã",
+        "¢"
+      ],
+      [
+        "â",
+        "Ĥ"
+      ],
+      [
+        "âĤ",
+        "¬"
+      ],
+      [
+        "O",
+        "n"
+      ],
+      [
+        "Ġ",
+        "k"
+      ],
+      [
+        "v",
+        "ed"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "Ġo",
+        "n"
+      ],
+      [
+        "ir",
+        "l"
+      ],
+      [
+        "On",
+        "ce"
+      ],
+      [
+        "u",
+        "g"
+      ],
+      [
+        "\"",
+        "ĊĊ"
+      ],
+      [
+        "i",
+        "ll"
+      ],
+      [
+        "Ġg",
+        "irl"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "Ġe",
+        "x"
+      ],
+      [
+        "'",
+        "t"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "Ġb",
+        "ut"
+      ],
+      [
+        "Ġl",
+        "oo"
+      ],
+      [
+        "Ġl",
+        "i"
+      ],
+      [
+        "Ġb",
+        "o"
+      ],
+      [
+        "Ġwe",
+        "re"
+      ],
+      [
+        "O",
+        "ne"
+      ],
+      [
+        "Ġwa",
+        "n"
+      ],
+      [
+        "Ġhapp",
+        "y"
+      ],
+      [
+        "a",
+        "ke"
+      ],
+      [
+        "o",
+        "re"
+      ],
+      [
+        "Ġb",
+        "ig"
+      ],
+      [
+        "f",
+        "u"
+      ],
+      [
+        "Ġs",
+        "p"
+      ],
+      [
+        "id",
+        "e"
+      ],
+      [
+        "Ġsa",
+        "w"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "h",
+        "ing"
+      ],
+      [
+        "Ġup",
+        "on"
+      ],
+      [
+        "ar",
+        "d"
+      ],
+      [
+        "Ġc",
+        "ould"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "Ġ",
+        "out"
+      ],
+      [
+        "il",
+        "ed"
+      ],
+      [
+        "o",
+        "ne"
+      ],
+      [
+        "r",
+        "ound"
+      ],
+      [
+        "r",
+        "a"
+      ],
+      [
+        "r",
+        "y"
+      ],
+      [
+        "Ġsm",
+        "iled"
+      ],
+      [
+        "Ġh",
+        "im"
+      ],
+      [
+        "Ġ",
+        "A"
+      ],
+      [
+        "Ġm",
+        "om"
+      ],
+      [
+        "he",
+        "n"
+      ],
+      [
+        "w",
+        "ay"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "ĠI",
+        "t"
+      ],
+      [
+        "fu",
+        "l"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "Ġw",
+        "ent"
+      ],
+      [
+        "Ġhe",
+        "l"
+      ],
+      [
+        "Ġn",
+        "ot"
+      ],
+      [
+        "ĠThe",
+        "y"
+      ],
+      [
+        "Ġwan",
+        "ted"
+      ],
+      [
+        "i",
+        "nd"
+      ],
+      [
+        "a",
+        "re"
+      ],
+      [
+        "e",
+        "ar"
+      ],
+      [
+        "Ġ",
+        "M"
+      ],
+      [
+        "Ġa",
+        "ll"
+      ],
+      [
+        "Ġfriend",
+        "s"
+      ],
+      [
+        "Ġto",
+        "o"
+      ],
+      [
+        "Ġg",
+        "o"
+      ],
+      [
+        "il",
+        "y"
+      ],
+      [
+        "am",
+        "e"
+      ],
+      [
+        "ĠT",
+        "im"
+      ],
+      [
+        "Ġhel",
+        "p"
+      ],
+      [
+        "om",
+        "et"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "Ġl",
+        "o"
+      ],
+      [
+        "g",
+        "ht"
+      ],
+      [
+        "Ġs",
+        "omet"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "Ġd",
+        "o"
+      ],
+      [
+        "Ġas",
+        "ked"
+      ],
+      [
+        "!",
+        "\""
+      ],
+      [
+        "Ġa",
+        "round"
+      ],
+      [
+        "Ġ",
+        "j"
+      ],
+      [
+        "re",
+        "e"
+      ],
+      [
+        "Ġloo",
+        "ked"
+      ],
+      [
+        "Ġsomet",
+        "hing"
+      ],
+      [
+        "Ġs",
+        "e"
+      ],
+      [
+        "Ġw",
+        "or"
+      ],
+      [
+        "d",
+        "d"
+      ],
+      [
+        "he",
+        "d"
+      ],
+      [
+        "oo",
+        "d"
+      ],
+      [
+        "Ġc",
+        "l"
+      ],
+      [
+        "am",
+        "ed"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "ar",
+        "k"
+      ],
+      [
+        "k",
+        "ing"
+      ],
+      [
+        "Ġ",
+        "E"
+      ],
+      [
+        "r",
+        "om"
+      ],
+      [
+        "Ġb",
+        "ack"
+      ],
+      [
+        "Ġex",
+        "c"
+      ],
+      [
+        "a",
+        "b"
+      ],
+      [
+        "i",
+        "ck"
+      ]
+    ]
+  }
+}

archive/runs_legacy_20260525/smoke/20260522-132336/config.json

@@ -0,0 +1,58 @@
+{
+  "args": {
+    "corpus": "/Users/mandeepsidhu/Desktop/code/transformer/data/tinystories_train.txt",
+    "corpus_glob": null,
+    "text_column": "text",
+    "output_dir": "runs/smoke",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 20000,
+    "stream_token_caps": [
+      20000,
+      40000,
+      80000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": true,
+    "force_retokenize": false,
+    "vocab_size": 512,
+    "tokenizer_train_chars": 200000,
+    "block_size": 32,
+    "batch_size": 4,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 1,
+    "stream_steps_per_stage": 1,
+    "eval_batches": 1,
+    "log_every": 0,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/smoke/20260522-132336/tokenizer.json",
+  "encoded_path": "runs/smoke/20260522-132336/tokens-v512-uint16.npy",
+  "train_tokens": 777983,
+  "val_tokens": 86442,
+  "effective_initial_tokens": 20000,
+  "effective_stream_token_caps": [
+    20000,
+    40000,
+    80000
+  ]
+}

archive/runs_legacy_20260525/smoke/20260522-132336/metrics.jsonl

@@ -0,0 +1,30 @@
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 1, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.238768100738525, "eval_loss": 6.236357688903809, "elapsed_sec": 0.15027999877929688, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 1, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.240777015686035, "eval_loss": 6.2333245277404785, "elapsed_sec": 0.13346290588378906, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 1, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.2400970458984375, "eval_loss": 6.237194538116455, "elapsed_sec": 0.04652595520019531, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.240694999694824, "eval_loss": 6.236194610595703, "elapsed_sec": 0.04595613479614258, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.233834266662598, "eval_loss": 6.2317962646484375, "elapsed_sec": 0.03716301918029785, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 1, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231166839599609, "eval_loss": 6.221035957336426, "elapsed_sec": 0.032035112380981445, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 2, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.237826824188232, "eval_loss": 6.234821796417236, "elapsed_sec": 0.0224301815032959, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 2, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.238691329956055, "eval_loss": 6.229793548583984, "elapsed_sec": 0.042351722717285156, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 2, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.236819267272949, "eval_loss": 6.234230995178223, "elapsed_sec": 0.04147815704345703, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.23726749420166, "eval_loss": 6.232943534851074, "elapsed_sec": 0.04183006286621094, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.2372612953186035, "eval_loss": 6.226807594299316, "elapsed_sec": 0.032605886459350586, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 2, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.228798866271973, "eval_loss": 6.214606285095215, "elapsed_sec": 0.03294706344604492, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 3, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.237939834594727, "eval_loss": 6.235240459442139, "elapsed_sec": 0.02249908447265625, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 3, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.237975120544434, "eval_loss": 6.229854583740234, "elapsed_sec": 0.04176592826843262, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 3, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.238975524902344, "eval_loss": 6.2339982986450195, "elapsed_sec": 0.04214000701904297, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.238128662109375, "eval_loss": 6.2329301834106445, "elapsed_sec": 0.04157400131225586, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.236502170562744, "eval_loss": 6.225698471069336, "elapsed_sec": 0.03307604789733887, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 3, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231315612792969, "eval_loss": 6.213197708129883, "elapsed_sec": 0.033010244369506836, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 4, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.239793300628662, "eval_loss": 6.237911224365234, "elapsed_sec": 0.02308821678161621, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 4, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.239620208740234, "eval_loss": 6.232203960418701, "elapsed_sec": 0.043370962142944336, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 4, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.239019393920898, "eval_loss": 6.237016677856445, "elapsed_sec": 0.04206585884094238, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.239429950714111, "eval_loss": 6.235345363616943, "elapsed_sec": 0.04175519943237305, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.23697566986084, "eval_loss": 6.229433536529541, "elapsed_sec": 0.03307700157165527, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 4, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.231019020080566, "eval_loss": 6.218218803405762, "elapsed_sec": 0.03255319595336914, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase1_baseline_comparison", "condition": "small_static", "seed": 5, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.239540100097656, "eval_loss": 6.2350263595581055, "elapsed_sec": 0.02461099624633789, "parameters": 917504, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 4, "n_head": 4, "n_embd": 128, "dropout": 0.1}}
+{"phase": "phase1_baseline_comparison", "condition": "large_static", "seed": 5, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.1, "train_loss_last": 6.23555850982666, "eval_loss": 6.231681823730469, "elapsed_sec": 0.04686999320983887, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.1}}
+{"phase": "phase2_fixed_data_dropout_optimization", "condition": "large_dropout_decay_fixed_data", "seed": 5, "stage": null, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.8, "train_loss_last": 6.239316463470459, "eval_loss": 6.237186431884766, "elapsed_sec": 0.04636812210083008, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 0, "token_limit": 20000, "steps": 1, "tokens_seen": 128, "dropout": 0.6974873734152917, "train_loss_last": 6.239423751831055, "eval_loss": 6.236317157745361, "elapsed_sec": 0.045987844467163086, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 1, "token_limit": 40000, "steps": 1, "tokens_seen": 256, "dropout": 0.45000000000000007, "train_loss_last": 6.234818458557129, "eval_loss": 6.230718612670898, "elapsed_sec": 0.034147024154663086, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}
+{"phase": "phase3_simulated_streaming_scale_up", "condition": "large_dropout_decay_streaming", "seed": 5, "stage": 2, "token_limit": 80000, "steps": 1, "tokens_seen": 384, "dropout": 0.1, "train_loss_last": 6.232948303222656, "eval_loss": 6.2211151123046875, "elapsed_sec": 0.03441500663757324, "parameters": 6553600, "model_config": {"block_size": 32, "vocab_size": 512, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.8}}

archive/runs_legacy_20260525/smoke/20260522-132336/summary.csv

@@ -0,0 +1,7 @@
+phase,condition,stage,token_limit,parameters,n,mean_eval_loss,std_eval_loss
+phase1_baseline_comparison,large_static,,20000,6553600,5,6.231371688842773,0.0015325284625100665
+phase1_baseline_comparison,small_static,,20000,917504,5,6.235871505737305,0.0012856134041782643
+phase2_fixed_data_dropout_optimization,large_dropout_decay_fixed_data,,20000,6553600,5,6.235925388336182,0.0016565421564777797
+phase3_simulated_streaming_scale_up,large_dropout_decay_streaming,0,20000,6553600,5,6.2347461700439455,0.0016935404053936482
+phase3_simulated_streaming_scale_up,large_dropout_decay_streaming,1,40000,6553600,5,6.228890895843506,0.0025791421476287733
+phase3_simulated_streaming_scale_up,large_dropout_decay_streaming,2,80000,6553600,5,6.217634773254394,0.0036359727629622267

archive/runs_legacy_20260525/smoke/20260522-132336/summary.json

@@ -0,0 +1,62 @@
+[
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "large_static",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.231371688842773,
+    "std_eval_loss": 0.0015325284625100665
+  },
+  {
+    "phase": "phase1_baseline_comparison",
+    "condition": "small_static",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 917504,
+    "n": 5,
+    "mean_eval_loss": 6.235871505737305,
+    "std_eval_loss": 0.0012856134041782643
+  },
+  {
+    "phase": "phase2_fixed_data_dropout_optimization",
+    "condition": "large_dropout_decay_fixed_data",
+    "stage": null,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.235925388336182,
+    "std_eval_loss": 0.0016565421564777797
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 0,
+    "token_limit": 20000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.2347461700439455,
+    "std_eval_loss": 0.0016935404053936482
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 1,
+    "token_limit": 40000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.228890895843506,
+    "std_eval_loss": 0.0025791421476287733
+  },
+  {
+    "phase": "phase3_simulated_streaming_scale_up",
+    "condition": "large_dropout_decay_streaming",
+    "stage": 2,
+    "token_limit": 80000,
+    "parameters": 6553600,
+    "n": 5,
+    "mean_eval_loss": 6.217634773254394,
+    "std_eval_loss": 0.0036359727629622267
+  }
+]

archive/runs_legacy_20260525/smoke/20260522-132336/tokenizer.json

@@ -0,0 +1,1629 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<|bos|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<|user_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<|user_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<|assistant_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<|assistant_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "<|python_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 6,
+      "content": "<|python_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 7,
+      "content": "<|output_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 8,
+      "content": "<|output_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Sequence",
+    "pretokenizers": [
+      {
+        "type": "Split",
+        "pattern": {
+          "Regex": "'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?+\\p{L}+|\\p{N}{1,2}| ?[^\\s\\p{L}\\p{N}]++[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+"
+        },
+        "behavior": "Isolated",
+        "invert": false
+      },
+      {
+        "type": "ByteLevel",
+        "add_prefix_space": false,
+        "trim_offsets": true,
+        "use_regex": false
+      }
+    ]
+  },
+  "post_processor": null,
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": true,
+    "ignore_merges": false,
+    "vocab": {
+      "<|bos|>": 0,
+      "<|user_start|>": 1,
+      "<|user_end|>": 2,
+      "<|assistant_start|>": 3,
+      "<|assistant_end|>": 4,
+      "<|python_start|>": 5,
+      "<|python_end|>": 6,
+      "<|output_start|>": 7,
+      "<|output_end|>": 8,
+      "!": 9,
+      "\"": 10,
+      "#": 11,
+      "$": 12,
+      "%": 13,
+      "&": 14,
+      "'": 15,
+      "(": 16,
+      ")": 17,
+      "*": 18,
+      "+": 19,
+      ",": 20,
+      "-": 21,
+      ".": 22,
+      "/": 23,
+      "0": 24,
+      "1": 25,
+      "2": 26,
+      "3": 27,
+      "4": 28,
+      "5": 29,
+      "6": 30,
+      "7": 31,
+      "8": 32,
+      "9": 33,
+      ":": 34,
+      ";": 35,
+      "<": 36,
+      "=": 37,
+      ">": 38,
+      "?": 39,
+      "@": 40,
+      "A": 41,
+      "B": 42,
+      "C": 43,
+      "D": 44,
+      "E": 45,
+      "F": 46,
+      "G": 47,
+      "H": 48,
+      "I": 49,
+      "J": 50,
+      "K": 51,
+      "L": 52,
+      "M": 53,
+      "N": 54,
+      "O": 55,
+      "P": 56,
+      "Q": 57,
+      "R": 58,
+      "S": 59,
+      "T": 60,
+      "U": 61,
+      "V": 62,
+      "W": 63,
+      "X": 64,
+      "Y": 65,
+      "Z": 66,
+      "[": 67,
+      "\\": 68,
+      "]": 69,
+      "^": 70,
+      "_": 71,
+      "`": 72,
+      "a": 73,
+      "b": 74,
+      "c": 75,
+      "d": 76,
+      "e": 77,
+      "f": 78,
+      "g": 79,
+      "h": 80,
+      "i": 81,
+      "j": 82,
+      "k": 83,
+      "l": 84,
+      "m": 85,
+      "n": 86,
+      "o": 87,
+      "p": 88,
+      "q": 89,
+      "r": 90,
+      "s": 91,
+      "t": 92,
+      "u": 93,
+      "v": 94,
+      "w": 95,
+      "x": 96,
+      "y": 97,
+      "z": 98,
+      "{": 99,
+      "|": 100,
+      "}": 101,
+      "~": 102,
+      "¡": 103,
+      "¢": 104,
+      "£": 105,
+      "¤": 106,
+      "¥": 107,
+      "¦": 108,
+      "§": 109,
+      "¨": 110,
+      "©": 111,
+      "ª": 112,
+      "«": 113,
+      "¬": 114,
+      "®": 115,
+      "¯": 116,
+      "°": 117,
+      "±": 118,
+      "²": 119,
+      "³": 120,
+      "´": 121,
+      "µ": 122,
+      "¶": 123,
+      "·": 124,
+      "¸": 125,
+      "¹": 126,
+      "º": 127,
+      "»": 128,
+      "¼": 129,
+      "½": 130,
+      "¾": 131,
+      "¿": 132,
+      "À": 133,
+      "Á": 134,
+      "Â": 135,
+      "Ã": 136,
+      "Ä": 137,
+      "Å": 138,
+      "Æ": 139,
+      "Ç": 140,
+      "È": 141,
+      "É": 142,
+      "Ê": 143,
+      "Ë": 144,
+      "Ì": 145,
+      "Í": 146,
+      "Î": 147,
+      "Ï": 148,
+      "Ð": 149,
+      "Ñ": 150,
+      "Ò": 151,
+      "Ó": 152,
+      "Ô": 153,
+      "Õ": 154,
+      "Ö": 155,
+      "×": 156,
+      "Ø": 157,
+      "Ù": 158,
+      "Ú": 159,
+      "Û": 160,
+      "Ü": 161,
+      "Ý": 162,
+      "Þ": 163,
+      "ß": 164,
+      "à": 165,
+      "á": 166,
+      "â": 167,
+      "ã": 168,
+      "ä": 169,
+      "å": 170,
+      "æ": 171,
+      "ç": 172,
+      "è": 173,
+      "é": 174,
+      "ê": 175,
+      "ë": 176,
+      "ì": 177,
+      "í": 178,
+      "î": 179,
+      "ï": 180,
+      "ð": 181,
+      "ñ": 182,
+      "ò": 183,
+      "ó": 184,
+      "ô": 185,
+      "õ": 186,
+      "ö": 187,
+      "÷": 188,
+      "ø": 189,
+      "ù": 190,
+      "ú": 191,
+      "û": 192,
+      "ü": 193,
+      "ý": 194,
+      "þ": 195,
+      "ÿ": 196,
+      "Ā": 197,
+      "ā": 198,
+      "Ă": 199,
+      "ă": 200,
+      "Ą": 201,
+      "ą": 202,
+      "Ć": 203,
+      "ć": 204,
+      "Ĉ": 205,
+      "ĉ": 206,
+      "Ċ": 207,
+      "ċ": 208,
+      "Č": 209,
+      "č": 210,
+      "Ď": 211,
+      "ď": 212,
+      "Đ": 213,
+      "đ": 214,
+      "Ē": 215,
+      "ē": 216,
+      "Ĕ": 217,
+      "ĕ": 218,
+      "Ė": 219,
+      "ė": 220,
+      "Ę": 221,
+      "ę": 222,
+      "Ě": 223,
+      "ě": 224,
+      "Ĝ": 225,
+      "ĝ": 226,
+      "Ğ": 227,
+      "ğ": 228,
+      "Ġ": 229,
+      "ġ": 230,
+      "Ģ": 231,
+      "ģ": 232,
+      "Ĥ": 233,
+      "ĥ": 234,
+      "Ħ": 235,
+      "ħ": 236,
+      "Ĩ": 237,
+      "ĩ": 238,
+      "Ī": 239,
+      "ī": 240,
+      "Ĭ": 241,
+      "ĭ": 242,
+      "Į": 243,
+      "į": 244,
+      "İ": 245,
+      "ı": 246,
+      "IJ": 247,
+      "ij": 248,
+      "Ĵ": 249,
+      "ĵ": 250,
+      "Ķ": 251,
+      "ķ": 252,
+      "ĸ": 253,
+      "Ĺ": 254,
+      "ĺ": 255,
+      "Ļ": 256,
+      "ļ": 257,
+      "Ľ": 258,
+      "ľ": 259,
+      "Ŀ": 260,
+      "ŀ": 261,
+      "Ł": 262,
+      "ł": 263,
+      "Ń": 264,
+      "he": 265,
+      "Ġt": 266,
+      "Ġa": 267,
+      "Ġs": 268,
+      "Ġw": 269,
+      "Ġthe": 270,
+      "nd": 271,
+      "ed": 272,
+      "in": 273,
+      "Ġand": 274,
+      "Ġwa": 275,
+      "Ġb": 276,
+      "Ġto": 277,
+      "re": 278,
+      "Ġh": 279,
+      "ou": 280,
+      "it": 281,
+      "Ġf": 282,
+      "er": 283,
+      "ĊĊ": 284,
+      "Ġwas": 285,
+      "Ġl": 286,
+      "Ġc": 287,
+      "Ġhe": 288,
+      "Ġp": 289,
+      "ing": 290,
+      "Ġd": 291,
+      "Ġm": 292,
+      "Ġo": 293,
+      "Ġg": 294,
+      "ar": 295,
+      "is": 296,
+      "id": 297,
+      "ay": 298,
+      "om": 299,
+      "at": 300,
+      "ll": 301,
+      "en": 302,
+      "Ġsa": 303,
+      "ne": 304,
+      "The": 305,
+      ".ĊĊ": 306,
+      "le": 307,
+      "Ġth": 308,
+      "im": 309,
+      "an": 310,
+      "Ġha": 311,
+      "or": 312,
+      "Ġit": 313,
+      "et": 314,
+      "ver": 315,
+      "ld": 316,
+      "Ġin": 317,
+      "ĠS": 318,
+      "on": 319,
+      "Ġe": 320,
+      "ce": 321,
+      "Ġbe": 322,
+      "Ġher": 323,
+      "ir": 324,
+      "Ġ\"": 325,
+      "ĠH": 326,
+      "Ġu": 327,
+      "Ġsaid": 328,
+      "Ġn": 329,
+      "ck": 330,
+      "ow": 331,
+      "ri": 332,
+      "ĠThe": 333,
+      "Ġshe": 334,
+      "Ġso": 335,
+      "st": 336,
+      "Ġy": 337,
+      "ot": 338,
+      "ĠHe": 339,
+      "Ġof": 340,
+      "il": 341,
+      "Ġst": 342,
+      "ut": 343,
+      "ke": 344,
+      "am": 345,
+      "ked": 346,
+      "oo": 347,
+      "pp": 348,
+      "Ġr": 349,
+      "ĠShe": 350,
+      "very": 351,
+      "ĠI": 352,
+      "ve": 353,
+      "Ġthat": 354,
+      "ig": 355,
+      "ith": 356,
+      "Ġhis": 357,
+      "Ġup": 358,
+      "ĠĊĊ": 359,
+      "Ġday": 360,
+      "Ġwith": 361,
+      "Ġpl": 362,
+      "Ġyou": 363,
+      "itt": 364,
+      "ould": 365,
+      "el": 366,
+      "ted": 367,
+      "ent": 368,
+      "ad": 369,
+      "Ġhad": 370,
+      "ound": 371,
+      "al": 372,
+      "ĠJ": 373,
+      "Ġwe": 374,
+      "her": 375,
+      "ittle": 376,
+      "'s": 377,
+      "Ġsm": 378,
+      "Ġplay": 379,
+      "end": 380,
+      "Ġthey": 381,
+      "ack": 382,
+      "Ġthere": 383,
+      "ime": 384,
+      "ly": 385,
+      "Ġsh": 386,
+      "Ġlittle": 387,
+      "Ġre": 388,
+      "Ġne": 389,
+      "Ġtime": 390,
+      "out": 391,
+      "Ġfor": 392,
+      "un": 393,
+      "ch": 394,
+      "se": 395,
+      "Ġhapp": 396,
+      "Ġwh": 397,
+      "my": 398,
+      "ome": 399,
+      "ht": 400,
+      "um": 401,
+      "Ġfri": 402,
+      "Ġas": 403,
+      "Ġfriend": 404,
+      "Ġvery": 405,
+      "all": 406,
+      "ter": 407,
+      "â": 408,
+      "âĤ": 409,
+      "âĤ¬": 410,
+      "On": 411,
+      "Ġk": 412,
+      "ved": 413,
+      "ĠT": 414,
+      "Ġon": 415,
+      "irl": 416,
+      "Once": 417,
+      "ug": 418,
+      "\"ĊĊ": 419,
+      "ill": 420,
+      "Ġgirl": 421,
+      "Ġan": 422,
+      "es": 423,
+      "Ġex": 424,
+      "'t": 425,
+      "ec": 426,
+      "Ġbut": 427,
+      "Ġloo": 428,
+      "Ġli": 429,
+      "Ġbo": 430,
+      "Ġwere": 431,
+      "One": 432,
+      "Ġwan": 433,
+      "Ġhappy": 434,
+      "ake": 435,
+      "ore": 436,
+      "Ġbig": 437,
+      "fu": 438,
+      "Ġsp": 439,
+      "ide": 440,
+      "Ġsaw": 441,
+      "ĠB": 442,
+      "hing": 443,
+      "Ġupon": 444,
+      "ard": 445,
+      "Ġcould": 446,
+      "ic": 447,
+      "Ġout": 448,
+      "iled": 449,
+      "one": 450,
+      "round": 451,
+      "ra": 452,
+      "ry": 453,
+      "Ġsmiled": 454,
+      "Ġhim": 455,
+      "ĠA": 456,
+      "Ġmom": 457,
+      "hen": 458,
+      "way": 459,
+      "ur": 460,
+      "ĠIt": 461,
+      "ful": 462,
+      "ain": 463,
+      "Ġwent": 464,
+      "Ġhel": 465,
+      "Ġnot": 466,
+      "ĠThey": 467,
+      "Ġwanted": 468,
+      "ind": 469,
+      "are": 470,
+      "ear": 471,
+      "ĠM": 472,
+      "Ġall": 473,
+      "Ġfriends": 474,
+      "Ġtoo": 475,
+      "Ġgo": 476,
+      "ily": 477,
+      "ame": 478,
+      "ĠTim": 479,
+      "Ġhelp": 480,
+      "omet": 481,
+      "ĠL": 482,
+      "Ġlo": 483,
+      "ght": 484,
+      "Ġsomet": 485,
+      "Ġat": 486,
+      "Ġdo": 487,
+      "Ġasked": 488,
+      "!\"": 489,
+      "Ġaround": 490,
+      "Ġj": 491,
+      "ree": 492,
+      "Ġlooked": 493,
+      "Ġsomething": 494,
+      "Ġse": 495,
+      "Ġwor": 496,
+      "dd": 497,
+      "hed": 498,
+      "ood": 499,
+      "Ġcl": 500,
+      "amed": 501,
+      "ro": 502,
+      "Ġcan": 503,
+      "ark": 504,
+      "king": 505,
+      "ĠE": 506,
+      "rom": 507,
+      "Ġback": 508,
+      "Ġexc": 509,
+      "ab": 510,
+      "ick": 511
+    },
+    "merges": [
+      [
+        "h",
+        "e"
+      ],
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "n",
+        "d"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "Ġa",
+        "nd"
+      ],
+      [
+        "Ġw",
+        "a"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "Ċ",
+        "Ċ"
+      ],
+      [
+        "Ġwa",
+        "s"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "Ġ",
+        "g"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "a",
+        "y"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "l",
+        "l"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġs",
+        "a"
+      ],
+      [
+        "n",
+        "e"
+      ],
+      [
+        "T",
+        "he"
+      ],
+      [
+        ".",
+        "ĊĊ"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "Ġh",
+        "a"
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "Ġ",
+        "S"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġ",
+        "e"
+      ],
+      [
+        "c",
+        "e"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "Ġhe",
+        "r"
+      ],
+      [
+        "i",
+        "r"
+      ],
+      [
+        "Ġ",
+        "\""
+      ],
+      [
+        "Ġ",
+        "H"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "Ġsa",
+        "id"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "c",
+        "k"
+      ],
+      [
+        "o",
+        "w"
+      ],
+      [
+        "r",
+        "i"
+      ],
+      [
+        "Ġ",
+        "The"
+      ],
+      [
+        "Ġs",
+        "he"
+      ],
+      [
+        "Ġs",
+        "o"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "Ġ",
+        "y"
+      ],
+      [
+        "o",
+        "t"
+      ],
+      [
+        "ĠH",
+        "e"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "Ġs",
+        "t"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "k",
+        "e"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "k",
+        "ed"
+      ],
+      [
+        "o",
+        "o"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "ĠS",
+        "he"
+      ],
+      [
+        "ver",
+        "y"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "i",
+        "g"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "Ġh",
+        "is"
+      ],
+      [
+        "Ġu",
+        "p"
+      ],
+      [
+        "Ġ",
+        "ĊĊ"
+      ],
+      [
+        "Ġd",
+        "ay"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "Ġp",
+        "l"
+      ],
+      [
+        "Ġy",
+        "ou"
+      ],
+      [
+        "it",
+        "t"
+      ],
+      [
+        "ou",
+        "ld"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "t",
+        "ed"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "Ġha",
+        "d"
+      ],
+      [
+        "ou",
+        "nd"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "Ġ",
+        "J"
+      ],
+      [
+        "Ġw",
+        "e"
+      ],
+      [
+        "he",
+        "r"
+      ],
+      [
+        "itt",
+        "le"
+      ],
+      [
+        "'",
+        "s"
+      ],
+      [
+        "Ġs",
+        "m"
+      ],
+      [
+        "Ġpl",
+        "ay"
+      ],
+      [
+        "e",
+        "nd"
+      ],
+      [
+        "Ġthe",
+        "y"
+      ],
+      [
+        "a",
+        "ck"
+      ],
+      [
+        "Ġthe",
+        "re"
+      ],
+      [
+        "im",
+        "e"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "Ġs",
+        "h"
+      ],
+      [
+        "Ġl",
+        "ittle"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġ",
+        "ne"
+      ],
+      [
+        "Ġt",
+        "ime"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "u",
+        "n"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "Ġha",
+        "pp"
+      ],
+      [
+        "Ġw",
+        "h"
+      ],
+      [
+        "m",
+        "y"
+      ],
+      [
+        "om",
+        "e"
+      ],
+      [
+        "h",
+        "t"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "Ġf",
+        "ri"
+      ],
+      [
+        "Ġa",
+        "s"
+      ],
+      [
+        "Ġfri",
+        "end"
+      ],
+      [
+        "Ġ",
+        "very"
+      ],
+      [
+        "a",
+        "ll"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ã",
+        "¢"
+      ],
+      [
+        "â",
+        "Ĥ"
+      ],
+      [
+        "âĤ",
+        "¬"
+      ],
+      [
+        "O",
+        "n"
+      ],
+      [
+        "Ġ",
+        "k"
+      ],
+      [
+        "v",
+        "ed"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "Ġo",
+        "n"
+      ],
+      [
+        "ir",
+        "l"
+      ],
+      [
+        "On",
+        "ce"
+      ],
+      [
+        "u",
+        "g"
+      ],
+      [
+        "\"",
+        "ĊĊ"
+      ],
+      [
+        "i",
+        "ll"
+      ],
+      [
+        "Ġg",
+        "irl"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "Ġe",
+        "x"
+      ],
+      [
+        "'",
+        "t"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "Ġb",
+        "ut"
+      ],
+      [
+        "Ġl",
+        "oo"
+      ],
+      [
+        "Ġl",
+        "i"
+      ],
+      [
+        "Ġb",
+        "o"
+      ],
+      [
+        "Ġwe",
+        "re"
+      ],
+      [
+        "O",
+        "ne"
+      ],
+      [
+        "Ġwa",
+        "n"
+      ],
+      [
+        "Ġhapp",
+        "y"
+      ],
+      [
+        "a",
+        "ke"
+      ],
+      [
+        "o",
+        "re"
+      ],
+      [
+        "Ġb",
+        "ig"
+      ],
+      [
+        "f",
+        "u"
+      ],
+      [
+        "Ġs",
+        "p"
+      ],
+      [
+        "id",
+        "e"
+      ],
+      [
+        "Ġsa",
+        "w"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "h",
+        "ing"
+      ],
+      [
+        "Ġup",
+        "on"
+      ],
+      [
+        "ar",
+        "d"
+      ],
+      [
+        "Ġc",
+        "ould"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "Ġ",
+        "out"
+      ],
+      [
+        "il",
+        "ed"
+      ],
+      [
+        "o",
+        "ne"
+      ],
+      [
+        "r",
+        "ound"
+      ],
+      [
+        "r",
+        "a"
+      ],
+      [
+        "r",
+        "y"
+      ],
+      [
+        "Ġsm",
+        "iled"
+      ],
+      [
+        "Ġh",
+        "im"
+      ],
+      [
+        "Ġ",
+        "A"
+      ],
+      [
+        "Ġm",
+        "om"
+      ],
+      [
+        "he",
+        "n"
+      ],
+      [
+        "w",
+        "ay"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "ĠI",
+        "t"
+      ],
+      [
+        "fu",
+        "l"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "Ġw",
+        "ent"
+      ],
+      [
+        "Ġhe",
+        "l"
+      ],
+      [
+        "Ġn",
+        "ot"
+      ],
+      [
+        "ĠThe",
+        "y"
+      ],
+      [
+        "Ġwan",
+        "ted"
+      ],
+      [
+        "i",
+        "nd"
+      ],
+      [
+        "a",
+        "re"
+      ],
+      [
+        "e",
+        "ar"
+      ],
+      [
+        "Ġ",
+        "M"
+      ],
+      [
+        "Ġa",
+        "ll"
+      ],
+      [
+        "Ġfriend",
+        "s"
+      ],
+      [
+        "Ġto",
+        "o"
+      ],
+      [
+        "Ġg",
+        "o"
+      ],
+      [
+        "il",
+        "y"
+      ],
+      [
+        "am",
+        "e"
+      ],
+      [
+        "ĠT",
+        "im"
+      ],
+      [
+        "Ġhel",
+        "p"
+      ],
+      [
+        "om",
+        "et"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "Ġl",
+        "o"
+      ],
+      [
+        "g",
+        "ht"
+      ],
+      [
+        "Ġs",
+        "omet"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "Ġd",
+        "o"
+      ],
+      [
+        "Ġas",
+        "ked"
+      ],
+      [
+        "!",
+        "\""
+      ],
+      [
+        "Ġa",
+        "round"
+      ],
+      [
+        "Ġ",
+        "j"
+      ],
+      [
+        "re",
+        "e"
+      ],
+      [
+        "Ġloo",
+        "ked"
+      ],
+      [
+        "Ġsomet",
+        "hing"
+      ],
+      [
+        "Ġs",
+        "e"
+      ],
+      [
+        "Ġw",
+        "or"
+      ],
+      [
+        "d",
+        "d"
+      ],
+      [
+        "he",
+        "d"
+      ],
+      [
+        "oo",
+        "d"
+      ],
+      [
+        "Ġc",
+        "l"
+      ],
+      [
+        "am",
+        "ed"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "ar",
+        "k"
+      ],
+      [
+        "k",
+        "ing"
+      ],
+      [
+        "Ġ",
+        "E"
+      ],
+      [
+        "r",
+        "om"
+      ],
+      [
+        "Ġb",
+        "ack"
+      ],
+      [
+        "Ġex",
+        "c"
+      ],
+      [
+        "a",
+        "b"
+      ],
+      [
+        "i",
+        "ck"
+      ]
+    ]
+  }
+}

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-080428/config.json

@@ -0,0 +1,68 @@
+{
+  "args": {
+    "corpus": null,
+    "corpus_glob": "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet",
+    "text_column": "text",
+    "output_dir": "runs/stage-dropout-multiseed",
+    "suite": "stage_dropouts",
+    "condition_name": "mild_decay_030_022_014_010",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 5000000,
+    "stream_token_caps": [
+      5000000,
+      10000000,
+      20000000,
+      40000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 2000,
+    "stream_steps_per_stage": 1000,
+    "eval_batches": 64,
+    "log_every": 500,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "stage_dropouts": [
+      0.3,
+      0.22,
+      0.14,
+      0.1
+    ],
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/stage-dropout-multiseed/20260523-080428/tokenizer.json",
+  "encoded_path": "runs/stage-dropout-multiseed/20260523-080428/tokens-v4096-uint16.npy",
+  "train_tokens": 40000160,
+  "val_tokens": 500000,
+  "effective_initial_tokens": 5000000,
+  "effective_stream_token_caps": [
+    5000000,
+    10000000,
+    20000000,
+    40000000
+  ]
+}

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-080428/metrics.jsonl

@@ -0,0 +1,20 @@
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.3, "train_loss_last": 5.58997917175293, "eval_loss": 5.751125849783421, "elapsed_sec": 44.69669818878174, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.22, "train_loss_last": 5.062872886657715, "eval_loss": 5.196203097701073, "elapsed_sec": 44.78069806098938, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.14, "train_loss_last": 4.952004432678223, "eval_loss": 4.905544601380825, "elapsed_sec": 45.88518309593201, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.558305263519287, "eval_loss": 4.7524150758981705, "elapsed_sec": 48.00120186805725, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 2, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.3, "train_loss_last": 5.574131011962891, "eval_loss": 5.8797784596681595, "elapsed_sec": 49.94093990325928, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 2, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.22, "train_loss_last": 5.157837867736816, "eval_loss": 5.257638864219189, "elapsed_sec": 51.3558988571167, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 2, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.14, "train_loss_last": 4.9385457038879395, "eval_loss": 4.926844872534275, "elapsed_sec": 52.127033948898315, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 2, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.661777973175049, "eval_loss": 4.708192884922028, "elapsed_sec": 52.372743129730225, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 3, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.3, "train_loss_last": 5.643334865570068, "eval_loss": 5.8101136311888695, "elapsed_sec": 54.14305257797241, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 3, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.22, "train_loss_last": 5.042954921722412, "eval_loss": 5.163986533880234, "elapsed_sec": 53.76744985580444, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 3, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.14, "train_loss_last": 4.887018203735352, "eval_loss": 4.921479769051075, "elapsed_sec": 53.323330879211426, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 3, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.6436262130737305, "eval_loss": 4.727723777294159, "elapsed_sec": 53.821449995040894, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 4, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.3, "train_loss_last": 5.597998142242432, "eval_loss": 5.849507503211498, "elapsed_sec": 52.95270609855652, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 4, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.22, "train_loss_last": 5.1180315017700195, "eval_loss": 5.21679612249136, "elapsed_sec": 52.781530141830444, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 4, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.14, "train_loss_last": 4.8136138916015625, "eval_loss": 4.85420498996973, "elapsed_sec": 52.83119297027588, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 4, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.631038665771484, "eval_loss": 4.644085839390755, "elapsed_sec": 52.84165716171265, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 5, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.3, "train_loss_last": 5.687386989593506, "eval_loss": 5.774208791553974, "elapsed_sec": 52.76228904724121, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 5, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.22, "train_loss_last": 5.182829856872559, "eval_loss": 5.188988164067268, "elapsed_sec": 52.99323225021362, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 5, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.14, "train_loss_last": 4.755054473876953, "eval_loss": 4.887606129050255, "elapsed_sec": 52.93570303916931, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 5, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.790690898895264, "eval_loss": 4.684750318527222, "elapsed_sec": 52.974026918411255, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-080428/summary.csv

@@ -0,0 +1,5 @@
+phase,condition,stage,token_limit,parameters,n,mean_eval_loss,std_eval_loss
+phase5_stage_dropout_search,mild_decay_030_022_014_010,0,5000000,8388608,5,5.812946847081184,0.05275804626515845
+phase5_stage_dropout_search,mild_decay_030_022_014_010,1,10000000,8388608,5,5.2047225564718245,0.03509089070083401
+phase5_stage_dropout_search,mild_decay_030_022_014_010,2,20000000,8388608,5,4.899136072397232,0.02941015721914179
+phase5_stage_dropout_search,mild_decay_030_022_014_010,3,40000000,8388608,5,4.703433579206466,0.04148213526035924

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-080428/summary.json

@@ -0,0 +1,42 @@
+[
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 0,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 5.812946847081184,
+    "std_eval_loss": 0.05275804626515845
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 1,
+    "token_limit": 10000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 5.2047225564718245,
+    "std_eval_loss": 0.03509089070083401
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 2,
+    "token_limit": 20000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.899136072397232,
+    "std_eval_loss": 0.02941015721914179
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 3,
+    "token_limit": 40000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.703433579206466,
+    "std_eval_loss": 0.04148213526035924
+  }
+]

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-082216/config.json

@@ -0,0 +1,68 @@
+{
+  "args": {
+    "corpus": null,
+    "corpus_glob": "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet",
+    "text_column": "text",
+    "output_dir": "runs/stage-dropout-multiseed",
+    "suite": "stage_dropouts",
+    "condition_name": "fast_decay_050_010_010_010",
+    "seeds": [
+      1,
+      2,
+      3,
+      4,
+      5
+    ],
+    "initial_tokens": 5000000,
+    "stream_token_caps": [
+      5000000,
+      10000000,
+      20000000,
+      40000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 2000,
+    "stream_steps_per_stage": 1000,
+    "eval_batches": 64,
+    "log_every": 500,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "stage_dropouts": [
+      0.5,
+      0.1,
+      0.1,
+      0.1
+    ],
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/stage-dropout-multiseed/20260523-082216/tokenizer.json",
+  "encoded_path": "runs/stage-dropout-multiseed/20260523-082216/tokens-v4096-uint16.npy",
+  "train_tokens": 40000160,
+  "val_tokens": 500000,
+  "effective_initial_tokens": 5000000,
+  "effective_stream_token_caps": [
+    5000000,
+    10000000,
+    20000000,
+    40000000
+  ]
+}

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-082216/metrics.jsonl

@@ -0,0 +1,20 @@
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 1, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.5, "train_loss_last": 5.845836162567139, "eval_loss": 6.154372617602348, "elapsed_sec": 49.94526386260986, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 1, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.0081586837768555, "eval_loss": 5.162623845040798, "elapsed_sec": 49.56790518760681, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 1, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.929193496704102, "eval_loss": 4.882447637617588, "elapsed_sec": 50.52619385719299, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 1, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.557829856872559, "eval_loss": 4.758259400725365, "elapsed_sec": 52.02269697189331, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 2, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.5, "train_loss_last": 5.822956085205078, "eval_loss": 6.242104984819889, "elapsed_sec": 52.7607798576355, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 2, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.105504035949707, "eval_loss": 5.223574630916119, "elapsed_sec": 53.23060607910156, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 2, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.897387504577637, "eval_loss": 4.94177483022213, "elapsed_sec": 53.174906969070435, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 2, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.672600746154785, "eval_loss": 4.714911535382271, "elapsed_sec": 53.14971899986267, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 3, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.5, "train_loss_last": 5.9172844886779785, "eval_loss": 6.228554867208004, "elapsed_sec": 53.18459105491638, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 3, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.001498222351074, "eval_loss": 5.157509610056877, "elapsed_sec": 53.15500092506409, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 3, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.853842735290527, "eval_loss": 4.896874591708183, "elapsed_sec": 53.38427805900574, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 3, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.641301155090332, "eval_loss": 4.713992565870285, "elapsed_sec": 53.49088001251221, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 4, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.5, "train_loss_last": 5.845858573913574, "eval_loss": 6.229139141738415, "elapsed_sec": 53.52108716964722, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 4, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.085396766662598, "eval_loss": 5.225671216845512, "elapsed_sec": 53.089759826660156, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 4, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.777719020843506, "eval_loss": 4.850255951285362, "elapsed_sec": 52.85016393661499, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 4, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.632856369018555, "eval_loss": 4.651223048567772, "elapsed_sec": 52.797423124313354, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 5, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.5, "train_loss_last": 5.914695739746094, "eval_loss": 6.148795463144779, "elapsed_sec": 52.897231101989746, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 5, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.1, "train_loss_last": 5.125888347625732, "eval_loss": 5.160975947976112, "elapsed_sec": 52.83045816421509, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 5, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.1, "train_loss_last": 4.719311714172363, "eval_loss": 4.865957573056221, "elapsed_sec": 52.88761901855469, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "fast_decay_050_010_010_010", "seed": 5, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.78352165222168, "eval_loss": 4.673870116472244, "elapsed_sec": 52.91818380355835, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-082216/summary.csv

@@ -0,0 +1,5 @@
+phase,condition,stage,token_limit,parameters,n,mean_eval_loss,std_eval_loss
+phase5_stage_dropout_search,fast_decay_050_010_010_010,0,5000000,8388608,5,6.200593414902687,0.0451090392326937
+phase5_stage_dropout_search,fast_decay_050_010_010_010,1,10000000,8388608,5,5.186071050167084,0.0352490539353081
+phase5_stage_dropout_search,fast_decay_050_010_010_010,2,20000000,8388608,5,4.887462116777897,0.03503696147846281
+phase5_stage_dropout_search,fast_decay_050_010_010_010,3,40000000,8388608,5,4.702451333403587,0.041364901144835056

archive/runs_legacy_20260525/stage-dropout-multiseed/20260523-082216/summary.json

@@ -0,0 +1,42 @@
+[
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "fast_decay_050_010_010_010",
+    "stage": 0,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 6.200593414902687,
+    "std_eval_loss": 0.0451090392326937
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "fast_decay_050_010_010_010",
+    "stage": 1,
+    "token_limit": 10000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 5.186071050167084,
+    "std_eval_loss": 0.0352490539353081
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "fast_decay_050_010_010_010",
+    "stage": 2,
+    "token_limit": 20000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.887462116777897,
+    "std_eval_loss": 0.03503696147846281
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "fast_decay_050_010_010_010",
+    "stage": 3,
+    "token_limit": 40000000,
+    "parameters": 8388608,
+    "n": 5,
+    "mean_eval_loss": 4.702451333403587,
+    "std_eval_loss": 0.041364901144835056
+  }
+]

archive/runs_legacy_20260525/stage-dropout-single/20260523-075308/config.json

@@ -0,0 +1,64 @@
+{
+  "args": {
+    "corpus": null,
+    "corpus_glob": "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet",
+    "text_column": "text",
+    "output_dir": "runs/stage-dropout-single",
+    "suite": "stage_dropouts",
+    "condition_name": "mild_decay_030_022_014_010",
+    "seeds": [
+      1
+    ],
+    "initial_tokens": 5000000,
+    "stream_token_caps": [
+      5000000,
+      10000000,
+      20000000,
+      40000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 2000,
+    "stream_steps_per_stage": 1000,
+    "eval_batches": 64,
+    "log_every": 500,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "stage_dropouts": [
+      0.3,
+      0.22,
+      0.14,
+      0.1
+    ],
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/stage-dropout-single/20260523-075308/tokenizer.json",
+  "encoded_path": "runs/stage-dropout-single/20260523-075308/tokens-v4096-uint16.npy",
+  "train_tokens": 40000160,
+  "val_tokens": 500000,
+  "effective_initial_tokens": 5000000,
+  "effective_stream_token_caps": [
+    5000000,
+    10000000,
+    20000000,
+    40000000
+  ]
+}

archive/runs_legacy_20260525/stage-dropout-single/20260523-075308/metrics.jsonl

@@ -0,0 +1,4 @@
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.3, "train_loss_last": 5.58997917175293, "eval_loss": 5.751125819981098, "elapsed_sec": 41.68040323257446, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.22, "train_loss_last": 5.062872886657715, "eval_loss": 5.1962031945586205, "elapsed_sec": 42.56834697723389, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.14, "train_loss_last": 4.952004432678223, "eval_loss": 4.905544675886631, "elapsed_sec": 42.689231157302856, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}
+{"phase": "phase5_stage_dropout_search", "condition": "mild_decay_030_022_014_010", "seed": 1, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.558305740356445, "eval_loss": 4.752415433526039, "elapsed_sec": 42.67094111442566, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.3}}

archive/runs_legacy_20260525/stage-dropout-single/20260523-075308/summary.csv

@@ -0,0 +1,5 @@
+phase,condition,stage,token_limit,parameters,n,mean_eval_loss,std_eval_loss
+phase5_stage_dropout_search,mild_decay_030_022_014_010,0,5000000,8388608,1,5.751125819981098,0.0
+phase5_stage_dropout_search,mild_decay_030_022_014_010,1,10000000,8388608,1,5.1962031945586205,0.0
+phase5_stage_dropout_search,mild_decay_030_022_014_010,2,20000000,8388608,1,4.905544675886631,0.0
+phase5_stage_dropout_search,mild_decay_030_022_014_010,3,40000000,8388608,1,4.752415433526039,0.0

archive/runs_legacy_20260525/stage-dropout-single/20260523-075308/summary.json

@@ -0,0 +1,42 @@
+[
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 0,
+    "token_limit": 5000000,
+    "parameters": 8388608,
+    "n": 1,
+    "mean_eval_loss": 5.751125819981098,
+    "std_eval_loss": 0.0
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 1,
+    "token_limit": 10000000,
+    "parameters": 8388608,
+    "n": 1,
+    "mean_eval_loss": 5.1962031945586205,
+    "std_eval_loss": 0.0
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 2,
+    "token_limit": 20000000,
+    "parameters": 8388608,
+    "n": 1,
+    "mean_eval_loss": 4.905544675886631,
+    "std_eval_loss": 0.0
+  },
+  {
+    "phase": "phase5_stage_dropout_search",
+    "condition": "mild_decay_030_022_014_010",
+    "stage": 3,
+    "token_limit": 40000000,
+    "parameters": 8388608,
+    "n": 1,
+    "mean_eval_loss": 4.752415433526039,
+    "std_eval_loss": 0.0
+  }
+]

archive/runs_legacy_20260525/stage-dropout-single/20260523-075636/config.json

@@ -0,0 +1,64 @@
+{
+  "args": {
+    "corpus": null,
+    "corpus_glob": "/Users/mandeepsidhu/Desktop/code/nanochat/.nanochat-cache/base_data_climbmix/shard_*.parquet",
+    "text_column": "text",
+    "output_dir": "runs/stage-dropout-single",
+    "suite": "stage_dropouts",
+    "condition_name": "medium_decay_050_032_016_010",
+    "seeds": [
+      1
+    ],
+    "initial_tokens": 5000000,
+    "stream_token_caps": [
+      5000000,
+      10000000,
+      20000000,
+      40000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "small_layers": 4,
+    "small_heads": 4,
+    "small_embd": 128,
+    "large_layers": 8,
+    "large_heads": 8,
+    "large_embd": 256,
+    "steps_per_run": 2000,
+    "stream_steps_per_stage": 1000,
+    "eval_batches": 64,
+    "log_every": 500,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "baseline_dropout": 0.1,
+    "high_dropout": 0.8,
+    "stage_dropouts": [
+      0.5,
+      0.32,
+      0.16,
+      0.1
+    ],
+    "dropout_decay_tokens": null,
+    "dropout_schedule": "cosine"
+  },
+  "device": "mps",
+  "torch": "2.9.1",
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": "runs/stage-dropout-single/20260523-075636/tokenizer.json",
+  "encoded_path": "runs/stage-dropout-single/20260523-075636/tokens-v4096-uint16.npy",
+  "train_tokens": 40000160,
+  "val_tokens": 500000,
+  "effective_initial_tokens": 5000000,
+  "effective_stream_token_caps": [
+    5000000,
+    10000000,
+    20000000,
+    40000000
+  ]
+}

archive/runs_legacy_20260525/stage-dropout-single/20260523-075636/metrics.jsonl

@@ -0,0 +1,4 @@
+{"phase": "phase5_stage_dropout_search", "condition": "medium_decay_050_032_016_010", "seed": 1, "stage": 0, "token_limit": 5000000, "steps": 1000, "tokens_seen": 2048000, "dropout": 0.5, "train_loss_last": 5.845836162567139, "eval_loss": 6.154372535645962, "elapsed_sec": 41.894405126571655, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "medium_decay_050_032_016_010", "seed": 1, "stage": 1, "token_limit": 10000000, "steps": 1000, "tokens_seen": 4096000, "dropout": 0.32, "train_loss_last": 5.22263240814209, "eval_loss": 5.326081059873104, "elapsed_sec": 42.87401509284973, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "medium_decay_050_032_016_010", "seed": 1, "stage": 2, "token_limit": 20000000, "steps": 1000, "tokens_seen": 6144000, "dropout": 0.16, "train_loss_last": 5.021544456481934, "eval_loss": 4.956579685211182, "elapsed_sec": 44.26910591125488, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}
+{"phase": "phase5_stage_dropout_search", "condition": "medium_decay_050_032_016_010", "seed": 1, "stage": 3, "token_limit": 40000000, "steps": 1000, "tokens_seen": 8192000, "dropout": 0.1, "train_loss_last": 4.601977348327637, "eval_loss": 4.809504576027393, "elapsed_sec": 45.77408504486084, "parameters": 8388608, "model_config": {"block_size": 128, "vocab_size": 4096, "n_layer": 8, "n_head": 8, "n_embd": 256, "dropout": 0.5}}