Add WikiText corpus holdout results

REPRODUCING.md

@@ -58,6 +58,19 @@ validation tokens: 500,000
 vocab size: 4,096
 ```
 
+The public WikiText-103 holdout can be rebuilt from source:
+
+```bash
+.venv/bin/python scripts/prepare_wikitext103.py
+```
+
+The pinned parquet source is verified as:
+
+```text
+bytes: 156,700,942
+sha256: 75aa65dee9de2a7c10ba1808efd2408c3f4eb008104c3ccac47f8ed19300ebdd
+```
+
 ## Smoke Test
 
 This verifies cached-data loading without running a Torch experiment:
@@ -215,6 +228,49 @@ best static final validation: 4.4946 +/- 0.0087
 best mean trajectory:        static 0.18, 4.9064 vs formula 4.9073
 ```
 
+## Reproduce WikiText-103 Corpus Holdout
+
+Prepare the public corpus first:
+
+```bash
+.venv/bin/python scripts/prepare_wikitext103.py
+```
+
+Then run the frozen L12 formula against a broad static grid:
+
+```bash
+PYTHONPATH=src .venv/bin/python scripts/run_experiments.py \
+  --mode locked_stream \
+  --corpus data/wikitext103_raw/train-00001-of-00002.parquet \
+  --cache-dir .cache/dropout_decay_wikitext103 \
+  --output-dir runs/corpus_holdout_wikitext103_l12 \
+  --models L12_H8_D320=12x8x320 \
+  --seeds 1 2 3 \
+  --stream-token-caps 250000 500000 1000000 2000000 4000000 \
+  --dropout-rates 0.00 0.02 0.06 0.08 0.10 0.14 0.18 0.20 0.26 0.30 \
+  --anchor-decays formula_l12_wikitext103:250000=0.300,500000=0.260,1000000=0.180,2000000=0.090,4000000=0.020 \
+  --stage-steps 1000 \
+  --batch-size 16 \
+  --block-size 128 \
+  --eval-batches 64 \
+  --train-eval-batches 32 \
+  --trace-eval-batches 8 \
+  --log-every 500 \
+  --vocab-size 4096 \
+  --val-tokens 500000 \
+  --lr 0.0003 \
+  --weight-decay 0.1 \
+  --grad-clip 1.0
+```
+
+Completed reference result:
+
+```text
+formula final validation:     4.0836 +/- 0.0258
+best static final validation: 4.1081 +/- 0.0258
+best mean trajectory:        formula, 4.5728 vs static 0.18, 4.5759
+```
+
 ## Notes for Publication
 
 - Do not claim the formula is universal.

docs/dropout_decay_research_report_v2.md

@@ -42,12 +42,13 @@ p = clamp(0.02, 0.65,
 ```
 
 Across the completed headline validation runs, the formula schedule wins
-`21/21` paired final-loss comparisons across five model sizes and two
-architecture-shape holdouts. The evidence supports final-validation improvement
-under this nanochat-style Transformer and expanding-prefix protocol. It does not
-yet establish a universal dropout law across datasets, architectures, or
-training scales, and the width-heavy holdout shows that the current formula can
-overestimate the best early-prefix dropout for some architecture shapes.
+`24/24` paired final-loss comparisons across five model sizes, two
+architecture-shape holdouts, and one public-corpus holdout. The evidence
+supports final-validation improvement under this nanochat-style Transformer and
+expanding-prefix protocol. It does not yet establish a universal dropout law
+across datasets, architectures, or training scales. The width-heavy and
+WikiText-103 holdouts both show that the current formula can overestimate the
+best early-prefix dropout even when it wins final loss.
 
 ## System Under Test
 
@@ -413,7 +414,8 @@ Completed headline evidence:
 | Model-size validation | `15/15` paired final-loss wins |
 | Deep/narrow architecture holdout | `3/3` paired final-loss wins |
 | Width-heavy architecture holdout | `3/3` paired final-loss wins |
-| Combined paired final-loss comparisons | `21/21` wins |
+| WikiText-103 corpus holdout | `3/3` paired final-loss wins |
+| Combined paired final-loss comparisons | `24/24` wins |
 | Update-pressure direction | Supported on L12 |
 | Sampled-pressure coefficient | Supported on L12 |
 | High arbitrary initial dropout | Rejected |
@@ -448,6 +450,7 @@ below expose more of the completed run surface.
 | `l12_sample_pressure_ablation_053842` | L12 sampled-pressure coefficient ablation | 3 |
 | `deep_narrow_h8_112117` | Deep/narrow architecture-shape holdout | 3 |
 | `wide_h8_151721` | Width-heavy architecture-shape holdout | 3 |
+| `wikitext103_l12_183213` | Public-corpus holdout on WikiText-103 raw | 3 |
 
 ### Static Screen Optima
 
@@ -637,6 +640,43 @@ loss because it decayed to low dropout at the largest prefixes. This suggests
 that final-loss transfer is real, but an architecture-shape term may be needed
 to avoid overestimating early dropout for wide models.
 
+### Corpus Holdout Final Controls
+
+The first public-corpus holdout freezes the L12 formula and reruns the same
+locked-stream protocol on WikiText-103 raw text, using a fresh local 4,096-token
+BPE cache from the public Hugging Face parquet source. This tests whether the
+schedule transfers beyond the original local cached corpus.
+
+| Condition | N | Final val | Val std | Final train | Final gap |
+|---|---:|---:|---:|---:|---:|
+| formula | 3 | 4.0836 | 0.0258 | 3.8130 | 0.2707 |
+| static 0.10 | 3 | 4.1081 | 0.0258 | 3.8531 | 0.2549 |
+| static 0.08 | 3 | 4.1101 | 0.0247 | 3.8331 | 0.2769 |
+| static 0.06 | 3 | 4.1181 | 0.0107 | 3.8177 | 0.3003 |
+| static 0.14 | 3 | 4.1266 | 0.0198 | 3.8825 | 0.2441 |
+| static 0.18 | 3 | 4.1288 | 0.0181 | 3.9090 | 0.2198 |
+| static 0.20 | 3 | 4.1388 | 0.0236 | 3.9251 | 0.2136 |
+| static 0.02 | 3 | 4.1520 | 0.0199 | 3.7968 | 0.3552 |
+| static 0.26 | 3 | 4.1772 | 0.0203 | 3.9888 | 0.1883 |
+| static 0.00 | 3 | 4.1796 | 0.0217 | 3.7873 | 0.3922 |
+| static 0.30 | 3 | 4.1961 | 0.0179 | 4.0251 | 0.1710 |
+
+Best static final loss varied by seed, but formula again beat the best static
+condition in every paired final comparison:
+
+```text
+seed 1: 4.0623 vs static 0.10 at 4.0807, delta -0.0184
+seed 2: 4.1123 vs static 0.06 at 4.1304, delta -0.0181
+seed 3: 4.0763 vs static 0.08 at 4.1036, delta -0.0272
+```
+
+This is an important positive holdout because formula also had the best mean
+trajectory loss, `4.5728`, versus best static `0.18` at `4.5759`. It still
+exposes a weakness: the first two prefixes favored static rates around
+`0.14-0.20`, while formula started at `0.30 -> 0.26`. The final win comes from
+decaying below the best early static rate, not from perfectly predicting the
+early optimum.
+
 ## Interpretation
 
 The most plausible mechanism is pressure tracking:
@@ -653,10 +693,11 @@ The most plausible mechanism is pressure tracking:
 
 This interpretation is consistent with the static screens, the model-size
 interpolation results, the update-pressure sweep, the sampled-pressure
-coefficient ablation, and the two architecture-shape holdouts. The width-heavy
-holdout adds an important refinement: parameter count alone does not fully
-describe architecture capacity, because the formula's early dropout was higher
-than the measured early-prefix static optimum for that shape.
+coefficient ablation, the two architecture-shape holdouts, and the WikiText-103
+corpus holdout. The holdouts add an important refinement: parameter count alone
+does not fully describe capacity or data difficulty, because the formula's
+early dropout was higher than the measured early-prefix static optimum for the
+wide shape and for WikiText-103.
 
 ## What This Report Does Not Prove
 
@@ -685,7 +726,7 @@ The strongest safe paper claim is:
 > In nanochat-style causal Transformers trained under an expanding-prefix
 > streaming protocol, a pressure-aware dropout schedule improves final
 > validation loss over fixed-dropout baselines across model sizes, update
-> pressures, and two architecture-shape holdouts.
+> pressures, two architecture-shape holdouts, and one public-corpus holdout.
 
 Claims to avoid:
 
@@ -698,11 +739,13 @@ Claims to avoid:
 
 The next experiments that would most strengthen a paper are:
 
-1. Corpus/domain holdout: freeze the formula and run on a different text
-   distribution. This is the largest missing generalization test.
-2. Architecture-shape refinement: add a small feature such as depth/width ratio
-   or embedding dimension to reduce early-dropout overestimation on wide models,
-   then validate it on held-out shapes.
+1. Second corpus/domain holdout: freeze the formula and run on another public
+   text distribution. This checks whether the WikiText-103 result is a single
+   favorable domain or a broader transfer result.
+2. Architecture/data-shape refinement: add a small feature such as
+   depth/width ratio, embedding dimension, or a dataset/tokenization statistic
+   to reduce early-dropout overestimation, then validate it on held-out shapes
+   and corpora.
 3. L8 and L16 sampled-pressure ablations: repeat the `0x`, `0.5x`, `1.0x`, and
    `1.5x` coefficient ablation outside L12.
 4. Oracle schedule comparison: compare the formula against a stage-wise oracle

runs/corpus_holdout_wikitext103_l12/locked_stream/20260528-183213/RESULT_SUMMARY.md

@@ -0,0 +1,104 @@
+# Locked Streaming Dropout Summary
+
+Run directory: `runs/corpus_holdout_wikitext103_l12/locked_stream/20260528-183213`
+
+Model: `L12_H8_D320` causal Transformer, 17,367,040 parameters, 12 layers, 8 heads, 320 embedding dim.
+Training per stage: 1,000 steps. Sampled tokens are cumulative in each stage row. Seeds present: 1, 2, 3.
+
+## Condition Ranking
+
+| Condition | Kind | Final dropout | Mean trajectory val loss | Final val loss | Final gap | Dropout path |
+|---|---|---:|---:|---:|---:|---|
+| `formula_l12_wikitext103` | anchor_decay | 0.02 | 4.5728 | 4.0836 | 0.2707 | 0.30 -> 0.26 -> 0.18 -> 0.09 -> 0.02 |
+| `static_dropout_0.18` | static | 0.18 | 4.5759 | 4.1288 | 0.2198 | 0.18 -> 0.18 -> 0.18 -> 0.18 -> 0.18 |
+| `static_dropout_0.14` | static | 0.14 | 4.5775 | 4.1266 | 0.2441 | 0.14 -> 0.14 -> 0.14 -> 0.14 -> 0.14 |
+| `static_dropout_0.2` | static | 0.20 | 4.5797 | 4.1388 | 0.2136 | 0.20 -> 0.20 -> 0.20 -> 0.20 -> 0.20 |
+| `static_dropout_0.1` | static | 0.10 | 4.5844 | 4.1081 | 0.2549 | 0.10 -> 0.10 -> 0.10 -> 0.10 -> 0.10 |
+| `static_dropout_0.08` | static | 0.08 | 4.5976 | 4.1101 | 0.2769 | 0.08 -> 0.08 -> 0.08 -> 0.08 -> 0.08 |
+| `static_dropout_0.26` | static | 0.26 | 4.6081 | 4.1772 | 0.1883 | 0.26 -> 0.26 -> 0.26 -> 0.26 -> 0.26 |
+| `static_dropout_0.06` | static | 0.06 | 4.6174 | 4.1181 | 0.3003 | 0.06 -> 0.06 -> 0.06 -> 0.06 -> 0.06 |
+| `static_dropout_0.3` | static | 0.30 | 4.6266 | 4.1961 | 0.1710 | 0.30 -> 0.30 -> 0.30 -> 0.30 -> 0.30 |
+| `static_dropout_0.02` | static | 0.02 | 4.7008 | 4.1520 | 0.3552 | 0.02 -> 0.02 -> 0.02 -> 0.02 -> 0.02 |
+| `static_dropout_0` | static | 0.00 | 4.7735 | 4.1796 | 0.3922 | 0.00 -> 0.00 -> 0.00 -> 0.00 -> 0.00 |
+
+## Stage Trajectory
+
+### Stage 0: 250,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `static_dropout_0.18` | 0.18 | 5.1632 | 4.0077 | 1.1555 | 3 |
+| `static_dropout_0.14` | 0.14 | 5.1693 | 3.9113 | 1.2579 | 3 |
+| `static_dropout_0.2` | 0.20 | 5.1730 | 4.0440 | 1.1290 | 3 |
+| `static_dropout_0.1` | 0.10 | 5.1741 | 3.8214 | 1.3527 | 3 |
+| `static_dropout_0.08` | 0.08 | 5.1900 | 3.7643 | 1.4257 | 3 |
+| `static_dropout_0.26` | 0.26 | 5.1977 | 4.1614 | 1.0363 | 3 |
+| `static_dropout_0.06` | 0.06 | 5.2117 | 3.7253 | 1.4864 | 3 |
+| `formula_l12_wikitext103` | 0.30 | 5.2139 | 4.2195 | 0.9944 | 3 |
+| `static_dropout_0.3` | 0.30 | 5.2139 | 4.2195 | 0.9944 | 3 |
+| `static_dropout_0.02` | 0.02 | 5.2779 | 3.5912 | 1.6867 | 3 |
+| `static_dropout_0` | 0.00 | 5.3396 | 3.5193 | 1.8203 | 3 |
+
+### Stage 1: 500,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `static_dropout_0.2` | 0.20 | 4.7875 | 3.7917 | 0.9959 | 3 |
+| `static_dropout_0.18` | 0.18 | 4.7958 | 3.7620 | 1.0338 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.8048 | 3.6746 | 1.1302 | 3 |
+| `formula_l12_wikitext103` | 0.26 | 4.8070 | 3.9220 | 0.8851 | 3 |
+| `static_dropout_0.26` | 0.26 | 4.8103 | 3.9032 | 0.9070 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.8266 | 3.9801 | 0.8465 | 3 |
+| `static_dropout_0.1` | 0.10 | 4.8324 | 3.5595 | 1.2729 | 3 |
+| `static_dropout_0.08` | 0.08 | 4.8634 | 3.4963 | 1.3671 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.8951 | 3.4347 | 1.4603 | 3 |
+| `static_dropout_0.02` | 0.02 | 5.0488 | 3.2697 | 1.7791 | 3 |
+| `static_dropout_0` | 0.00 | 5.1751 | 3.1573 | 2.0178 | 3 |
+
+### Stage 2: 1,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `formula_l12_wikitext103` | 0.18 | 4.4945 | 3.8363 | 0.6582 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.5009 | 3.7184 | 0.7825 | 3 |
+| `static_dropout_0.2` | 0.20 | 4.5016 | 3.8116 | 0.6900 | 3 |
+| `static_dropout_0.18` | 0.18 | 4.5027 | 3.7867 | 0.7160 | 3 |
+| `static_dropout_0.1` | 0.10 | 4.5167 | 3.6543 | 0.8624 | 3 |
+| `static_dropout_0.26` | 0.26 | 4.5273 | 3.9097 | 0.6175 | 3 |
+| `static_dropout_0.08` | 0.08 | 4.5324 | 3.6026 | 0.9298 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.5468 | 3.9773 | 0.5695 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.5575 | 3.5541 | 1.0034 | 3 |
+| `static_dropout_0.02` | 0.02 | 4.6572 | 3.4398 | 1.2173 | 3 |
+| `static_dropout_0` | 0.00 | 4.7597 | 3.3638 | 1.3959 | 3 |
+
+### Stage 3: 2,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `formula_l12_wikitext103` | 0.09 | 4.2650 | 3.8133 | 0.4518 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.2859 | 3.8283 | 0.4575 | 3 |
+| `static_dropout_0.18` | 0.18 | 4.2890 | 3.8752 | 0.4138 | 3 |
+| `static_dropout_0.1` | 0.10 | 4.2908 | 3.7856 | 0.5053 | 3 |
+| `static_dropout_0.08` | 0.08 | 4.2920 | 3.7518 | 0.5403 | 3 |
+| `static_dropout_0.2` | 0.20 | 4.2976 | 3.8983 | 0.3993 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.3047 | 3.7288 | 0.5759 | 3 |
+| `static_dropout_0.26` | 0.26 | 4.3278 | 3.9675 | 0.3604 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.3497 | 4.0121 | 0.3376 | 3 |
+| `static_dropout_0.02` | 0.02 | 4.3680 | 3.6687 | 0.6993 | 3 |
+| `static_dropout_0` | 0.00 | 4.4136 | 3.6351 | 0.7785 | 3 |
+
+### Stage 4: 4,000,000 Prefix Tokens
+
+| Condition | Dropout | Mean val loss | Mean train loss | Mean gap | N |
+|---|---:|---:|---:|---:|---:|
+| `formula_l12_wikitext103` | 0.02 | 4.0836 | 3.8130 | 0.2707 | 3 |
+| `static_dropout_0.1` | 0.10 | 4.1081 | 3.8531 | 0.2549 | 3 |
+| `static_dropout_0.08` | 0.08 | 4.1101 | 3.8331 | 0.2769 | 3 |
+| `static_dropout_0.06` | 0.06 | 4.1181 | 3.8177 | 0.3003 | 3 |
+| `static_dropout_0.14` | 0.14 | 4.1266 | 3.8825 | 0.2441 | 3 |
+| `static_dropout_0.18` | 0.18 | 4.1288 | 3.9090 | 0.2198 | 3 |
+| `static_dropout_0.2` | 0.20 | 4.1388 | 3.9251 | 0.2136 | 3 |
+| `static_dropout_0.02` | 0.02 | 4.1520 | 3.7968 | 0.3552 | 3 |
+| `static_dropout_0.26` | 0.26 | 4.1772 | 3.9888 | 0.1883 | 3 |
+| `static_dropout_0` | 0.00 | 4.1796 | 3.7873 | 0.3922 | 3 |
+| `static_dropout_0.3` | 0.30 | 4.1961 | 4.0251 | 0.1710 | 3 |

runs/corpus_holdout_wikitext103_l12/locked_stream/20260528-183213/config.json

@@ -0,0 +1,134 @@
+{
+  "args": {
+    "mode": "locked_stream",
+    "corpus": null,
+    "corpus_glob": null,
+    "text_column": "text",
+    "use_cached_data": true,
+    "output_dir": "runs/corpus_holdout_wikitext103_l12",
+    "resume_from": null,
+    "cache_dir": ".cache/dropout_decay_wikitext103",
+    "models": [
+      "L12_H8_D320=12x8x320"
+    ],
+    "seeds": [
+      1,
+      2,
+      3
+    ],
+    "token_limits": [
+      5000000
+    ],
+    "stream_token_caps": [
+      250000,
+      500000,
+      1000000,
+      2000000,
+      4000000
+    ],
+    "val_tokens": 500000,
+    "allow_short_corpus": false,
+    "force_retokenize": false,
+    "vocab_size": 4096,
+    "tokenizer_train_chars": 10000000,
+    "block_size": 128,
+    "batch_size": 16,
+    "steps": 2000,
+    "stage_steps": 1000,
+    "dropout_rates": [
+      0.0,
+      0.02,
+      0.06,
+      0.08,
+      0.1,
+      0.14,
+      0.18,
+      0.2,
+      0.26,
+      0.3
+    ],
+    "decays": [],
+    "anchor_decays": [
+      {
+        "name": "formula_l12_wikitext103",
+        "kind": "anchor_decay",
+        "initial": 0.3,
+        "final": 0.02,
+        "schedule": "log_prefix_anchor",
+        "decay_tokens": null,
+        "anchors": [
+          [
+            250000,
+            0.3
+          ],
+          [
+            500000,
+            0.26
+          ],
+          [
+            1000000,
+            0.18
+          ],
+          [
+            2000000,
+            0.09
+          ],
+          [
+            4000000,
+            0.02
+          ]
+        ]
+      }
+    ],
+    "decay_tokens": null,
+    "eval_batches": 64,
+    "train_eval_batches": 32,
+    "trace_eval_batches": 8,
+    "eval_every": 0,
+    "log_every": 500,
+    "lr": 0.0003,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "plateau_delta": 0.01,
+    "target_min_dropout": 0.1,
+    "min_nonzero_margin": 0.01,
+    "min_high_dropout_margin": 0.03,
+    "screen_early_stop": false,
+    "screen_prune_patience": 3,
+    "screen_prune_min_delta": 0.01
+  },
+  "mode": "locked_stream",
+  "seeds": [
+    1,
+    2,
+    3
+  ],
+  "models": [
+    {
+      "model_name": "L12_H8_D320",
+      "n_layer": 12,
+      "n_head": 8,
+      "n_embd": 320
+    }
+  ],
+  "device": "mps",
+  "torch": "2.12.0",
+  "python": "3.11.15 (main, Mar  3 2026, 00:52:57) [Clang 21.0.0 (clang-2100.0.123.102)]",
+  "mps_available": true,
+  "attribution": "Derived from Andrej Karpathy's nanochat project (https://github.com/karpathy/nanochat), MIT License, Copyright (c) 2025 Andrej Karpathy.",
+  "tokenizer_path": ".cache/dropout_decay_wikitext103/tokenizer-v4096.json",
+  "encoded_path": ".cache/dropout_decay_wikitext103/tokens-v4096-uint16.npy",
+  "train_tokens": 4500020,
+  "val_tokens": 500000,
+  "effective_token_limits": [
+    4500020
+  ],
+  "effective_stream_token_caps": [
+    250000,
+    500000,
+    1000000,
+    2000000,
+    4000000
+  ],
+  "resume_from": null
+}

runs/corpus_holdout_wikitext103_l12/locked_stream/20260528-183213/summary.csv

@@ -0,0 +1,56 @@
+run_mode,condition,condition_kind,stage,token_limit,model_name,n_layer,n_head,n_embd,parameters,dropout_initial,dropout_final,dropout_schedule,n,mean_train_eval_loss,std_train_eval_loss,mean_val_eval_loss,std_val_eval_loss,mean_generalization_gap,std_generalization_gap
+locked_stream,formula_l12_wikitext103,anchor_decay,0,250000,L12_H8_D320,12,8,320,17367040,0.3,0.02,log_prefix_anchor,3,4.219510187705358,0.009400458871163125,5.213888607919216,0.02540367830871252,0.9943784202138582,0.034794492047624956
+locked_stream,static_dropout_0,static,0,250000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,3,3.5193328907092414,0.04538277641898639,5.339607280989488,0.03803900159576214,1.8202743902802467,0.07017278062165234
+locked_stream,static_dropout_0.02,static,0,250000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,3,3.591204417248567,0.009258206352807653,5.27785774320364,0.03450217928391973,1.6866533259550731,0.041632245818121416
+locked_stream,static_dropout_0.06,static,0,250000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,3,3.725302261610826,0.008063637282408817,5.211690753698349,0.02339268745574712,1.4863884920875232,0.031438438863035205
+locked_stream,static_dropout_0.08,static,0,250000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,3,3.764298364520073,0.01498956532264627,5.190002868572871,0.020721921300465723,1.425704504052798,0.014047879271355385
+locked_stream,static_dropout_0.1,static,0,250000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,3,3.821372576057911,0.021364905351568726,5.17405295620362,0.02772884387617372,1.3526803801457088,0.04389058367879646
+locked_stream,static_dropout_0.14,static,0,250000,L12_H8_D320,12,8,320,17367040,0.14,0.14,constant,3,3.9113145818312964,0.013390634728517344,5.169258703788121,0.02886637125607666,1.2579441219568253,0.03490443394763099
+locked_stream,static_dropout_0.18,static,0,250000,L12_H8_D320,12,8,320,17367040,0.18,0.18,constant,3,4.007677458226681,0.011584990895915435,5.163212870558103,0.019303530325978743,1.155535412331422,0.021457887456797633
+locked_stream,static_dropout_0.2,static,0,250000,L12_H8_D320,12,8,320,17367040,0.2,0.2,constant,3,4.0440234914422035,0.006683199962460588,5.173003261288007,0.018770276566139252,1.1289797698458035,0.01557915612774489
+locked_stream,static_dropout_0.26,static,0,250000,L12_H8_D320,12,8,320,17367040,0.26,0.26,constant,3,4.161390465994676,0.017199456198519997,5.197733856737614,0.020026368980316463,1.0363433907429378,0.02452558695313914
+locked_stream,static_dropout_0.3,static,0,250000,L12_H8_D320,12,8,320,17367040,0.3,0.3,constant,3,4.21951021750768,0.00940042069294641,5.213888632754485,0.02540370559129739,0.9943784152468046,0.03479448069461329
+locked_stream,formula_l12_wikitext103,anchor_decay,1,500000,L12_H8_D320,12,8,320,17367040,0.3,0.02,log_prefix_anchor,3,3.921963239709536,0.012701856161083026,4.807015274961789,0.03255901757339794,0.8850520352522532,0.020904354460357738
+locked_stream,static_dropout_0,static,1,500000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,3,3.157259744902452,0.033266149836286125,5.17509716997544,0.027225613071693588,2.017837425072988,0.027271551103611697
+locked_stream,static_dropout_0.02,static,1,500000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,3,3.2696756149331727,0.010229343900978308,5.048807606101036,0.015919759088441253,1.7791319911678631,0.015163975400676688
+locked_stream,static_dropout_0.06,static,1,500000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,3,3.4347175483902297,0.01608227845233568,4.895065948367119,0.025796230814273288,1.4603483999768894,0.016219356742392815
+locked_stream,static_dropout_0.08,static,1,500000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,3,3.49625497063001,0.02296095075813903,4.8633880987763405,0.03137654346429713,1.3671331281463306,0.02205679071854492
+locked_stream,static_dropout_0.1,static,1,500000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,3,3.559451498091221,0.03242743348226214,4.832371026277542,0.03379083503310632,1.2729195281863213,0.004581876102180898
+locked_stream,static_dropout_0.14,static,1,500000,L12_H8_D320,12,8,320,17367040,0.14,0.14,constant,3,3.6746241599321365,0.026787166240812375,4.804830571015676,0.02283391501446146,1.1302064110835393,0.004574911720412448
+locked_stream,static_dropout_0.18,static,1,500000,L12_H8_D320,12,8,320,17367040,0.18,0.18,constant,3,3.761968764166037,0.030310143678321872,4.79575655857722,0.02839125459863536,1.0337877944111824,0.02395919557232544
+locked_stream,static_dropout_0.2,static,1,500000,L12_H8_D320,12,8,320,17367040,0.2,0.2,constant,3,3.7916674092411995,0.02457212816143504,4.78752597173055,0.03269980279988427,0.9958585624893507,0.010890773437236328
+locked_stream,static_dropout_0.26,static,1,500000,L12_H8_D320,12,8,320,17367040,0.26,0.26,constant,3,3.9032329618930817,0.03313463368600539,4.810270811120669,0.029030451515976875,0.9070378492275873,0.0218852812037738
+locked_stream,static_dropout_0.3,static,1,500000,L12_H8_D320,12,8,320,17367040,0.3,0.3,constant,3,3.9801244686047235,0.011984368688744857,4.826624286671479,0.034146701924452114,0.8464998180667559,0.023039413394205236
+locked_stream,formula_l12_wikitext103,anchor_decay,2,1000000,L12_H8_D320,12,8,320,17367040,0.3,0.02,log_prefix_anchor,3,3.836283288896084,0.007165129109779279,4.4944643552104635,0.01869570507899432,0.6581810663143793,0.02324475775596518
+locked_stream,static_dropout_0,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,3,3.363803490996361,0.03468523458495426,4.759654012819131,0.029874213126619024,1.3958505218227704,0.061691590097926664
+locked_stream,static_dropout_0.02,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,3,3.439823806285858,0.005562641857737584,4.657156705856323,0.020918879251822793,1.217332899570465,0.015357844279503235
+locked_stream,static_dropout_0.06,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,3,3.5540901521841683,0.017924779970058147,4.557474325100581,0.016603375950540906,1.0033841729164124,0.03385453586409587
+locked_stream,static_dropout_0.08,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,3,3.60255632797877,0.00575740505926527,4.5323765849073725,0.009481504316560474,0.9298202569286028,0.012211546495961454
+locked_stream,static_dropout_0.1,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,3,3.6542825972040496,0.030316068353894877,4.516653408606847,0.012694329764198367,0.8623708114027977,0.04203932410029853
+locked_stream,static_dropout_0.14,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.14,0.14,constant,3,3.7183619191249213,0.02332517298978488,4.5008573432763415,0.018900687447980586,0.7824954241514206,0.04221659685113837
+locked_stream,static_dropout_0.18,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.18,0.18,constant,3,3.786718524992466,0.012788547677963957,4.50268988062938,0.019498788384197212,0.7159713556369146,0.02737366667147904
+locked_stream,static_dropout_0.2,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.2,0.2,constant,3,3.811583066980044,0.016414324708578124,4.501583576202393,0.018885880819448517,0.6900005092223486,0.02848926010067624
+locked_stream,static_dropout_0.26,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.26,0.26,constant,3,3.909716250995795,0.029922261968460786,4.527257425089677,0.012874091579002427,0.6175411740938822,0.038926303819109775
+locked_stream,static_dropout_0.3,static,2,1000000,L12_H8_D320,12,8,320,17367040,0.3,0.3,constant,3,3.9773165384928384,0.009735095632867338,4.546837595601876,0.01674020951247983,0.569521057109038,0.02091747765619741
+locked_stream,formula_l12_wikitext103,anchor_decay,3,2000000,L12_H8_D320,12,8,320,17367040,0.3,0.02,log_prefix_anchor,3,3.8132816379268966,0.021180137190910886,4.265036730716626,0.020952651949156013,0.4517550927897294,0.026508733332518825
+locked_stream,static_dropout_0,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,3,3.6350866481661797,0.016285186896951624,4.413571459551652,0.02523774966453558,0.7784848113854727,0.03399640276399005
+locked_stream,static_dropout_0.02,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,3,3.6686666384339333,0.015260894096187612,4.368014050026734,0.027178290223553957,0.6993474115928014,0.02886603382685387
+locked_stream,static_dropout_0.06,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,3,3.7288275261720023,0.02214638532765685,4.304687723517418,0.025605319026104526,0.5758601973454157,0.03169069142172165
+locked_stream,static_dropout_0.08,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,3,3.751750993231932,0.027325045963601875,4.292027606318395,0.022193268917965966,0.540276613086462,0.048123905643236306
+locked_stream,static_dropout_0.1,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,3,3.785589481393496,0.023254215859456147,4.290843218564987,0.024816447573560912,0.505253737171491,0.03955542155547533
+locked_stream,static_dropout_0.14,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.14,0.14,constant,3,3.8283161322275796,0.006946470327794601,4.285862573732932,0.029060804176488816,0.4575464415053527,0.02311386935886081
+locked_stream,static_dropout_0.18,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.18,0.18,constant,3,3.875220308701197,0.011227992287915711,4.2889708404739695,0.027610989551016634,0.4137505317727725,0.019195364852840884
+locked_stream,static_dropout_0.2,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.2,0.2,constant,3,3.8982935870687165,0.011753289842086798,4.297594143698613,0.0215564773236604,0.39930055662989616,0.015284717362206547
+locked_stream,static_dropout_0.26,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.26,0.26,constant,3,3.9674767553806305,0.024908308844425655,4.327832326292992,0.024543452908526633,0.36035557091236115,0.02569548028047761
+locked_stream,static_dropout_0.3,static,3,2000000,L12_H8_D320,12,8,320,17367040,0.3,0.3,constant,3,4.012147528429826,0.02012763898563558,4.349700070917606,0.019107888304911173,0.33755254248778027,0.025128923759912217
+locked_stream,formula_l12_wikitext103,anchor_decay,4,4000000,L12_H8_D320,12,8,320,17367040,0.3,0.02,log_prefix_anchor,3,3.812962586681048,0.025352240743704624,4.083636499941349,0.025765009042835362,0.27067391326030094,0.05009449170635578
+locked_stream,static_dropout_0,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.0,0.0,constant,3,3.7873488490780196,0.026205633809150763,4.179581480721633,0.021736412927032305,0.39223263164361316,0.03552201232307707
+locked_stream,static_dropout_0.02,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.02,0.02,constant,3,3.796787646909555,0.024529861961376253,4.152008826533954,0.019904634278940705,0.35522117962439853,0.04352816945908601
+locked_stream,static_dropout_0.06,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.06,0.06,constant,3,3.817745270828406,0.022916843434376433,4.118092642476161,0.010664122574525757,0.3003473716477553,0.03183478268164145
+locked_stream,static_dropout_0.08,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.08,0.08,constant,3,3.833147312204043,0.02624431695895608,4.110093618432681,0.024734680784555277,0.2769463062286377,0.04976013223808796
+locked_stream,static_dropout_0.1,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.1,0.1,constant,3,3.85314512749513,0.017044693431674442,4.108063347637653,0.025798124596532718,0.25491822014252347,0.03599137702525591
+locked_stream,static_dropout_0.14,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.14,0.14,constant,3,3.8825049375494323,0.020653196329518318,4.126605914284785,0.019834322361978202,0.24410097673535347,0.03709848360553261
+locked_stream,static_dropout_0.18,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.18,0.18,constant,3,3.9089941332737603,0.02100733745422538,4.128837997714679,0.018137202414878383,0.21984386444091797,0.03802532083941853
+locked_stream,static_dropout_0.2,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.2,0.2,constant,3,3.925126865506172,0.0200711438797493,4.138758639494578,0.02359678741043119,0.21363177398840585,0.04159847214281183
+locked_stream,static_dropout_0.26,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.26,0.26,constant,3,3.98883589108785,0.021597891789016974,4.17716755097111,0.020328951540567387,0.18833165988326073,0.03975692522805749
+locked_stream,static_dropout_0.3,static,4,4000000,L12_H8_D320,12,8,320,17367040,0.3,0.3,constant,3,4.025067411363125,0.021409684612983128,4.196084383875132,0.017949538707409805,0.17101697251200676,0.03846054936653385

runs/corpus_holdout_wikitext103_l12/locked_stream/20260528-183213/summary.json

@@ -0,0 +1,1212 @@
+[
+  {
+    "run_mode": "locked_stream",
+    "condition": "formula_l12_wikitext103",
+    "condition_kind": "anchor_decay",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 4.219510187705358,
+    "std_train_eval_loss": 0.009400458871163125,
+    "mean_val_eval_loss": 5.213888607919216,
+    "std_val_eval_loss": 0.02540367830871252,
+    "mean_generalization_gap": 0.9943784202138582,
+    "std_generalization_gap": 0.034794492047624956
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5193328907092414,
+    "std_train_eval_loss": 0.04538277641898639,
+    "mean_val_eval_loss": 5.339607280989488,
+    "std_val_eval_loss": 0.03803900159576214,
+    "mean_generalization_gap": 1.8202743902802467,
+    "std_generalization_gap": 0.07017278062165234
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.591204417248567,
+    "std_train_eval_loss": 0.009258206352807653,
+    "mean_val_eval_loss": 5.27785774320364,
+    "std_val_eval_loss": 0.03450217928391973,
+    "mean_generalization_gap": 1.6866533259550731,
+    "std_generalization_gap": 0.041632245818121416
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.725302261610826,
+    "std_train_eval_loss": 0.008063637282408817,
+    "mean_val_eval_loss": 5.211690753698349,
+    "std_val_eval_loss": 0.02339268745574712,
+    "mean_generalization_gap": 1.4863884920875232,
+    "std_generalization_gap": 0.031438438863035205
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.764298364520073,
+    "std_train_eval_loss": 0.01498956532264627,
+    "mean_val_eval_loss": 5.190002868572871,
+    "std_val_eval_loss": 0.020721921300465723,
+    "mean_generalization_gap": 1.425704504052798,
+    "std_generalization_gap": 0.014047879271355385
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.821372576057911,
+    "std_train_eval_loss": 0.021364905351568726,
+    "mean_val_eval_loss": 5.17405295620362,
+    "std_val_eval_loss": 0.02772884387617372,
+    "mean_generalization_gap": 1.3526803801457088,
+    "std_generalization_gap": 0.04389058367879646
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9113145818312964,
+    "std_train_eval_loss": 0.013390634728517344,
+    "mean_val_eval_loss": 5.169258703788121,
+    "std_val_eval_loss": 0.02886637125607666,
+    "mean_generalization_gap": 1.2579441219568253,
+    "std_generalization_gap": 0.03490443394763099
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 4.007677458226681,
+    "std_train_eval_loss": 0.011584990895915435,
+    "mean_val_eval_loss": 5.163212870558103,
+    "std_val_eval_loss": 0.019303530325978743,
+    "mean_generalization_gap": 1.155535412331422,
+    "std_generalization_gap": 0.021457887456797633
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 4.0440234914422035,
+    "std_train_eval_loss": 0.006683199962460588,
+    "mean_val_eval_loss": 5.173003261288007,
+    "std_val_eval_loss": 0.018770276566139252,
+    "mean_generalization_gap": 1.1289797698458035,
+    "std_generalization_gap": 0.01557915612774489
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 4.161390465994676,
+    "std_train_eval_loss": 0.017199456198519997,
+    "mean_val_eval_loss": 5.197733856737614,
+    "std_val_eval_loss": 0.020026368980316463,
+    "mean_generalization_gap": 1.0363433907429378,
+    "std_generalization_gap": 0.02452558695313914
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 0,
+    "token_limit": 250000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 4.21951021750768,
+    "std_train_eval_loss": 0.00940042069294641,
+    "mean_val_eval_loss": 5.213888632754485,
+    "std_val_eval_loss": 0.02540370559129739,
+    "mean_generalization_gap": 0.9943784152468046,
+    "std_generalization_gap": 0.03479448069461329
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "formula_l12_wikitext103",
+    "condition_kind": "anchor_decay",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.921963239709536,
+    "std_train_eval_loss": 0.012701856161083026,
+    "mean_val_eval_loss": 4.807015274961789,
+    "std_val_eval_loss": 0.03255901757339794,
+    "mean_generalization_gap": 0.8850520352522532,
+    "std_generalization_gap": 0.020904354460357738
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.157259744902452,
+    "std_train_eval_loss": 0.033266149836286125,
+    "mean_val_eval_loss": 5.17509716997544,
+    "std_val_eval_loss": 0.027225613071693588,
+    "mean_generalization_gap": 2.017837425072988,
+    "std_generalization_gap": 0.027271551103611697
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.2696756149331727,
+    "std_train_eval_loss": 0.010229343900978308,
+    "mean_val_eval_loss": 5.048807606101036,
+    "std_val_eval_loss": 0.015919759088441253,
+    "mean_generalization_gap": 1.7791319911678631,
+    "std_generalization_gap": 0.015163975400676688
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.4347175483902297,
+    "std_train_eval_loss": 0.01608227845233568,
+    "mean_val_eval_loss": 4.895065948367119,
+    "std_val_eval_loss": 0.025796230814273288,
+    "mean_generalization_gap": 1.4603483999768894,
+    "std_generalization_gap": 0.016219356742392815
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.49625497063001,
+    "std_train_eval_loss": 0.02296095075813903,
+    "mean_val_eval_loss": 4.8633880987763405,
+    "std_val_eval_loss": 0.03137654346429713,
+    "mean_generalization_gap": 1.3671331281463306,
+    "std_generalization_gap": 0.02205679071854492
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.559451498091221,
+    "std_train_eval_loss": 0.03242743348226214,
+    "mean_val_eval_loss": 4.832371026277542,
+    "std_val_eval_loss": 0.03379083503310632,
+    "mean_generalization_gap": 1.2729195281863213,
+    "std_generalization_gap": 0.004581876102180898
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6746241599321365,
+    "std_train_eval_loss": 0.026787166240812375,
+    "mean_val_eval_loss": 4.804830571015676,
+    "std_val_eval_loss": 0.02283391501446146,
+    "mean_generalization_gap": 1.1302064110835393,
+    "std_generalization_gap": 0.004574911720412448
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.761968764166037,
+    "std_train_eval_loss": 0.030310143678321872,
+    "mean_val_eval_loss": 4.79575655857722,
+    "std_val_eval_loss": 0.02839125459863536,
+    "mean_generalization_gap": 1.0337877944111824,
+    "std_generalization_gap": 0.02395919557232544
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.7916674092411995,
+    "std_train_eval_loss": 0.02457212816143504,
+    "mean_val_eval_loss": 4.78752597173055,
+    "std_val_eval_loss": 0.03269980279988427,
+    "mean_generalization_gap": 0.9958585624893507,
+    "std_generalization_gap": 0.010890773437236328
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9032329618930817,
+    "std_train_eval_loss": 0.03313463368600539,
+    "mean_val_eval_loss": 4.810270811120669,
+    "std_val_eval_loss": 0.029030451515976875,
+    "mean_generalization_gap": 0.9070378492275873,
+    "std_generalization_gap": 0.0218852812037738
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 1,
+    "token_limit": 500000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9801244686047235,
+    "std_train_eval_loss": 0.011984368688744857,
+    "mean_val_eval_loss": 4.826624286671479,
+    "std_val_eval_loss": 0.034146701924452114,
+    "mean_generalization_gap": 0.8464998180667559,
+    "std_generalization_gap": 0.023039413394205236
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "formula_l12_wikitext103",
+    "condition_kind": "anchor_decay",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.836283288896084,
+    "std_train_eval_loss": 0.007165129109779279,
+    "mean_val_eval_loss": 4.4944643552104635,
+    "std_val_eval_loss": 0.01869570507899432,
+    "mean_generalization_gap": 0.6581810663143793,
+    "std_generalization_gap": 0.02324475775596518
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.363803490996361,
+    "std_train_eval_loss": 0.03468523458495426,
+    "mean_val_eval_loss": 4.759654012819131,
+    "std_val_eval_loss": 0.029874213126619024,
+    "mean_generalization_gap": 1.3958505218227704,
+    "std_generalization_gap": 0.061691590097926664
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.439823806285858,
+    "std_train_eval_loss": 0.005562641857737584,
+    "mean_val_eval_loss": 4.657156705856323,
+    "std_val_eval_loss": 0.020918879251822793,
+    "mean_generalization_gap": 1.217332899570465,
+    "std_generalization_gap": 0.015357844279503235
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.5540901521841683,
+    "std_train_eval_loss": 0.017924779970058147,
+    "mean_val_eval_loss": 4.557474325100581,
+    "std_val_eval_loss": 0.016603375950540906,
+    "mean_generalization_gap": 1.0033841729164124,
+    "std_generalization_gap": 0.03385453586409587
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.60255632797877,
+    "std_train_eval_loss": 0.00575740505926527,
+    "mean_val_eval_loss": 4.5323765849073725,
+    "std_val_eval_loss": 0.009481504316560474,
+    "mean_generalization_gap": 0.9298202569286028,
+    "std_generalization_gap": 0.012211546495961454
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6542825972040496,
+    "std_train_eval_loss": 0.030316068353894877,
+    "mean_val_eval_loss": 4.516653408606847,
+    "std_val_eval_loss": 0.012694329764198367,
+    "mean_generalization_gap": 0.8623708114027977,
+    "std_generalization_gap": 0.04203932410029853
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.7183619191249213,
+    "std_train_eval_loss": 0.02332517298978488,
+    "mean_val_eval_loss": 4.5008573432763415,
+    "std_val_eval_loss": 0.018900687447980586,
+    "mean_generalization_gap": 0.7824954241514206,
+    "std_generalization_gap": 0.04221659685113837
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.786718524992466,
+    "std_train_eval_loss": 0.012788547677963957,
+    "mean_val_eval_loss": 4.50268988062938,
+    "std_val_eval_loss": 0.019498788384197212,
+    "mean_generalization_gap": 0.7159713556369146,
+    "std_generalization_gap": 0.02737366667147904
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.811583066980044,
+    "std_train_eval_loss": 0.016414324708578124,
+    "mean_val_eval_loss": 4.501583576202393,
+    "std_val_eval_loss": 0.018885880819448517,
+    "mean_generalization_gap": 0.6900005092223486,
+    "std_generalization_gap": 0.02848926010067624
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.909716250995795,
+    "std_train_eval_loss": 0.029922261968460786,
+    "mean_val_eval_loss": 4.527257425089677,
+    "std_val_eval_loss": 0.012874091579002427,
+    "mean_generalization_gap": 0.6175411740938822,
+    "std_generalization_gap": 0.038926303819109775
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 2,
+    "token_limit": 1000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9773165384928384,
+    "std_train_eval_loss": 0.009735095632867338,
+    "mean_val_eval_loss": 4.546837595601876,
+    "std_val_eval_loss": 0.01674020951247983,
+    "mean_generalization_gap": 0.569521057109038,
+    "std_generalization_gap": 0.02091747765619741
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "formula_l12_wikitext103",
+    "condition_kind": "anchor_decay",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.8132816379268966,
+    "std_train_eval_loss": 0.021180137190910886,
+    "mean_val_eval_loss": 4.265036730716626,
+    "std_val_eval_loss": 0.020952651949156013,
+    "mean_generalization_gap": 0.4517550927897294,
+    "std_generalization_gap": 0.026508733332518825
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6350866481661797,
+    "std_train_eval_loss": 0.016285186896951624,
+    "mean_val_eval_loss": 4.413571459551652,
+    "std_val_eval_loss": 0.02523774966453558,
+    "mean_generalization_gap": 0.7784848113854727,
+    "std_generalization_gap": 0.03399640276399005
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.6686666384339333,
+    "std_train_eval_loss": 0.015260894096187612,
+    "mean_val_eval_loss": 4.368014050026734,
+    "std_val_eval_loss": 0.027178290223553957,
+    "mean_generalization_gap": 0.6993474115928014,
+    "std_generalization_gap": 0.02886603382685387
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.7288275261720023,
+    "std_train_eval_loss": 0.02214638532765685,
+    "mean_val_eval_loss": 4.304687723517418,
+    "std_val_eval_loss": 0.025605319026104526,
+    "mean_generalization_gap": 0.5758601973454157,
+    "std_generalization_gap": 0.03169069142172165
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.751750993231932,
+    "std_train_eval_loss": 0.027325045963601875,
+    "mean_val_eval_loss": 4.292027606318395,
+    "std_val_eval_loss": 0.022193268917965966,
+    "mean_generalization_gap": 0.540276613086462,
+    "std_generalization_gap": 0.048123905643236306
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.785589481393496,
+    "std_train_eval_loss": 0.023254215859456147,
+    "mean_val_eval_loss": 4.290843218564987,
+    "std_val_eval_loss": 0.024816447573560912,
+    "mean_generalization_gap": 0.505253737171491,
+    "std_generalization_gap": 0.03955542155547533
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.8283161322275796,
+    "std_train_eval_loss": 0.006946470327794601,
+    "mean_val_eval_loss": 4.285862573732932,
+    "std_val_eval_loss": 0.029060804176488816,
+    "mean_generalization_gap": 0.4575464415053527,
+    "std_generalization_gap": 0.02311386935886081
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.875220308701197,
+    "std_train_eval_loss": 0.011227992287915711,
+    "mean_val_eval_loss": 4.2889708404739695,
+    "std_val_eval_loss": 0.027610989551016634,
+    "mean_generalization_gap": 0.4137505317727725,
+    "std_generalization_gap": 0.019195364852840884
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.8982935870687165,
+    "std_train_eval_loss": 0.011753289842086798,
+    "mean_val_eval_loss": 4.297594143698613,
+    "std_val_eval_loss": 0.0215564773236604,
+    "mean_generalization_gap": 0.39930055662989616,
+    "std_generalization_gap": 0.015284717362206547
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9674767553806305,
+    "std_train_eval_loss": 0.024908308844425655,
+    "mean_val_eval_loss": 4.327832326292992,
+    "std_val_eval_loss": 0.024543452908526633,
+    "mean_generalization_gap": 0.36035557091236115,
+    "std_generalization_gap": 0.02569548028047761
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 3,
+    "token_limit": 2000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 4.012147528429826,
+    "std_train_eval_loss": 0.02012763898563558,
+    "mean_val_eval_loss": 4.349700070917606,
+    "std_val_eval_loss": 0.019107888304911173,
+    "mean_generalization_gap": 0.33755254248778027,
+    "std_generalization_gap": 0.025128923759912217
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "formula_l12_wikitext103",
+    "condition_kind": "anchor_decay",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.02,
+    "dropout_schedule": "log_prefix_anchor",
+    "n": 3,
+    "mean_train_eval_loss": 3.812962586681048,
+    "std_train_eval_loss": 0.025352240743704624,
+    "mean_val_eval_loss": 4.083636499941349,
+    "std_val_eval_loss": 0.025765009042835362,
+    "mean_generalization_gap": 0.27067391326030094,
+    "std_generalization_gap": 0.05009449170635578
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.0,
+    "dropout_final": 0.0,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.7873488490780196,
+    "std_train_eval_loss": 0.026205633809150763,
+    "mean_val_eval_loss": 4.179581480721633,
+    "std_val_eval_loss": 0.021736412927032305,
+    "mean_generalization_gap": 0.39223263164361316,
+    "std_generalization_gap": 0.03552201232307707
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.02",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.02,
+    "dropout_final": 0.02,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.796787646909555,
+    "std_train_eval_loss": 0.024529861961376253,
+    "mean_val_eval_loss": 4.152008826533954,
+    "std_val_eval_loss": 0.019904634278940705,
+    "mean_generalization_gap": 0.35522117962439853,
+    "std_generalization_gap": 0.04352816945908601
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.06",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.06,
+    "dropout_final": 0.06,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.817745270828406,
+    "std_train_eval_loss": 0.022916843434376433,
+    "mean_val_eval_loss": 4.118092642476161,
+    "std_val_eval_loss": 0.010664122574525757,
+    "mean_generalization_gap": 0.3003473716477553,
+    "std_generalization_gap": 0.03183478268164145
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.08",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.08,
+    "dropout_final": 0.08,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.833147312204043,
+    "std_train_eval_loss": 0.02624431695895608,
+    "mean_val_eval_loss": 4.110093618432681,
+    "std_val_eval_loss": 0.024734680784555277,
+    "mean_generalization_gap": 0.2769463062286377,
+    "std_generalization_gap": 0.04976013223808796
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.1",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.1,
+    "dropout_final": 0.1,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.85314512749513,
+    "std_train_eval_loss": 0.017044693431674442,
+    "mean_val_eval_loss": 4.108063347637653,
+    "std_val_eval_loss": 0.025798124596532718,
+    "mean_generalization_gap": 0.25491822014252347,
+    "std_generalization_gap": 0.03599137702525591
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.14",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.14,
+    "dropout_final": 0.14,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.8825049375494323,
+    "std_train_eval_loss": 0.020653196329518318,
+    "mean_val_eval_loss": 4.126605914284785,
+    "std_val_eval_loss": 0.019834322361978202,
+    "mean_generalization_gap": 0.24410097673535347,
+    "std_generalization_gap": 0.03709848360553261
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.18",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.18,
+    "dropout_final": 0.18,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.9089941332737603,
+    "std_train_eval_loss": 0.02100733745422538,
+    "mean_val_eval_loss": 4.128837997714679,
+    "std_val_eval_loss": 0.018137202414878383,
+    "mean_generalization_gap": 0.21984386444091797,
+    "std_generalization_gap": 0.03802532083941853
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.2",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.2,
+    "dropout_final": 0.2,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.925126865506172,
+    "std_train_eval_loss": 0.0200711438797493,
+    "mean_val_eval_loss": 4.138758639494578,
+    "std_val_eval_loss": 0.02359678741043119,
+    "mean_generalization_gap": 0.21363177398840585,
+    "std_generalization_gap": 0.04159847214281183
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.26",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.26,
+    "dropout_final": 0.26,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 3.98883589108785,
+    "std_train_eval_loss": 0.021597891789016974,
+    "mean_val_eval_loss": 4.17716755097111,
+    "std_val_eval_loss": 0.020328951540567387,
+    "mean_generalization_gap": 0.18833165988326073,
+    "std_generalization_gap": 0.03975692522805749
+  },
+  {
+    "run_mode": "locked_stream",
+    "condition": "static_dropout_0.3",
+    "condition_kind": "static",
+    "stage": 4,
+    "token_limit": 4000000,
+    "model_name": "L12_H8_D320",
+    "n_layer": 12,
+    "n_head": 8,
+    "n_embd": 320,
+    "parameters": 17367040,
+    "dropout_initial": 0.3,
+    "dropout_final": 0.3,
+    "dropout_schedule": "constant",
+    "n": 3,
+    "mean_train_eval_loss": 4.025067411363125,
+    "std_train_eval_loss": 0.021409684612983128,
+    "mean_val_eval_loss": 4.196084383875132,
+    "std_val_eval_loss": 0.017949538707409805,
+    "mean_generalization_gap": 0.17101697251200676,
+    "std_generalization_gap": 0.03846054936653385
+  }
+]

runs/corpus_holdout_wikitext103_l12/locked_stream/20260528-183213/trace.jsonl

@@ -0,0 +1,330 @@
+{"condition": "formula_l12_wikitext103", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.10919189453125}
+{"condition": "formula_l12_wikitext103", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.3267059326171875}
+{"condition": "formula_l12_wikitext103", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.150979042053223}
+{"condition": "formula_l12_wikitext103", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.038363456726074}
+{"condition": "formula_l12_wikitext103", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.993682861328125}
+{"condition": "formula_l12_wikitext103", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.025256633758545}
+{"condition": "formula_l12_wikitext103", "dropout": 0.09, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.174747943878174}
+{"condition": "formula_l12_wikitext103", "dropout": 0.09, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.870220184326172}
+{"condition": "formula_l12_wikitext103", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.9933338165283203}
+{"condition": "formula_l12_wikitext103", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.751016855239868}
+{"condition": "formula_l12_wikitext103", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.02793025970459}
+{"condition": "formula_l12_wikitext103", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.466760635375977}
+{"condition": "formula_l12_wikitext103", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.282154083251953}
+{"condition": "formula_l12_wikitext103", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.091670036315918}
+{"condition": "formula_l12_wikitext103", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.201129913330078}
+{"condition": "formula_l12_wikitext103", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.089809894561768}
+{"condition": "formula_l12_wikitext103", "dropout": 0.09, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.933499813079834}
+{"condition": "formula_l12_wikitext103", "dropout": 0.09, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.8695573806762695}
+{"condition": "formula_l12_wikitext103", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.0093183517456055}
+{"condition": "formula_l12_wikitext103", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.987300395965576}
+{"condition": "formula_l12_wikitext103", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.0945048332214355}
+{"condition": "formula_l12_wikitext103", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.291036605834961}
+{"condition": "formula_l12_wikitext103", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.300796985626221}
+{"condition": "formula_l12_wikitext103", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.205198287963867}
+{"condition": "formula_l12_wikitext103", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.238568305969238}
+{"condition": "formula_l12_wikitext103", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.775209426879883}
+{"condition": "formula_l12_wikitext103", "dropout": 0.09, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.786513328552246}
+{"condition": "formula_l12_wikitext103", "dropout": 0.09, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.115474700927734}
+{"condition": "formula_l12_wikitext103", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.8349270820617676}
+{"condition": "formula_l12_wikitext103", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.794011116027832}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.586461067199707}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.4980170726776123}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.706185817718506}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.010756015777588}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.846466302871704}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.2843170166015625}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.748643398284912}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.6820337772369385}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.5873708724975586}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.545558452606201}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.423453330993652}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.617307186126709}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.6751105785369873}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 2.825775146484375}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.784053325653076}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.3288044929504395}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.545459747314453}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.469001293182373}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.6900553703308105}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.053542137145996}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.557775974273682}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.530925750732422}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.377972364425659}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.2348737716674805}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.526865005493164}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.300647497177124}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.4643568992614746}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.544159412384033}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.6731104850769043}
+{"condition": "static_dropout_0", "dropout": 0.0, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.626936435699463}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.596406936645508}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.5398497581481934}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.487887144088745}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.1747987270355225}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.5900020599365234}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.4777474403381348}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.7301249504089355}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.790534019470215}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.079158782958984}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.661900520324707}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.517829418182373}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.5424184799194336}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.4532995223999023}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.393620252609253}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.6632490158081055}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.5351672172546387}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.7610929012298584}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.4371466636657715}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.9270126819610596}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.961052656173706}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.554469585418701}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.6152591705322266}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.6096372604370117}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.1891748905181885}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.523820161819458}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.5947442054748535}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.066094398498535}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.6390342712402344}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.002377510070801}
+{"condition": "static_dropout_0.02", "dropout": 0.02, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.785369873046875}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.710752964019775}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.9541187286376953}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.050755977630615}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.6150665283203125}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.892446756362915}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.7684245109558105}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.7994470596313477}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.783784866333008}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.948446750640869}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.7330846786499023}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.726563453674316}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.838132381439209}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.9320147037506104}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.3768904209136963}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.9188880920410156}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.764596939086914}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.872814655303955}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.8399839401245117}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.9642767906188965}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.817599296569824}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.6344380378723145}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.6540794372558594}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.642949104309082}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.537709951400757}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.9372754096984863}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.012999534606934}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.8797926902770996}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.944365978240967}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.346187114715576}
+{"condition": "static_dropout_0.06", "dropout": 0.06, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.762101888656616}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.694372177124023}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.008996963500977}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.710146427154541}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.538743495941162}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.9808402061462402}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.636242389678955}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.011699676513672}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.6634960174560547}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.111110687255859}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.8889031410217285}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.579659461975098}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.9213414192199707}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.2153167724609375}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.624877691268921}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.9966182708740234}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.9975638389587402}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.217199802398682}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.8261024951934814}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.059680938720703}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.039473056793213}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.775473117828369}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.8681042194366455}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.858368158340454}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.6299901008605957}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.762925148010254}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.398933172225952}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.915220260620117}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.801541805267334}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.091124534606934}
+{"condition": "static_dropout_0.08", "dropout": 0.08, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.713860034942627}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.502629280090332}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.026566028594971}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.839831829071045}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.62257719039917}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.067403793334961}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.841139793395996}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.967017889022827}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.688873291015625}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.087314605712891}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.86462140083313}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.7392988204956055}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.893677234649658}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.870542526245117}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.8797426223754883}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.9220707416534424}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.793069362640381}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.9290614128112793}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.9442577362060547}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.7859325408935547}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.145358085632324}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.739326477050781}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.062891960144043}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.149168014526367}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.7364869117736816}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.7814455032348633}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.7253265380859375}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.9168343544006348}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.791567325592041}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.778740406036377}
+{"condition": "static_dropout_0.1", "dropout": 0.1, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.776935577392578}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.7961297035217285}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 3.8898675441741943}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.075655937194824}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.756572723388672}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.029130935668945}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.773857831954956}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.1858930587768555}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.8487486839294434}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.9703285694122314}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.8854053020477295}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.818635940551758}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.108892917633057}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.21990966796875}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.81386137008667}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.809004545211792}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.7727158069610596}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.847060441970825}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.098654747009277}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.098649024963379}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.016416549682617}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.850597381591797}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.025087356567383}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.976827621459961}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.8091471195220947}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.069204807281494}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.657958507537842}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 3.9937562942504883}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.972700357437134}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.0872650146484375}
+{"condition": "static_dropout_0.14", "dropout": 0.14, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.936558246612549}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.747689247131348}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.080568313598633}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.200558662414551}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.979367971420288}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.182689666748047}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.7715864181518555}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.135184288024902}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.08739709854126}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.0746169090271}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.0843729972839355}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.756750106811523}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.258420467376709}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.110930919647217}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.029372215270996}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 3.872436046600342}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.009469509124756}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.128143310546875}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.8418898582458496}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.1407575607299805}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.095191955566406}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.976916790008545}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.044127464294434}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.275222301483154}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.9059031009674072}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.260745525360107}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.906364917755127}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.124902248382568}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.0458292961120605}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.184656143188477}
+{"condition": "static_dropout_0.18", "dropout": 0.18, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.018331050872803}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.769725799560547}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.0768561363220215}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.300281047821045}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.8637545108795166}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.150639533996582}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.981992244720459}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.188844203948975}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.9941892623901367}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.077834129333496}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.117517471313477}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.81527042388916}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.054525375366211}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.37611198425293}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.8969826698303223}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.113924980163574}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.9721953868865967}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.022497177124023}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.092113494873047}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.1637492179870605}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.9015634059906006}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.844550132751465}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.1090593338012695}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 3.9657397270202637}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.7497055530548096}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.205092430114746}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.052699089050293}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.3501362800598145}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.8875317573547363}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 3.977046251296997}
+{"condition": "static_dropout_0.2", "dropout": 0.2, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 3.952976703643799}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.029112815856934}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.155546188354492}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.2847490310668945}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.293614387512207}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.426083087921143}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.8696696758270264}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.210818290710449}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 3.7679266929626465}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.164118766784668}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.19427490234375}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.084468841552734}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.303183555603027}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.257471084594727}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 3.9919092655181885}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.075634956359863}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.116327285766602}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.130469799041748}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.175629615783691}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.153221130371094}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.002399444580078}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 4.839096546173096}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.228353500366211}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.0907487869262695}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.135189056396484}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.315141677856445}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.052082061767578}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.100449562072754}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.0784711837768555}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.183749198913574}
+{"condition": "static_dropout_0.26", "dropout": 0.26, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.022092819213867}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.10919189453125}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.3267059326171875}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.192840576171875}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.094118118286133}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.15631628036499}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.1938676834106445}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.386846542358398}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.115250587463379}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.304389476776123}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 1, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.113180160522461}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.02793025970459}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.466761112213135}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.32354736328125}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.149529933929443}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.344442367553711}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 4.2312116622924805}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.160443305969238}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.132419109344482}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.286291599273682}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 2, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.308834552764893}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 500, "steps": 1000, "token_limit": 250000, "tokens_seen": 1024000, "train_batch_loss": 5.0945048332214355}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 0, "step": 1000, "steps": 1000, "token_limit": 250000, "tokens_seen": 2048000, "train_batch_loss": 4.291036605834961}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 500, "steps": 1000, "token_limit": 500000, "tokens_seen": 3072000, "train_batch_loss": 4.351649761199951}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 1, "step": 1000, "steps": 1000, "token_limit": 500000, "tokens_seen": 4096000, "train_batch_loss": 4.264379024505615}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 500, "steps": 1000, "token_limit": 1000000, "tokens_seen": 5120000, "train_batch_loss": 4.378478527069092}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 2, "step": 1000, "steps": 1000, "token_limit": 1000000, "tokens_seen": 6144000, "train_batch_loss": 3.95513916015625}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 500, "steps": 1000, "token_limit": 2000000, "tokens_seen": 7168000, "train_batch_loss": 4.055840492248535}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 3, "step": 1000, "steps": 1000, "token_limit": 2000000, "tokens_seen": 8192000, "train_batch_loss": 4.364241600036621}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 500, "steps": 1000, "token_limit": 4000000, "tokens_seen": 9216000, "train_batch_loss": 4.2040815353393555}
+{"condition": "static_dropout_0.3", "dropout": 0.3, "event": "train_step", "model_name": "L12_H8_D320", "run_mode": "locked_stream", "seed": 3, "stage": 4, "step": 1000, "steps": 1000, "token_limit": 4000000, "tokens_seen": 10240000, "train_batch_loss": 4.1234211921691895}