python-124m-muon-5b-20261021-v1: 50065393 training targets

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+training-curves.png filter=lfs diff=lfs merge=lfs -text
+validation-curves.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,71 @@
+---
+language: en
+library_name: pytorch
+tags:
+- causal-lm
+- python
+- from-scratch
+- muon
+- experimental
+---
+# Simple LLM Python 124M / 5B
+
+Run: `python-124m-muon-5b-20261021-v1`. This is an **intermediate** base-model checkpoint, freshly initialized.
+Processed **50,065,393 nonpadding training targets** in **549 optimizer updates**.
+The exact target is 5,000,000,000. This model is not instruction-tuned.
+
+124,389,632 parameters; 14 layers, width 768, 12 attention heads, RMSNorm,
+RoPE, SwiGLU, tied embeddings and the existing frozen 32K tokenizer. Training
+uses eight H100s, 8K context, cropped padding, cached RoPE, torch.compile,
+microbatch 2 and global batch 32 document-local windows. Hidden matrices use
+the previously tested five-step Muon (LR .02, momentum .95); tied embeddings,
+norms and biases use auxiliary AdamW (LR .0003, betas .9/.95). Weight decay .1,
+gradient clipping 1, 20M-target warmup and cosine decay to 10%.
+
+## Data
+Target proportions: 65% Python implementation, 15% Python tests/examples,
+10% Python programming documentation (including explanatory technical Q&A),
+5% CS/math, 3% general educational prose, 2% licensed/public-domain books.
+Complete recent collected repository snapshots are preferred. Supplemental
+sources include pinned CodeParrot pools, Common Pile StackExchange, FineWeb-Edu,
+FineMath and public-domain Gutenberg material. All code allocations are Python.
+The capacity table and exact per-category targets are in `capacity.json` and
+`data-manifest.json`; one pass is preferred and two is the maximum. Public source
+pins, aggregate collected-source capacity and filtering limitations are recorded
+in `data-provenance.json`.
+
+Repository/thread splits precede windows. Exact held-out hashes and repositories
+are excluded across sources. Exact deduplication and repository caps are applied;
+near clones can remain. Path/content filtering and license metadata are imperfect.
+Some supplements are older and may contain outdated practices or errors.
+Private documents and sensitive source-level provenance are not published.
+StackExchange sources carry per-document CC-BY-SA attribution in the private
+provenance ledger; Common Pile source revision is
+`5ec3aa2e65b7c61b7c18f1ff18ccbc4d3279a23c`.
+
+## Evaluation
+Held-out category losses, all fixed greedy/sampled Python generations, syntax,
+repetition, length/EOS metrics and hidden-test execution results are included.
+Function evaluation uses 32 reproducibly selected MBPP sanitized test tasks
+(CC-BY-4.0, Google Research), revision
+`4bb6404fdc6cacfda99d4ac4205087b89d32030c`. Tests are omitted from model prompts.
+Exact reference hashes are excluded from training; no exhaustive benchmark
+contamination audit or benchmark-clean claim is made. Generated code executes
+in isolated network-blocked, credential-free sandboxes with resource limits.
+Low early accuracy alone does not stop training.
+
+![Training curves](training-curves.png)
+![Validation curves](validation-curves.png)
+
+All fixed-suite generations are retained without selection in `evaluations/`.
+Each Hub commit preserves an inference checkpoint. Optimizer, RNG and data cursors
+remain in private Modal training checkpoints for exact continuation.
+Weight hashes are in `provenance.json`; runtime/cost figures are explicitly estimates.
+Use the bundled custom PyTorch model, not Transformers AutoModel:
+
+```bash
+pip install -r requirements.txt
+python generate.py --prompt 'def add(a, b):' --tokens 128
+```
+
+Code and execution evidence: https://github.com/koogle/lm-exploration/pull/30 .

capacity.json

@@ -0,0 +1,154 @@
+{
+  "stage": "ready_to_freeze",
+  "updated_unix": 1792550906.6174269,
+  "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+  "categories": {
+    "python_implementation": {
+      "target": 3250000000,
+      "unique_train_targets": 3344840434,
+      "validation_targets": 202402128,
+      "required_passes": 0.9716457523545949,
+      "two_pass_shortfall": 0,
+      "one_pass_shortfall": 0
+    },
+    "python_tests_examples": {
+      "target": 750000000,
+      "unique_train_targets": 1455959517,
+      "validation_targets": 88631252,
+      "required_passes": 0.5151242127565282,
+      "two_pass_shortfall": 0,
+      "one_pass_shortfall": 0
+    },
+    "python_documentation": {
+      "target": 500000000,
+      "unique_train_targets": 638756689,
+      "validation_targets": 32309371,
+      "required_passes": 0.782770667783958,
+      "two_pass_shortfall": 0,
+      "one_pass_shortfall": 0
+    },
+    "cs_math": {
+      "target": 250000000,
+      "unique_train_targets": 255001393,
+      "validation_targets": 31729746,
+      "required_passes": 0.9803868012595524,
+      "two_pass_shortfall": 0,
+      "one_pass_shortfall": 0
+    },
+    "general_education": {
+      "target": 150000000,
+      "unique_train_targets": 153000047,
+      "validation_targets": 59391033,
+      "required_passes": 0.9803918556966195,
+      "two_pass_shortfall": 0,
+      "one_pass_shortfall": 0
+    },
+    "books": {
+      "target": 100000000,
+      "unique_train_targets": 102061503,
+      "validation_targets": 14413928,
+      "required_passes": 0.9798013654570618,
+      "two_pass_shortfall": 0,
+      "one_pass_shortfall": 0
+    }
+  },
+  "capacity_sufficient": true,
+  "rejections": {
+    "repository_license_unresolved": 5,
+    "generated": 37974,
+    "not_python_material": 15299,
+    "size": 1236,
+    "notice_or_changelog": 121,
+    "minified_or_long_line": 171,
+    "exact_duplicate": 910960,
+    "malformed_or_size": 19,
+    "boilerplate": 566,
+    "excluded_path": 76203,
+    "repository_one_percent_cap": 1372,
+    "reserved_or_empty_encoding": 4,
+    "invalid_python3": 175412,
+    "non_utf8": 17,
+    "reserved_test_record": 203318,
+    "heldout_hash_exclusion": 273597,
+    "older_collected_repository_version": 1440,
+    "heldout_repository_exclusion": 287838,
+    "placeholder_test_scaffold": 23,
+    "documentation_references_heldout_repository": 2926,
+    "python2_specific_documentation": 5529
+  },
+  "repository_count": 58,
+  "by_origin": [
+    {
+      "category": "books",
+      "origin": "existing_pinned_pool",
+      "documents": 923,
+      "unique_train_targets": 102061503
+    },
+    {
+      "category": "cs_math",
+      "origin": "existing_pinned_pool",
+      "documents": 153044,
+      "unique_train_targets": 255001393
+    },
+    {
+      "category": "general_education",
+      "origin": "existing_pinned_pool",
+      "documents": 113805,
+      "unique_train_targets": 153000047
+    },
+    {
+      "category": "python_documentation",
+      "origin": "collected_git",
+      "documents": 8796,
+      "unique_train_targets": 27901636
+    },
+    {
+      "category": "python_documentation",
+      "origin": "existing_pinned_pool",
+      "documents": 55,
+      "unique_train_targets": 1975348
+    },
+    {
+      "category": "python_documentation",
+      "origin": "python_explanatory_qa",
+      "documents": 620563,
+      "unique_train_targets": 608879705
+    },
+    {
+      "category": "python_implementation",
+      "origin": "collected_git",
+      "documents": 10392,
+      "unique_train_targets": 41406577
+    },
+    {
+      "category": "python_implementation",
+      "origin": "existing_pinned_pool",
+      "documents": 708699,
+      "unique_train_targets": 1422349105
+    },
+    {
+      "category": "python_implementation",
+      "origin": "supplemental_python_file",
+      "documents": 870723,
+      "unique_train_targets": 1881084752
+    },
+    {
+      "category": "python_tests_examples",
+      "origin": "collected_git",
+      "documents": 11828,
+      "unique_train_targets": 38755717
+    },
+    {
+      "category": "python_tests_examples",
+      "origin": "existing_pinned_pool",
+      "documents": 274648,
+      "unique_train_targets": 680803158
+    },
+    {
+      "category": "python_tests_examples",
+      "origin": "supplemental_python_file",
+      "documents": 281242,
+      "unique_train_targets": 736400642
+    }
+  ]
+}

capacity.md

@@ -0,0 +1,8 @@
+| Category | Target | Unique train targets | Validation targets | Required passes | Two-pass shortfall |
+|---|---:|---:|---:|---:|---:|
+| python_implementation | 3,250,000,000 | 3,344,840,434 | 202,402,128 | 0.972 | 0 |
+| python_tests_examples | 750,000,000 | 1,455,959,517 | 88,631,252 | 0.515 | 0 |
+| python_documentation | 500,000,000 | 638,756,689 | 32,309,371 | 0.783 | 0 |
+| cs_math | 250,000,000 | 255,001,393 | 31,729,746 | 0.980 | 0 |
+| general_education | 150,000,000 | 153,000,047 | 59,391,033 | 0.980 | 0 |
+| books | 100,000,000 | 102,061,503 | 14,413,928 | 0.980 | 0 |

config.json

@@ -0,0 +1,8 @@
+{
+  "n_layer": 14,
+  "n_head": 12,
+  "n_embd": 768,
+  "vocab_size": 32768,
+  "block_size": 8192,
+  "architecture_version": 2
+}

data-manifest.json

@@ -0,0 +1,106 @@
+{
+  "format": "scholarly-raw-v1",
+  "training_ready": true,
+  "max_passes": 2,
+  "experiment": "python-124m-muon-5b",
+  "prefer_collected": true,
+  "target_tokens": 5000000000,
+  "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+  "database": {
+    "path": "raw.sqlite",
+    "sha256": "3942c0e7edccc426cbf7f0c3b9d69144f0511960a36e6615151645c38da41565"
+  },
+  "sources": {
+    "python_implementation": {
+      "weight_percent": 65,
+      "target_tokens": 3250000000,
+      "max_passes": 1,
+      "splits": {
+        "train": {
+          "loss_tokens": 3344840434
+        },
+        "validation": {
+          "loss_tokens": 202402128
+        }
+      }
+    },
+    "python_tests_examples": {
+      "weight_percent": 15,
+      "target_tokens": 750000000,
+      "max_passes": 1,
+      "splits": {
+        "train": {
+          "loss_tokens": 1455959517
+        },
+        "validation": {
+          "loss_tokens": 88631252
+        }
+      }
+    },
+    "python_documentation": {
+      "weight_percent": 10,
+      "target_tokens": 500000000,
+      "max_passes": 1,
+      "splits": {
+        "train": {
+          "loss_tokens": 638756689
+        },
+        "validation": {
+          "loss_tokens": 32309371
+        }
+      }
+    },
+    "cs_math": {
+      "weight_percent": 5,
+      "target_tokens": 250000000,
+      "max_passes": 1,
+      "splits": {
+        "train": {
+          "loss_tokens": 255001393
+        },
+        "validation": {
+          "loss_tokens": 31729746
+        }
+      }
+    },
+    "general_education": {
+      "weight_percent": 3,
+      "target_tokens": 150000000,
+      "max_passes": 1,
+      "splits": {
+        "train": {
+          "loss_tokens": 153000047
+        },
+        "validation": {
+          "loss_tokens": 59391033
+        }
+      }
+    },
+    "books": {
+      "weight_percent": 2,
+      "target_tokens": 100000000,
+      "max_passes": 1,
+      "splits": {
+        "train": {
+          "loss_tokens": 102061503
+        },
+        "validation": {
+          "loss_tokens": 14413928
+        }
+      }
+    }
+  },
+  "heldout": {
+    "path": "heldout.jsonl.gz",
+    "sha256": "58ed450084b99893f1e1b09d8f7556653ab36a66db1eb41644e8f92029490325",
+    "windows": 192,
+    "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9"
+  },
+  "limitations": [
+    "Exact file/document deduplication; near clones can remain",
+    "Path and content heuristics are imperfect",
+    "Documentation includes explanatory Python technical Q&A",
+    "Collected recent repositories preferred; supplements include older material",
+    "No benchmark-contamination claim; detailed source provenance remains private"
+  ]
+}

data-provenance.json

@@ -0,0 +1,104 @@
+{
+  "experiment": "python-124m-muon-5b",
+  "public_source_snapshots": [
+    {
+      "dataset": "codeparrot/codeparrot-clean-train",
+      "revision": "3e6ab65f2864931e041f6a82db9b5a6ec2b71ab4",
+      "representation": "complete Python files; dataset snapshot is pinned, individual upstream Git commits are not supplied"
+    },
+    {
+      "dataset": "common-pile/stackexchange",
+      "revision": "5ec3aa2e65b7c61b7c18f1ff18ccbc4d3279a23c",
+      "representation": "complete explanatory Python Stack Overflow threads with per-record CC-BY-SA attribution"
+    }
+  ],
+  "collected_repositories": "Complete pinned Git blobs from the user's collected snapshots; prefer recent snapshots and consume retained collected files first within each category. Repository identities, commits and notices remain in the private ledger.",
+  "existing_source_pools": "Pinned prepared CodeParrot, FineWeb-Edu, FineMath, Gutenberg and programming-documentation pools. Original database hashes, upstream record metadata and license evidence remain in the private ledger.",
+  "filters": [
+    "Exclusive implementation, tests/examples, Python documentation and supporting-text categories",
+    "Exclude vendored, generated, minified, boilerplate-heavy and malformed files; preserve accepted text unchanged",
+    "Python 3.12 AST validation; newer syntax can be conservatively excluded",
+    "Exact file/document deduplication across sources; near clones can remain",
+    "Per-repository/category cap of 1 percent of that category's training budget",
+    "Repository/thread holdouts before window construction; cross-source heldout repository and exact document-hash exclusions",
+    "Exclude exact MBPP reference hashes; no exhaustive benchmark-contamination claim"
+  ],
+  "token_accounting": "Nonpadding next-token labels, including each document's EOS target. Existing frozen tokenizer; exact six-category budgets; prefer one pass, never exceed two.",
+  "limitations": "Supplemental code and technical Q&A include older material. Heuristic filters and upstream licensing metadata are imperfect. Detailed provenance is retained privately; this file contains only aggregate and public-source information.",
+  "corpus_database_sha256": "3942c0e7edccc426cbf7f0c3b9d69144f0511960a36e6615151645c38da41565",
+  "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+  "capacity_by_origin": [
+    {
+      "category": "books",
+      "origin": "existing_pinned_pool",
+      "documents": 923,
+      "unique_train_targets": 102061503
+    },
+    {
+      "category": "cs_math",
+      "origin": "existing_pinned_pool",
+      "documents": 153044,
+      "unique_train_targets": 255001393
+    },
+    {
+      "category": "general_education",
+      "origin": "existing_pinned_pool",
+      "documents": 113805,
+      "unique_train_targets": 153000047
+    },
+    {
+      "category": "python_documentation",
+      "origin": "collected_git",
+      "documents": 8796,
+      "unique_train_targets": 27901636
+    },
+    {
+      "category": "python_documentation",
+      "origin": "existing_pinned_pool",
+      "documents": 55,
+      "unique_train_targets": 1975348
+    },
+    {
+      "category": "python_documentation",
+      "origin": "python_explanatory_qa",
+      "documents": 620563,
+      "unique_train_targets": 608879705
+    },
+    {
+      "category": "python_implementation",
+      "origin": "collected_git",
+      "documents": 10392,
+      "unique_train_targets": 41406577
+    },
+    {
+      "category": "python_implementation",
+      "origin": "existing_pinned_pool",
+      "documents": 708699,
+      "unique_train_targets": 1422349105
+    },
+    {
+      "category": "python_implementation",
+      "origin": "supplemental_python_file",
+      "documents": 870723,
+      "unique_train_targets": 1881084752
+    },
+    {
+      "category": "python_tests_examples",
+      "origin": "collected_git",
+      "documents": 11828,
+      "unique_train_targets": 38755717
+    },
+    {
+      "category": "python_tests_examples",
+      "origin": "existing_pinned_pool",
+      "documents": 274648,
+      "unique_train_targets": 680803158
+    },
+    {
+      "category": "python_tests_examples",
+      "origin": "supplemental_python_file",
+      "documents": 281242,
+      "unique_train_targets": 736400642
+    }
+  ]
+}

execution-harness.json

@@ -0,0 +1,369 @@
+{
+  "reference_harness": true,
+  "execution": [
+    {
+      "task_id": 20,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ71RN4QV6MRSND68C55PG",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 307,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ71JXJBAF4EAD0B8BJMWM",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 226,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ71NFAA1WFH8JJGBQ61C3",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 415,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ729EHQR68Y3WDS52A7KH",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 230,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ7250MQ00RRXR689TA84J",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 394,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ72EK9NNG5HD1DYJ0K8CW",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 429,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ72BKY2A8RNRRW0WF9A7C",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 108,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ72PYNRXBZ0EG7Z19JZZQ",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 397,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ72T6DF5PMGJJ57EJ26XE",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 120,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ72WH6H92HH0G68VB39JW",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 451,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ732M07ZYWK6EVT0FVQ6V",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 454,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ73E6PDK4BF7DD4W2VFAA",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 472,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ73B7TKG8K0ZSG5H4QJRB",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 431,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ73G81G4JTH1M768BCA2J",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 257,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ73TQ9EWGKAS7KXSV2C77",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 290,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ73MQEE4BTZN58ESSB0P6",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 237,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ7402RZ6P5Z249KA42WY2",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 297,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ741ZHWYDB45BPHT5R46M",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 247,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ740MSG2GXGNDDG4KBWXK",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 401,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ746PRGKM95006RTJ7KYY",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 283,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ74F032TWXAAMS4Y9FXYG",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 391,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ74GMQ2RNYNTBAY7MHSK1",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 305,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ74QKAVTNDMRPVDK6QEPR",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 67,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ74SH9J7WV1PVCTGYW6K2",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 460,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ751FXJ1VV5YZCNQR91Y8",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 12,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ750Z83GFKT90W0GP13HM",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 70,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ7599NFB6B31RF04NVE4F",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 277,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ75ENPGS36Z3J4GYZ428F",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 244,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ75F80K8KH4A48JVWFY87",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 19,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ75K6QYA0KWK4GG0CBTES",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 72,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ75FEKS3JTBMK17280T9Y",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": 457,
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ75V55YCBXKGE9K1KQC1M",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    },
+    {
+      "task_id": "isolation_probe",
+      "passed": true,
+      "sandbox_id": "sb-01M5DZ75SSWDD1CEAWE4X9RN4Y",
+      "returncode": 0,
+      "network": "blocked",
+      "credentials": "none",
+      "cpu_seconds_limit": 2,
+      "wall_seconds_limit": 10,
+      "address_space_bytes_limit": 268435456
+    }
+  ],
+  "execution_pass_rate": 1.0
+}

generate.py

@@ -0,0 +1,52 @@
+"""Generate text with the standalone Python model with the fixed evaluation sampling policy."""
+
+import argparse
+import json
+from pathlib import Path
+
+import torch
+from model_100m.model import GPT, ModelConfig
+from safetensors.torch import load_file
+from tokenizers import Tokenizer
+
+p = argparse.ArgumentParser(description=__doc__)
+p.add_argument("--prompt", default="The solar system consists of")
+p.add_argument("--tokens", type=int, default=128)
+p.add_argument("--seed", type=int, default=42)
+p.add_argument("--temperature", type=float, default=0.8, help="0 selects greedy decoding")
+p.add_argument("--top-k", type=int, default=50, help="0 samples the full vocabulary")
+p.add_argument("--device", default="cpu", choices=["cpu", "cuda"])
+a = p.parse_args()
+if a.tokens < 0 or a.temperature < 0 or a.top_k < 0:
+    p.error("Generation settings must be nonnegative")
+root = Path(__file__).resolve().parent
+torch.set_num_threads(2)
+torch.manual_seed(a.seed)
+model = GPT(ModelConfig(**json.loads((root / "config.json").read_text()))).to(a.device).eval()
+model.load_state_dict(load_file(str(root / "model.safetensors")), strict=True)
+tokenizer = Tokenizer.from_file(str(root / "tokenizer/tokenizer.json"))
+ids = [1] + tokenizer.encode(a.prompt, add_special_tokens=False).ids
+generator = torch.Generator(device=a.device).manual_seed(a.seed)
+torch.backends.cuda.matmul.allow_tf32 = False
+torch.backends.cudnn.allow_tf32 = False
+with torch.inference_mode():
+    for _ in range(a.tokens):
+        logits = model(torch.tensor([ids[-model.config.block_size :]], device=a.device))[
+            0, -1
+        ].float()
+        if not torch.isfinite(logits).all():
+            raise ValueError("Non-finite generation logits")
+        logits[[0, 1, 3]] = -torch.inf  # PAD/BOS/UNK; EOS remains eligible.
+        if a.temperature == 0:
+            token = int(logits.argmax())
+        elif a.top_k:
+            values, indices = torch.topk(logits / a.temperature, min(a.top_k, logits.numel()))
+            token = int(indices[torch.multinomial(values.softmax(-1), 1, generator=generator)])
+        else:
+            token = int(
+                torch.multinomial((logits / a.temperature).softmax(-1), 1, generator=generator)
+            )
+        ids.append(token)
+        if token == 2:
+            break
+print(tokenizer.decode(ids, skip_special_tokens=True))

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e5e55c0beaac47b09427c261d7dc7a9b94240758b6e0f3724d419ed43d8c1f5
+size 598238584

model_100m/PILOT.md

@@ -0,0 +1,62 @@
+# Scholarly/code subword pilot
+
+`modal_scholarly_pilot.py` connects the merged modern 100M-family model to the
+frozen 32K tokenizer and `TokenMixture`. It preserves the merged 14×768 shape
+(124,389,632 unique parameters), with context increased to 1,024. This is a
+bounded execution/learning pilot, not the planned 20B-token run.
+
+From `training/`, first prepare a fresh snapshot with `modal_prepare_scholarly.py`.
+Then:
+
+```sh
+modal run --detach modal_scholarly_pilot.py \
+  --run-id scholarly-124m-pilot-20261016-01 \
+  --snapshot scholarly-124m-20261016/snapshots/pilot-001
+```
+
+One L40S worker, four CPUs, 16 GiB host RAM, no retries, and a 30-minute hard
+worker timeout. Training stops at 600 optimizer updates, 5M loss-bearing tokens,
+20 minutes including intermediate checkpoint overhead, or source exhaustion.
+The existing loader enforces at most two passes per source. No full pretraining
+or automatic continuation is launched. The pilot deliberately permits snapshots
+with pending corpus audits; this override must not be used for substantial runs.
+
+Optimization: FP32 parameters, BF16 autocast, AdamW (betas 0.9/0.95, weight decay
+0.1), gradient norm clipping at 1, linear 30-update warmup to 3e-4 then constant
+learning rate. Batch size 2 with four accumulated microbatches; gradients are
+normalized by actual nonpadding targets across the complete optimizer batch.
+This short pilot does not select a production learning-rate schedule.
+
+The CPU qualification runs modern-model, tied-embedding and corpus-loader tests.
+The GPU worker verifies tiny-set fitting, then reloads the actual step-10
+checkpoint and checks identical next-batch IDs, labels and logits. Checkpoints
+include optimizer/RNG/data state, source hashes, model settings, token counts and
+manifest/tokenizer identities. Checkpoints and logs use fresh paths on
+`lm-exploration-training/<run-id>/`; the final report is also copied to `results/`.
+This pilot checkpoint format is separate from the character-only `lm-generate`
+format. Cross-job resume and generation CLI integration remain future work.
+
+Validation uses one fixed document-local window per source, reporting each source
+and the token-mixture-weighted mean. This small provisional validation sample is
+for detecting learning/failures, not capability claims or production checkpoint
+selection. Samples use three fixed prompts, seed 42, full-softmax temperature 1,
+and up to 64 new tokens. PAD/BOS/UNK are excluded from sampled outputs.
+
+The full corpus still needs split reconciliation, global near deduplication,
+provenance/eligibility and contamination audits, and capacity evidence. Measure
+throughput before selecting the substantial-run hardware and budget. The current
+model retains explicit attention; a supported SDPA kernel is a possible separately
+verified optimization.
+
+In the managed cloud environment, Modal's client needs the system CA trust:
+install `modal[api-proxy-support]==1.6.2` and `truststore`, call
+`truststore.inject_into_ssl()` before running the Modal CLI module. Keep inherited
+proxy settings and TLS verification enabled.
+
+## First measured run
+
+[The 2026-10-16 report](../results/scholarly-124m-pilot-20261016-01.md) records
+600 updates / 4,078,340 tokens, weighted pilot validation loss 10.5811 → 6.0272,
+23 passing qualification tests, and a successful checkpoint/data reload.
+The final checkpoint is `lm-exploration-training/scholarly-124m-pilot-20261016-01/step-000600.pt`.
+Samples remain incoherent; full corpus audits and substantial training are pending.

model_100m/README.md

@@ -0,0 +1,100 @@
+# Updated 100M-family architecture
+
+A subsequent [bounded subword pilot](PILOT.md) trained this architecture for
+4.08M tokens on Modal. See its [measured report](../results/scholarly-124m-pilot-20261016-01.md);
+the architecture-merge verification described below remains separate.
+
+`model.py` uses **RMSNorm, RoPE, SwiGLU and tied input/output embeddings** by
+default. There is one `100m` model choice, keeping 14 layers, width 768, 12 ordinary
+attention heads (head dimension 64) and context 256. The original vanilla 10M
+model and its individual ablation PRs are unchanged.
+
+| Component | Current default |
+| --- | --- |
+| Normalization | Pre-RMSNorm and final RMSNorm, epsilon 1e-5 |
+| Positions | RoPE on Q/K in every layer, theta 10,000 |
+| Feedforward | SwiGLU, hidden width 2,048 |
+| Projection biases | Present |
+| Attention kernel | PyTorch causal scaled-dot-product attention (SDPA) |
+| Initialization | Normal std 0.02 |
+| Input/output weights | One shared Parameter |
+
+## Mechanisms and references
+
+- **RMSNorm:** `gamma * x / sqrt(mean(x**2) + epsilon)` omits mean centering and
+  normalization offsets. Epsilon matches historical LayerNorm; low-precision
+  statistics accumulate in FP32. [Zhang & Sennrich (2019)](https://arxiv.org/abs/1910.07467)
+  motivate simpler normalization. Actual speed depends on kernels.
+- **RoPE:** rotates adjacent query/key feature pairs; values stay unchanged.
+  The positional dependence of Q/K dot products is relative.
+  [RoFormer, Su et al. (2021)](https://arxiv.org/abs/2104.09864) gives the construction.
+  Context remains capped by `block_size`: no GQA, KV cache or long-context scaling.
+- **SwiGLU:** `down(SiLU(gate(x)) * up(x))` provides input-dependent feature
+  modulation. Hidden width `ceil(8 * width / 3)` exactly matches the old MLP's
+  matrix-parameter budget at width 768. Projections keep biases.
+  [Shazeer (2020)](https://arxiv.org/abs/2002.05202) reports matched-budget T5 gains;
+  that motivates a causal-GPT experiment, not a claimed improvement here.
+- **Tied embeddings:** `self.lm_head.weight = self.token_embedding.weight`.
+  The shared table `E[vocabulary, width]` supplies input rows and output logits
+  `hidden @ E.T`. Gradients from both uses accumulate into the same Parameter,
+  which appears once in the optimizer's parameter list.
+  [Press & Wolf (2017)](https://arxiv.org/abs/1608.05859) study the parameter/quality
+  tradeoff. Sharing saves parameters but constrains the representations.
+
+[LLaMA §2.2](https://arxiv.org/abs/2302.13971) adopts RMSNorm, RoPE and SwiGLU.
+This update combines the requested choices; it cannot establish each component's
+quality contribution. The individual 10M ablations serve that purpose. No quality
+gain has been measured for this combined model.
+
+## Frozen tokenizer and parameter count
+
+Use merged `structured-code-v1`: **32,768 entries**, unchanged tokenizer hash
+`add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9`.
+At context 256 and the unchanged 14×768 shape, the model has **124,389,632 unique
+parameters**. Tying saves **25,165,824** (`32,768 × 768`) versus separate matrices
+on the same modern backbone. There is no maintained untied modern variant,
+separate package or recipe key for tying.
+
+The historical vanilla 100M had 149,760,000 parameters at this vocabulary.
+Besides tying, the modern blocks remove 196,608 position parameters and 22,272
+normalization offsets, and add 14,336 FFN bias parameters. FFN matrix parameters
+match. The `100m` label is historical: a 32K vocabulary changes the total relative
+to the character model. No saving is reinvested in extra layers.
+
+## Usage and compatibility
+
+```python
+from model_100m.model import GPT, ModelConfig
+
+model = GPT(ModelConfig(vocab_size=32768))
+assert model.lm_head.weight is model.token_embedding.weight
+```
+
+Recipes continue to select `model="100m"`; no tying flag is needed. New checkpoints
+record `architecture_version=2`. A small version-1 compatibility path reuses the
+original vanilla implementation to load historical 100M checkpoints, preserving
+their original untied weights and exact logits. Missing version metadata means
+legacy; the loader does not silently tie or modernize old weights. Resume across
+architecture versions is rejected. This is not a second modern model variant.
+
+The general recipe trainer and generation CLI remain character-only. The separate
+[subword pilot](PILOT.md) integrates the frozen BPE tokenizer and mixture loader.
+The [GPU comparison](../results/scholarly-124m-gpu-comparison-20261017.md) measures
+SDPA and fused AdamW on the full model, including two- and three-H100 DDP.
+The historical pilot used explicit attention; its original source is archived
+with its run. SDPA preserves the model shape and checkpoint keys, with output
+and gradient agreement checked against the explicit implementation.
+
+## Verification
+
+From `training/`:
+
+```sh
+uv sync --locked --extra corpus --extra research
+uv run --extra corpus --extra research python -m unittest discover -s tests -v
+```
+
+Checks cover independent block output/gradient references, causality, finite
+gradients, counts, config bounds and old checkpoint loading/resume. Tying checks
+cover shared identity, summed gradients against a test-only reference, optimizer
+deduplication, dtype conversion, checkpoint roundtrip and tokenizer ID bounds.

model_100m/__init__.py

@@ -0,0 +1 @@
+"""The architecture-only 100M model, installed by uv as model_100m."""

model_100m/model.py

@@ -0,0 +1,160 @@
+"""100M-family decoder: RMSNorm, RoPE, SwiGLU and tied embeddings by default.
+
+Keep 14 layers, width 768, 12 MHA heads, context 256, projection biases and
+vanilla weight initialization. architecture_version=1 reproduces the historical
+LayerNorm/learned-position/GELU model only for legacy checkpoint compatibility.
+"""
+
+from dataclasses import dataclass
+
+import torch
+from model_10m import model as vanilla
+from torch import nn
+from torch.nn import functional as F
+
+__all__ = ["GPT", "Attention", "Block", "ModelConfig", "RMSNorm", "SwiGLU", "causal_attention"]
+MODEL_DEPENDENCIES = {"model_10m_definition": vanilla.__file__}
+
+
+@dataclass(frozen=True)
+class ModelConfig(vanilla.ModelConfig):
+    n_layer: int = 14
+    n_head: int = 12
+    n_embd: int = 768
+    architecture_version: int = 2
+
+    def __post_init__(self):
+        if type(self.architecture_version) is not int or self.architecture_version not in (1, 2):
+            raise ValueError("architecture_version must be 1 (legacy) or 2 (RMSNorm/RoPE/SwiGLU)")
+        super().__post_init__()
+        if self.architecture_version == 2 and (self.n_embd // self.n_head) % 2:
+            raise ValueError("RoPE requires an even head dimension")
+
+
+def causal_attention(query, key, value):
+    """Standard causal SDPA; CUDA selects FlashAttention where supported.
+
+    Same scale, mask and zero dropout as the explicit 10M reference. The legacy
+    architecture retains its original implementation in model_10m.
+    """
+    return F.scaled_dot_product_attention(query, key, value, is_causal=True, dropout_p=0.0)
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, width, eps=1e-5):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(width))
+        self.eps = eps  # Match the historical LayerNorm epsilon.
+
+    def forward(self, x):
+        values = x.float() if x.dtype in (torch.float16, torch.bfloat16) else x
+        normalized = values * torch.rsqrt(values.square().mean(-1, keepdim=True) + self.eps)
+        return normalized.to(x.dtype) * self.weight.to(x.dtype)
+
+
+def rotary(x, offset=0):
+    """Rotate adjacent query/key feature pairs using position and theta 10,000."""
+    dtype = torch.float64 if x.dtype == torch.float64 else torch.float32
+    frequencies = 10000.0 ** (
+        -torch.arange(0, x.size(-1), 2, device=x.device, dtype=dtype) / x.size(-1)
+    )
+    positions = torch.arange(offset, offset + x.size(-2), device=x.device, dtype=dtype)
+    angles = torch.outer(positions, frequencies)
+    cos, sin = angles.cos().to(x.dtype), angles.sin().to(x.dtype)
+    even, odd = x[..., ::2], x[..., 1::2]
+    return torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1).flatten(-2)
+
+
+class Attention(vanilla.Attention):
+    def cache_rotary(self, length):
+        width = self.qkv.in_features // self.n_head
+        frequencies = 10000.0 ** (
+            -torch.arange(0, width, 2, device=self.qkv.weight.device).float() / width
+        )
+        angles = torch.outer(torch.arange(length, device=frequencies.device).float(), frequencies)
+        self.register_buffer("rope_cos", angles.cos(), persistent=False)
+        self.register_buffer("rope_sin", angles.sin(), persistent=False)
+
+    def rotate(self, x):
+        if not hasattr(self, "rope_cos"):
+            return rotary(x)
+        cos = self.rope_cos[: x.size(-2)].to(x.dtype)
+        sin = self.rope_sin[: x.size(-2)].to(x.dtype)
+        even, odd = x[..., ::2], x[..., 1::2]
+        return torch.stack((even * cos - odd * sin, even * sin + odd * cos), -1).flatten(-2)
+
+    def forward(self, x):
+        batch, length, width = x.shape
+        query, key, value = self.qkv(x).chunk(3, -1)
+        query, key, value = [
+            tensor.view(batch, length, self.n_head, width // self.n_head).transpose(1, 2)
+            for tensor in (query, key, value)
+        ]
+        attended = causal_attention(self.rotate(query), self.rotate(key), value)
+        return self.projection(attended.transpose(1, 2).contiguous().view(batch, length, width))
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, width):
+        super().__init__()
+        self.hidden_width = (8 * width + 2) // 3
+        self.gate = nn.Linear(width, self.hidden_width)
+        self.up = nn.Linear(width, self.hidden_width)
+        self.down = nn.Linear(self.hidden_width, width)
+
+    def forward(self, x):
+        return self.down(F.silu(self.gate(x)) * self.up(x))
+
+
+class Block(vanilla.Block):
+    def __init__(self, config):
+        super().__init__(config)
+        self.attention_norm = RMSNorm(config.n_embd)
+        self.attention = Attention(config)
+        self.mlp_norm = RMSNorm(config.n_embd)
+        self.mlp = SwiGLU(config.n_embd)
+
+
+class GPT(vanilla.GPT):
+    def __init__(self, config):
+        if config.architecture_version == 1:
+            super().__init__(config)
+            return
+        nn.Module.__init__(self)
+        self.config = config
+        self.token_embedding = nn.Embedding(config.vocab_size, config.n_embd)
+        self.blocks = nn.Sequential(*(Block(config) for _ in range(config.n_layer)))
+        self.final_norm = RMSNorm(config.n_embd)
+        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)
+        self.apply(self.initialize)
+        self.lm_head.weight = self.token_embedding.weight
+
+    def forward(self, tokens, labels=None, loss_chunk_size=0):
+        if self.config.architecture_version == 1:
+            return super().forward(tokens)
+        if not 0 < tokens.size(1) <= self.config.block_size:
+            raise ValueError("sequence length must be within the model's block_size")
+        hidden = self.final_norm(self.blocks(self.token_embedding(tokens)))
+        if labels is None:
+            return self.lm_head(hidden)
+        if not loss_chunk_size:
+            return F.cross_entropy(
+                self.lm_head(hidden).flatten(0, 1),
+                labels.flatten(),
+                ignore_index=-100,
+                reduction="sum",
+            )
+        # Recompute each projection during backward, bounding vocabulary-logit memory.
+        from torch.utils.checkpoint import checkpoint
+
+        def chunk_loss(features, targets):
+            return F.cross_entropy(
+                self.lm_head(features), targets, ignore_index=-100, reduction="sum"
+            )
+
+        return sum(
+            checkpoint(chunk_loss, features, targets, use_reentrant=False)
+            for features, targets in zip(
+                hidden.flatten(0, 1).split(loss_chunk_size), labels.flatten().split(loss_chunk_size)
+            )
+        )

model_10m/README.md

@@ -0,0 +1,80 @@
+# Vanilla Shakespeare GPT, approximately 10M parameters
+
+The complete, readable architecture is in [model.py](model.py): six pre-LayerNorm
+causal decoder blocks, width 384, six attention heads, learned positions, GELU
+MLPs, and a separate output projection. With 65 characters and context 256 it has
+**10,795,776 parameters**. The [100M architecture](../100m/README.md) now defaults
+to RMSNorm, RoPE and SwiGLU; its explicit version-1 mode preserves this vanilla
+block design at larger dimensions. This published 10M definition is unchanged.
+
+The model starts from random weights and learns next-character prediction from
+Tiny Shakespeare. This is a learning benchmark, not an instruction-following
+assistant. A contiguous 90/10 train/validation split prevents windows crossing
+the boundary; vocabulary is drawn from the whole corpus. There is no sealed test.
+
+## Bounded Modal training
+
+Run from `training/`, using a Python environment with Modal's API proxy support.
+The Modal image explicitly pins PyTorch 2.6.0, safetensors 0.8.0 and tqdm 4.70.1,
+independently of the newer version selected by the local training project's lock.
+The executed versions and source hashes are saved in every run's config.
+
+The published runs used the former `modal_shakespeare.py` (commit `ec5ebd1`). The
+same settings are now the [`shakespeare-10m` recipe](../recipes/shakespeare-10m.json),
+launched with the generic [recipe launcher](../recipes/README.md) (use a new run ID):
+
+```bash
+uv run --with 'modal[api-proxy-support]==1.6.1' modal run modal_train.py \
+  --recipe recipes/shakespeare-10m.json --run-id shakespeare-10m-smoke-02 --smoke
+```
+
+The original runs used app `lm-exploration-shakespeare-10m` and volume
+`lm-exploration-training`. The smoke worker has a 120-second hard timeout; the
+training worker a 1,500-second hard timeout and a cooperative 1,200-second training
+limit. Both use one L4, no retries, and persist artifacts on exit. A separate
+ignored `runs/shakespeare-budget.json` (now `runs/modal-budget.json`) reserved the full worker time at a
+conservative $5/hour, with a $3 allocation ceiling. This is reservation accounting,
+not an actual bill or an account-wide spending limit. No latent-workspace ledger
+or research volume is used.
+
+Training targets 4,000 updates with batch 32, context 256, FP32 AdamW,
+learning rate 0.0003, betas (0.9, 0.95), weight decay 0.1, clipping norm 1 and seed
+1337. Every 200 updates, evaluate 20 fixed-seed validation batches and save a
+checkpoint. Select the saved checkpoint with the lowest validation loss.
+Cooperative stopping logs/evaluates/saves the final completed update. Each run
+writes a report, metrics, source/runtime/dataset provenance and a sample.
+
+## Export and publication
+
+The published model was exported with `publish_shakespeare.py`, removed after
+publication (see commit `ec5ebd1`). It exported the validation-selected checkpoint
+to safetensors without optimizer or RNG state, checked the parameter count,
+selected update and architecture hash, and uploaded only inference files.
+The public bundle includes `model.py`, `generate.py`, safetensors weights, exact
+configuration/tokenizer, provenance, requirements and the measured training report.
+From the downloaded bundle run `python generate.py --prompt 'ROMEO:' --tokens 300`.
+The model uses custom PyTorch code, not Transformers AutoModel loading.
+
+## Completed training and public model
+
+[Public Hugging Face model](https://huggingface.co/jfrick/shakespeare-vanilla-gpt-10m). The bounded L4 run completed 4,000 updates
+in 503.3 worker seconds, with 32,768,000 training-character
+exposures. Validation selected **update 1,800**, after 14,745,600 exposures:
+**1.54483 nats/character**, character perplexity **4.687**.
+The final update scored 2.10690 on validation; later training overfit this small
+corpus, so the public weights come from the earlier checkpoint.
+
+[Full training report](../results/shakespeare-10m-train-01.json),
+[loss history](../results/shakespeare-10m-metrics.jsonl),
+[smoke report](../results/shakespeare-10m-smoke-01.json),
+and [publication receipt](../results/shakespeare-10m-publication.json).
+One seed, validation used for checkpoint selection, no untouched test result.
+
+Local verification covered the 10M parameter count, causal attention and gradients
+against SDPA, safe stopping/checkpoint saving, reproducible CPU resume, and exact
+exported logits/seeded generation against the selected training checkpoint.
+Ruff lint/format, lock consistency and Git diff checks passed.
+
+Pinned public release: `48b3eb9b56fd03056943bb905b8cd7637009588c`. Anonymous
+Hub download verified the weights and definition checksums, strict model loading
+and exact seeded generation from the downloaded standalone package.

model_10m/__init__.py

@@ -0,0 +1 @@
+"""Vanilla character GPT, approximately 10M parameters."""

model_10m/generate.py

@@ -0,0 +1,50 @@
+"""Standalone sampling from a downloaded Hugging Face Shakespeare model folder."""
+
+import argparse
+import json
+from pathlib import Path
+
+import torch
+from model import GPT, ModelConfig
+from safetensors.torch import load_file
+
+
+def load_model(folder, device="cpu"):
+    folder = Path(folder)
+    config = json.loads((folder / "config.json").read_text())
+    tokenizer = json.loads((folder / "tokenizer.json").read_text())
+    model = GPT(ModelConfig(**config)).to(device).eval()
+    model.load_state_dict(load_file(str(folder / "model.safetensors")), strict=True)
+    return model, tokenizer["characters"]
+
+
+@torch.no_grad()
+def generate(model, characters, prompt, tokens=300, seed=1339):
+    indices = {character: index for index, character in enumerate(characters)}
+    if set(prompt) - indices.keys():
+        raise ValueError("prompt contains characters absent from the training vocabulary")
+    context = [indices[c] for c in prompt] or [0]
+    random = torch.Generator().manual_seed(seed)
+    device = next(model.parameters()).device
+    output = []
+    for _ in range(tokens):
+        logits = model(torch.tensor([context[-model.config.block_size :]], device=device))
+        probabilities = logits[0, -1].softmax(-1).cpu()
+        token = torch.multinomial(probabilities, 1, generator=random).item()
+        context.append(token)
+        output.append(characters[token])
+    return prompt + "".join(output)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--folder", default=".")
+    parser.add_argument("--prompt", default="ROMEO:\n")
+    parser.add_argument("--tokens", type=int, default=300)
+    parser.add_argument("--seed", type=int, default=1339)
+    parser.add_argument("--device", default="cpu", choices=("cpu", "cuda", "mps"))
+    args = parser.parse_args()
+    if args.tokens < 0:
+        parser.error("tokens must be nonnegative")
+    model, characters = load_model(args.folder, args.device)
+    print(generate(model, characters, args.prompt, args.tokens, args.seed))

model_10m/model.py

@@ -0,0 +1,117 @@
+"""Architecture only: a roughly 10M-parameter, decoder-only character GPT.
+
+6 layers, width 384, 6 heads, learned positions, pre-LayerNorm and GELU MLPs.
+The exact count depends on context length and vocabulary; trainer/loop.py records it.
+Training, data loading, checkpoints and CLI code deliberately live elsewhere.
+"""
+
+import math
+from dataclasses import asdict, dataclass
+
+import torch
+from torch import nn
+
+
+# Model: B = batch, T = sequence length, C = embedding width, H = heads.
+@dataclass(frozen=True)
+class ModelConfig:
+    vocab_size: int
+    block_size: int = 256
+    n_layer: int = 6
+    n_head: int = 6
+    n_embd: int = 384
+
+    def __post_init__(self):
+        if any(value <= 0 for value in asdict(self).values()):
+            raise ValueError("all model dimensions must be positive")
+        if self.n_embd % self.n_head:
+            raise ValueError("n_embd must be divisible by n_head")
+
+
+def causal_attention(query, key, value):
+    # Q, K, V: [B, H, T, C/H]. A row may see itself and earlier columns only.
+    # For each query position, compare its feature vector with every key vector.
+    # Q and K are [B, H, T, D], so transpose K's last two axes to [B, H, D, T].
+    # This makes the matrix product [B, H, T, D] @ [B, H, D, T] yield
+    # [B, H, T, T]: one score for each query/key position pair. K itself is
+    # unchanged; transpose only changes the view of its axes for this product.
+    scores = query @ key.transpose(-2, -1) / math.sqrt(query.size(-1))
+    # Example causal mask for T = 4 (rows are queries; columns are keys):
+    # [[1, 0, 0, 0],
+    #  [1, 1, 0, 0],
+    #  [1, 1, 1, 0],
+    #  [1, 1, 1, 1]]
+    mask = torch.ones(query.size(-2), key.size(-2), device=query.device, dtype=torch.bool).tril()
+    weights = scores.masked_fill(~mask, float("-inf")).softmax(dim=-1)
+    return weights @ value
+
+
+class Attention(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.n_head = config.n_head
+        self.qkv = nn.Linear(config.n_embd, 3 * config.n_embd)
+        self.projection = nn.Linear(config.n_embd, config.n_embd)
+
+    def forward(self, x):
+        batch, length, width = x.shape
+        query, key, value = self.qkv(x).chunk(
+            3, dim=-1
+        )  # Split the combined projection into its three equal parts: query, key, and value.
+        # Split the embedding width evenly across heads; each head processes width / n_head features.
+        query, key, value = [
+            tensor.view(batch, length, self.n_head, width // self.n_head).transpose(1, 2)
+            for tensor in (query, key, value)
+        ]
+        attended = causal_attention(query, key, value)
+        attended = attended.transpose(1, 2).contiguous().view(batch, length, width)
+        return self.projection(attended)
+
+
+class Block(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.attention_norm = nn.LayerNorm(
+            config.n_embd
+        )  # n_embd is the embedding width (number of features per token).
+        self.attention = Attention(config)
+        self.mlp_norm = nn.LayerNorm(config.n_embd)
+        self.mlp = nn.Sequential(
+            nn.Linear(config.n_embd, 4 * config.n_embd),
+            nn.GELU(),
+            nn.Linear(4 * config.n_embd, config.n_embd),
+        )
+
+    def forward(self, x):
+        x = x + self.attention(self.attention_norm(x))
+        return x + self.mlp(self.mlp_norm(x))
+
+
+class GPT(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.token_embedding = nn.Embedding(config.vocab_size, config.n_embd)
+        self.position_embedding = nn.Embedding(config.block_size, config.n_embd)
+        self.blocks = nn.Sequential(*(Block(config) for _ in range(config.n_layer)))
+        self.final_norm = nn.LayerNorm(config.n_embd)
+        self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)
+        self.apply(self.initialize)
+
+    @staticmethod
+    def initialize(module):
+        if isinstance(module, (nn.Linear, nn.Embedding)):
+            nn.init.normal_(module.weight, std=0.02)
+            if isinstance(module, nn.Linear) and module.bias is not None:
+                nn.init.zeros_(module.bias)
+
+    def forward(self, tokens):
+        if not 0 < tokens.size(1) <= self.config.block_size:
+            raise ValueError("sequence length must be within the model's block_size")
+        positions = torch.arange(tokens.size(1), device=tokens.device)
+        x = self.token_embedding(tokens) + self.position_embedding(positions)
+        logits = self.lm_head(
+            self.final_norm(self.blocks(x))
+        )  # [B, T, vocabulary]; normalize the final hidden states before projecting them to token scores.
+
+        return logits

provenance.json

@@ -0,0 +1,68 @@
+{
+  "format": "scholarly-ddp-v1",
+  "config": {
+    "world_size": 8,
+    "global_windows": 32,
+    "microbatch": 2,
+    "crop_padding": true,
+    "loss_chunk_size": 0,
+    "compile_model": true,
+    "cache_rope": true,
+    "context": 8192,
+    "target_tokens": 5000000000,
+    "warmup_tokens": 20000000,
+    "optimizer": "muon",
+    "muon_lr": 0.02,
+    "muon_momentum": 0.95,
+    "peak_lr": 0.0003,
+    "min_lr_ratio": 0.1,
+    "weight_decay": 0.1,
+    "seed": 42,
+    "checkpoint_every": 1000,
+    "evaluate_every": 500,
+    "pilot": false,
+    "experimental_data": true,
+    "prefetch_batches": 4,
+    "n_layer": 14,
+    "n_head": 12,
+    "n_embd": 768
+  },
+  "data_identity": {
+    "manifest_sha256": "70f63fa22fe19807bace17499f53a25c46145ad2c4e4966f61bef7e310528146",
+    "context": 8192,
+    "split": "train",
+    "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9"
+  },
+  "source_sha256": {
+    "trainer": "e90356e9f0e7740c4aae2fbc257d7facef2afb4152731aeacfd8143a28c204a3",
+    "heldout_loader": "afa53c739b8c6b81540e4b9b6d85f90d5353802537ab564d97f1c29df196cda4",
+    "config": "b3a9a2438b5fac9f383d138bdc0bd9987d88497162e5260c2b96b2f9b7572a7b",
+    "optimizer": "de43032f664ab20a1471ffa9d5be987c8fa4634d7aafb43fa7f16a7ca4fe9759",
+    "model": "2c978432e84cad6b2f661c63e7e62bccfc393a2aeb9f0519f7ca3db6775ed326",
+    "loader": "7821e13e765a9f2c0e0a94744bc7f2df8eea647f26d7aa0069f62556618a0ef2",
+    "raw_loader": "4312f800170e51f0a1d2be2b978c55fe9951f6df681c2bb225d83ecc0f33a6e7",
+    "prepared_loader": "c485580bd27298502a9ab4e9950f50e5680001bfe0e40cbb515906928ff76755",
+    "prefetch": "1786b1423f5881b23223ff0db47284a522020237d77e3d7fa28d542deea5c218",
+    "live_loader": "64457d0e60f1f1b664d3b174183c6e6f3d0c618973fe56712f9e72cd2a118a75",
+    "tokenizer": "00e32adc13dab74f5a029c63304b4591bd48f7cd6b086b22d4624d244667b4da"
+  },
+  "torch_version": "2.6.0+cu124",
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "parameters": 124389632,
+  "resume_from": "/runs/python-124m-muon-5b-20261021-v1-s001/step-000000004.pt",
+  "step": 549,
+  "tokens": 50065393,
+  "run_id": "python-124m-muon-5b-20261021-v1",
+  "checkpoint": "/runs/python-124m-muon-5b-20261021-v1-s002/step-000000549.pt",
+  "weights_sha256": "8e5e55c0beaac47b09427c261d7dc7a9b94240758b6e0f3724d419ed43d8c1f5",
+  "weight_roundtrip_exact": true,
+  "cpu_logits_roundtrip_exact": true,
+  "loss_token_exposures": {
+    "python_implementation": 32533334,
+    "python_tests_examples": 7508444,
+    "python_documentation": 5009437,
+    "cs_math": 2506747,
+    "general_education": 1502082,
+    "books": 1005349
+  }
+}

qualification.json

@@ -0,0 +1,56 @@
+{
+  "passed": true,
+  "parameters": 124389632,
+  "initial_loss": 10.464919090270996,
+  "final_loss": 0.0014346196549013257,
+  "updates": 21,
+  "losses": [
+    10.461538314819336,
+    7.20192289352417,
+    4.79567289352417,
+    2.724158763885498,
+    1.326021671295166,
+    0.6623347401618958,
+    0.11782602220773697,
+    0.10554386675357819,
+    0.023414025083184242,
+    0.010023850947618484,
+    0.0060213529504835606,
+    0.004254854749888182,
+    0.003371018683537841,
+    0.002859849249944091,
+    0.0025118314661085606,
+    0.0022271962370723486,
+    0.002055828459560871,
+    0.0018862211145460606,
+    0.0017189612844958901,
+    0.0016150841256603599,
+    0.0014346196549013257
+  ],
+  "completion": {
+    "prompt": "def add(a, b):\n    ",
+    "completion": "    return a + b\n",
+    "token_ids": [
+      262,
+      401,
+      277,
+      361,
+      305,
+      202
+    ],
+    "temperature": 0.0,
+    "top_k": 0,
+    "seed": 42,
+    "new_tokens": 6,
+    "stop_reason": "eos",
+    "syntax_valid": true,
+    "repeated_token_4gram_fraction": 0.0,
+    "longest_tandem_repeat_tokens": 0,
+    "long_loop": false
+  },
+  "logit_alignment": "passed",
+  "causality": "passed",
+  "cached_rope": "passed",
+  "optimizer_partition": "passed",
+  "note": "Separate diagnostic model; its weights are discarded"
+}

requirements.txt

@@ -0,0 +1,3 @@
+torch==2.6.0
+safetensors>=0.4
+tokenizers==0.22.2

runtime-cost-estimate.json

@@ -0,0 +1,11 @@
+{
+  "kind": "GPU compute estimate, not measured invoice",
+  "training_worker_reported_seconds": 842.245771547,
+  "h100_usd_per_second": 0.001097,
+  "excludes": [
+    "GPU work outside recorded training segments",
+    "CPU preparation and evaluation",
+    "storage and transfer"
+  ],
+  "training_gpu_usd_estimate": 7.391548891096472
+}

tokenizer/config.json

@@ -0,0 +1,18 @@
+{
+  "format": "huggingface-tokenizers-json",
+  "vocab_size": 32768,
+  "special_tokens": {
+    "<|pad|>": 0,
+    "<|bos|>": 1,
+    "<|eos|>": 2,
+    "<|unk|>": 3
+  },
+  "normalization": null,
+  "pretokenization_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "add_prefix_space": false,
+  "automatic_bos_eos": false,
+  "padding": null,
+  "truncation": null,
+  "decode_skip_special_tokens_default_for_roundtrips": false,
+  "all_valid_UTF8_supported": true
+}

tokenizer/evaluation.json

@@ -0,0 +1,59 @@
+{
+  "name": "focused-64mib-32768",
+  "vocab": 32768,
+  "execution": "modal-cpu",
+  "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+  "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+  "metrics": {
+    "python": {
+      "bytes": 2513044,
+      "tokens": 615688,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 4.08168422967477,
+      "mean_tokens_per_example": 1202.515625,
+      "p95_tokens_per_example": 2184
+    },
+    "json": {
+      "bytes": 2782302,
+      "tokens": 722785,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 3.849418568453966,
+      "mean_tokens_per_example": 1411.689453125,
+      "p95_tokens_per_example": 2514
+    },
+    "git_diff": {
+      "bytes": 187152,
+      "tokens": 52862,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 3.5403881805455715,
+      "mean_tokens_per_example": 103.24609375,
+      "p95_tokens_per_example": 249
+    },
+    "tool_call": {
+      "bytes": 965682,
+      "tokens": 251703,
+      "examples": 512,
+      "roundtrip_failures": 0,
+      "unk_tokens": 0,
+      "bytes_per_token": 3.8365931276146887,
+      "mean_tokens_per_example": 491.607421875,
+      "p95_tokens_per_example": 2304
+    }
+  },
+  "fragmentation": {
+    "{\"name\":\"read_file\",\"arguments\":{\"path\":\"src/main.py\"}}": 14,
+    "diff --git a/main.py b/main.py\n@@ -1,2 +1,2 @@\n-old\n+new\n": 25,
+    "get_user_configuration": 3,
+    "\"additionalProperties\":false": 4,
+    "\"tool_call_id\":\"call_0123456789\"": 12
+  },
+  "correctness_passed": true,
+  "family": "focused",
+  "size_mib": 64
+}

tokenizer/freeze-manifest.json

@@ -0,0 +1,12 @@
+{
+  "id": "structured-code-bpe-v1",
+  "status": "frozen",
+  "vocab_size": 32768,
+  "files": {
+    "recipe.json": "a4328e99a51bee64daaee66efe25e80ce01a263d568a69a130b94f687edef407",
+    "config.json": "2a4fa59b0a93ebd0995f3ea308c01484647e9ede935e128b4dba163b384224aa",
+    "tokenizer.json": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+    "evaluation.json": "922b0fc33a0c38f06eaedbf3497a50723f4c66ac4dd8f96267038ad64764d80f"
+  },
+  "contract": "Immutable. Record tokenizer SHA256 in shards and checkpoints. Prior tokenizer data must be retokenized in a new version."
+}

tokenizer/publication-verification.json

@@ -0,0 +1,17 @@
+{
+  "location": "jfrick/scholarly-code-tokenizer",
+  "revision": "0ebdb4fe4c8d21ccc06cfe6f5b464eac74773a80",
+  "status": "passed",
+  "default_and_variants_checked": 20,
+  "probes_per_variant": 6,
+  "checks": [
+    "artifact_hashes",
+    "special_ids",
+    "vocabulary_size",
+    "token_id_equivalence",
+    "exact_roundtrips",
+    "no_automatic_boundaries"
+  ],
+  "transformers_version": "4.57.6",
+  "tokenizers_version": "0.22.2"
+}

tokenizer/publication.json

@@ -0,0 +1,9 @@
+{
+  "repo_id": "jfrick/scholarly-code-tokenizer",
+  "revision": "0ebdb4fe4c8d21ccc06cfe6f5b464eac74773a80",
+  "url": "https://huggingface.co/jfrick/scholarly-code-tokenizer",
+  "variant": "structured-code-v1-32768",
+  "tokenizer_sha256": "add5b2c7542e4efded329735825ecdcb94df1432617c3d3e4b7508aa459432f9",
+  "vocab_size": 32768,
+  "selection": "User explicitly selected code-focused 32K as repository and Hub default."
+}

tokenizer/recipe.json

@@ -0,0 +1,37 @@
+{
+  "seed": 20261016,
+  "target_bytes": 67108864,
+  "weights_percent": {
+    "python": 40,
+    "json": 20,
+    "git_diff": 20,
+    "tool_call": 20
+  },
+  "actual_bytes": {
+    "python": 26847131,
+    "json": 13424847,
+    "git_diff": 13421882,
+    "tool_call": 13422435
+  },
+  "training_examples": {
+    "python": 5059,
+    "json": 2290,
+    "git_diff": 33758,
+    "tool_call": 6555
+  },
+  "fitting_parents": 14911,
+  "evaluation_parents_per_category": 512,
+  "train_sha256": "cdc34355495f1c21bba4742ac11e9bcc0a0d0df836ab49eed9a1e01f97ab9a09",
+  "evaluation_sha256": "a9a82723c1e78b46ec52b33b81551a61a5498e46ac73a6f09e4c0da9e27775a0",
+  "source_train_sha256": "5e1c04e310375d9922e008142e7d9813f56e541918b729dd7764eaec3ffbadea",
+  "source_validation_sha256": "611010954c9c48ccad9b9a28f7377d56ab180373b5e7eb76fe87ab723b2b8040",
+  "implementation_sha256": "cd1f6e5351aad5acce1a1021af649c4dd8a099c2b0d506e7cc67cd5aee230163",
+  "limitations": [
+    "Python excerpts from the existing pinned codeparrot corpus, not necessarily complete parseable files",
+    "JSON, tool calls and patches are generated controlled fixtures, not independent production traces",
+    "Fitting/evaluation parents are disjoint; generator templates are shared",
+    "Each category quota includes its final whole example, so 64 MiB is approximate",
+    "Unified diffs cover insertions, deletions and identifier replacements, but not binary patches or renames",
+    "One fitting seed; no model-level tool-call or code-generation correctness measured"
+  ]
+}

validation.jsonl

@@ -0,0 +1,7 @@
+{"step": 0, "tokens": 0, "target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "per_stratum": {"python_implementation": 10.52420629872608, "python_tests_examples": 10.49302304090685, "python_documentation": 10.538044416813456, "cs_math": 10.498617869293456, "general_education": 10.580692296640718, "books": 10.552069240456483}, "stratum_target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "windows": 192, "per_source": {"python_implementation": 10.52420629872608, "python_tests_examples": 10.49302304090685, "python_documentation": 10.538044416813456, "cs_math": 10.498617869293456, "general_education": 10.580692296640718, "books": 10.552069240456483}, "weighted_loss": 10.521885039162349}
+{"step": 2, "tokens": 198806, "target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "per_stratum": {"python_implementation": 10.511782852054447, "python_tests_examples": 10.480197937202234, "python_documentation": 10.53078480641039, "cs_math": 10.490445001582403, "general_education": 10.573435295961549, "books": 10.54268923281465}, "stratum_target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "windows": 192, "per_source": {"python_implementation": 10.511782852054447, "python_tests_examples": 10.480197937202234, "python_documentation": 10.53078480641039, "cs_math": 10.490445001582403, "general_education": 10.573435295961549, "books": 10.54268923281465}, "weighted_loss": 10.510346118671023}
+{"step": 2, "tokens": 198806, "target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "per_stratum": {"python_implementation": 10.511782852054447, "python_tests_examples": 10.480197937202234, "python_documentation": 10.53078480641039, "cs_math": 10.490445001582403, "general_education": 10.573435295961549, "books": 10.54268923281465}, "stratum_target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "windows": 192, "per_source": {"python_implementation": 10.511782852054447, "python_tests_examples": 10.480197937202234, "python_documentation": 10.53078480641039, "cs_math": 10.490445001582403, "general_education": 10.573435295961549, "books": 10.54268923281465}, "weighted_loss": 10.510346118671023}
+{"step": 4, "tokens": 387014, "target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "per_stratum": {"python_implementation": 10.479941665000817, "python_tests_examples": 10.448600116025958, "python_documentation": 10.512487911442683, "cs_math": 10.469178979745234, "general_education": 10.556407058147432, "books": 10.516637554925667}, "stratum_target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "windows": 192, "per_source": {"python_implementation": 10.479941665000817, "python_tests_examples": 10.448600116025958, "python_documentation": 10.512487911442683, "cs_math": 10.469178979745234, "general_education": 10.556407058147432, "books": 10.516637554925667}, "weighted_loss": 10.480984802628893}
+{"step": 4, "tokens": 387014, "target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "per_stratum": {"python_implementation": 10.479941665000817, "python_tests_examples": 10.448600116025958, "python_documentation": 10.512487911442683, "cs_math": 10.469178979745234, "general_education": 10.556407058147432, "books": 10.516637554925667}, "stratum_target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "windows": 192, "per_source": {"python_implementation": 10.479941665000817, "python_tests_examples": 10.448600116025958, "python_documentation": 10.512487911442683, "cs_math": 10.469178979745234, "general_education": 10.556407058147432, "books": 10.516637554925667}, "weighted_loss": 10.480984802628893}
+{"step": 500, "tokens": 45650831, "target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "per_stratum": {"python_implementation": 4.162418328667418, "python_tests_examples": 3.918841802409977, "python_documentation": 5.080653659561132, "cs_math": 4.339001730203496, "general_education": 5.349691602343097, "books": 5.309149448988822}, "stratum_target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "windows": 192, "per_source": {"python_implementation": 4.162418328667418, "python_tests_examples": 3.918841802409977, "python_documentation": 5.080653659561132, "cs_math": 4.339001730203496, "general_education": 5.349691602343097, "books": 5.309149448988822}, "weighted_loss": 4.285087373511676}
+{"step": 549, "tokens": 50065393, "target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "per_stratum": {"python_implementation": 3.89459791981912, "python_tests_examples": 3.5985489956984402, "python_documentation": 4.833967628435524, "cs_math": 4.094523666119857, "general_education": 5.163108580551279, "books": 5.183435151830238}, "stratum_target_counts": {"python_implementation": 55027, "python_tests_examples": 97392, "python_documentation": 28953, "cs_math": 50556, "general_education": 38282, "books": 256483}, "windows": 192, "per_source": {"python_implementation": 3.89459791981912, "python_tests_examples": 3.5985489956984402, "python_documentation": 4.833967628435524, "cs_math": 4.094523666119857, "general_education": 5.163108580551279, "books": 5.183435151830238}, "weighted_loss": 4.017955903839883}