| --- |
| language: en |
| library_name: pytorch |
| tags: |
| - causal-lm |
| - python |
| - from-scratch |
| - muon |
| - experimental |
| --- |
| # Simple LLM Python 124M / 5B |
|
|
| Run: `python-124m-muon-5b-20261021-v1`. This is an **intermediate** base-model checkpoint, freshly initialized. |
| Processed **4,500,012,372 nonpadding training targets** in **77,343 optimizer updates**. |
| The exact target is 5,000,000,000. This model is not instruction-tuned. |
|
|
| 124,389,632 parameters; 14 layers, width 768, 12 attention heads, RMSNorm, |
| RoPE, SwiGLU, tied embeddings and the existing frozen 32K tokenizer. Training |
| uses eight H100s, 8K context, cropped padding, cached RoPE, torch.compile, |
| microbatch 2 and global batch 32 document-local windows. Hidden matrices use |
| the previously tested five-step Muon (LR .02, momentum .95); tied embeddings, |
| norms and biases use auxiliary AdamW (LR .0003, betas .9/.95). Weight decay .1, |
| gradient clipping 1, 20M-target warmup and cosine decay to 10%. |
|
|
| ## Data |
| Target proportions: 65% Python implementation, 15% Python tests/examples, |
| 10% Python programming documentation (including explanatory technical Q&A), |
| 5% CS/math, 3% general educational prose, 2% licensed/public-domain books. |
| Complete recent collected repository snapshots are preferred. Supplemental |
| sources include pinned CodeParrot pools, Common Pile StackExchange, FineWeb-Edu, |
| FineMath and public-domain Gutenberg material. All code allocations are Python. |
| The capacity table and exact per-category targets are in `capacity.json` and |
| `data-manifest.json`; one pass is preferred and two is the maximum. Public source |
| pins, aggregate collected-source capacity and filtering limitations are recorded |
| in `data-provenance.json`. |
|
|
| Repository/thread splits precede windows. Exact held-out hashes and repositories |
| are excluded across sources. Exact deduplication and repository caps are applied; |
| near clones can remain. Path/content filtering and license metadata are imperfect. |
| Some supplements are older and may contain outdated practices or errors. |
| Private documents and sensitive source-level provenance are not published. |
| StackExchange sources carry per-document CC-BY-SA attribution in the private |
| provenance ledger; Common Pile source revision is |
| `5ec3aa2e65b7c61b7c18f1ff18ccbc4d3279a23c`. |
|
|
| ## Evaluation |
| Held-out category losses, all fixed greedy/sampled Python generations, syntax, |
| repetition, length/EOS metrics and hidden-test execution results are included. |
| Function evaluation uses 32 reproducibly selected MBPP sanitized test tasks |
| (CC-BY-4.0, Google Research), revision |
| `4bb6404fdc6cacfda99d4ac4205087b89d32030c`. Tests are omitted from model prompts. |
| Exact reference hashes are excluded from training; no exhaustive benchmark |
| contamination audit or benchmark-clean claim is made. Generated code executes |
| in isolated network-blocked, credential-free sandboxes with resource limits. |
| Low early accuracy alone does not stop training. |
|
|
|  |
|  |
|
|
| All fixed-suite generations are retained without selection in `evaluations/`. |
| Each Hub commit preserves an inference checkpoint. Optimizer, RNG and data cursors |
| remain in private Modal training checkpoints for exact continuation. |
| Weight hashes are in `provenance.json`; runtime/cost figures are explicitly estimates. |
| Use the bundled custom PyTorch model, not Transformers AutoModel: |
|
|
| ```bash |
| pip install -r requirements.txt |
| python generate.py --prompt 'def add(a, b):' --tokens 128 |
| ``` |
|
|
| Code and execution evidence: https://github.com/koogle/lm-exploration/pull/30 . |
|
|
