File size: 3,544 Bytes
4719692 98cd910 4719692 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | ---
language: en
library_name: pytorch
tags:
- causal-lm
- python
- from-scratch
- muon
- experimental
---
# Simple LLM Python 124M / 5B
Run: `python-124m-muon-5b-20261021-v1`. This is an **intermediate** base-model checkpoint, freshly initialized.
Processed **4,500,012,372 nonpadding training targets** in **77,343 optimizer updates**.
The exact target is 5,000,000,000. This model is not instruction-tuned.
124,389,632 parameters; 14 layers, width 768, 12 attention heads, RMSNorm,
RoPE, SwiGLU, tied embeddings and the existing frozen 32K tokenizer. Training
uses eight H100s, 8K context, cropped padding, cached RoPE, torch.compile,
microbatch 2 and global batch 32 document-local windows. Hidden matrices use
the previously tested five-step Muon (LR .02, momentum .95); tied embeddings,
norms and biases use auxiliary AdamW (LR .0003, betas .9/.95). Weight decay .1,
gradient clipping 1, 20M-target warmup and cosine decay to 10%.
## Data
Target proportions: 65% Python implementation, 15% Python tests/examples,
10% Python programming documentation (including explanatory technical Q&A),
5% CS/math, 3% general educational prose, 2% licensed/public-domain books.
Complete recent collected repository snapshots are preferred. Supplemental
sources include pinned CodeParrot pools, Common Pile StackExchange, FineWeb-Edu,
FineMath and public-domain Gutenberg material. All code allocations are Python.
The capacity table and exact per-category targets are in `capacity.json` and
`data-manifest.json`; one pass is preferred and two is the maximum. Public source
pins, aggregate collected-source capacity and filtering limitations are recorded
in `data-provenance.json`.
Repository/thread splits precede windows. Exact held-out hashes and repositories
are excluded across sources. Exact deduplication and repository caps are applied;
near clones can remain. Path/content filtering and license metadata are imperfect.
Some supplements are older and may contain outdated practices or errors.
Private documents and sensitive source-level provenance are not published.
StackExchange sources carry per-document CC-BY-SA attribution in the private
provenance ledger; Common Pile source revision is
`5ec3aa2e65b7c61b7c18f1ff18ccbc4d3279a23c`.
## Evaluation
Held-out category losses, all fixed greedy/sampled Python generations, syntax,
repetition, length/EOS metrics and hidden-test execution results are included.
Function evaluation uses 32 reproducibly selected MBPP sanitized test tasks
(CC-BY-4.0, Google Research), revision
`4bb6404fdc6cacfda99d4ac4205087b89d32030c`. Tests are omitted from model prompts.
Exact reference hashes are excluded from training; no exhaustive benchmark
contamination audit or benchmark-clean claim is made. Generated code executes
in isolated network-blocked, credential-free sandboxes with resource limits.
Low early accuracy alone does not stop training.
All fixed-suite generations are retained without selection in `evaluations/`.
Each Hub commit preserves an inference checkpoint. Optimizer, RNG and data cursors
remain in private Modal training checkpoints for exact continuation.
Weight hashes are in `provenance.json`; runtime/cost figures are explicitly estimates.
Use the bundled custom PyTorch model, not Transformers AutoModel:
```bash
pip install -r requirements.txt
python generate.py --prompt 'def add(a, b):' --tokens 128
```
Code and execution evidence: https://github.com/koogle/lm-exploration/pull/30 .
|