| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - polish |
| - checkpoints |
| - training-history |
| --- |
| |
| # Slayer1B 1.01 — training checkpoints |
|
|
| Full trainer checkpoints of the run that produced Slayer1B 1.01: continued training of |
| [Slayer1B 1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00) from its last checkpoint before the learning-rate decay |
| ([`ckpt_1050.pt`](https://huggingface.co/SlayerLab/Slayer1B-1.00-checkpoints)), with a new total budget of twice the |
| tokens per parameter of 1.00. Each file holds the model weights **and the optimizer state**. |
|
|
| | | | |
| |---|---| |
| | Start | 1.00 `ckpt_1050.pt` (step 1050, constant lr 6e-4) | |
| | Steps | 2860 in total × 1 048 576 tokens = 2 998 927 360 tokens (≈ 2.8 tokens per parameter; 1.00: 1430 steps) | |
| | Schedule | constant lr 6e-4 through step 2289, then 1 − sqrt decay over steps 2290–2860 to lr / 10 | |
| | Data | steps 1051–1801: the same pack-00 stream as 1.00 (same windows); steps 1802–2860: pack-01 of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) (fresh web slice; shared components disjoint from those already used) | |
| | Last checkpoint before the decay | **`ckpt_2250.pt`** — starting point for further continued training | |
| | Training curve and evaluations during training | https://track.fabryka.ai/run/4934429a-d393-42e0-8aac-03eae6b71ff7 | |
| |
| Checkpoints every 150 steps from 1200 to 2850, plus the final step 2860. Released model (weights in safetensors) and |
| model card: [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01). |
| |
| ## File format |
| |
| `torch.save` dictionary of the trainer `train_gpt_ref_r23.py` (in the 1.00 model repository): `model`, `opt` (Muon + |
| AdamW), `step`, `config`. Load with `torch.load(path, map_location="cpu", weights_only=True)`. |
| |
| ## Run record (`run/`) |
| |
| `metrics.jsonl` (training loss and learning rate every 10 steps), `train.log`, `train.out`, `args.txt` (trainer |
| arguments), `checkpoints_SHA256SUMS` (written on the node at save time), data manifests and sums of both token streams |
| (`wsad_v2_*`: pack-00 stream, `wsad_p01_*`: pack-01 stream), `13gram_p01.json` (13-gram overlap of the pack-01 stream |
| with the evaluation sets), `pip_freeze.txt` (software environment). |
| |
| ## SHA-256 |
| |
| Recorded off the training node at the moment each checkpoint was written, before upload; files are downloaded back and |
| checked against these sums. |
| |
| ``` |
| 3637f1d90f60c2151724ad88b4b7c52925c87b7330a7680974b9093e8c1b4021 ckpt_1200.pt |
| d6d75584ffbd7261ed23f489ed4927d3e33fc3a6fef3bd9d3ad46b309b63daf4 ckpt_1350.pt |
| 3b2af30f8f4331f7bf549b1e7f0b2643ac2ec8ad3369a42b579d578930ba2934 ckpt_1500.pt |
| db254bfb77e302fc618f9cc7e3c4e23f1e3d6923386e54ad5fb930ff78a04bc4 ckpt_1650.pt |
| 9591dfa0d41f68b9e5cc40db2caf6f601b102c31b37c76d2ddeffa9f26cec49e ckpt_1800.pt |
| 6b8037e19b3fc42bb1c70599754749cdbb6a507a27fa4e2c3e316d676e969555 ckpt_1950.pt |
| b46f5e02f74dab2db1055fa009668c784411f403e01198dfff36deae9a0200f5 ckpt_2100.pt |
| aeddfb023abb11c8654b47c728c9680c232d1b6568697e3a1eee8128efd37054 ckpt_2250.pt |
| 616df6070fbc6e338e106d4b535030d06767063168e80494b67f508005904b73 ckpt_2400.pt |
| 82fd3d07914308411a2ddec965ce6881f296b8bb6956369096534bdc2c45b0c8 ckpt_2550.pt |
| e2b5b29f05d3113c0bc95a5a4d74fe07fee057031068bc82844f129a341c09f0 ckpt_2700.pt |
| decd0f02487fa15c65aaaaf9e2c2320f196571efa0337c8c588705915f7fd315 ckpt_2850.pt |
| b0a288798ae614c38e37ff8176a293f6650cec2221d59e46d5116afac7c31eba ckpt_2860.pt |
| ``` |
| |
| ## License |
| |
| Apache-2.0. |
| |
