| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - polish |
| - checkpoints |
| - training-history |
| --- |
| |
| # Slayer1B 1.03 — training checkpoints |
|
|
| Full trainer checkpoints of the run that produces Slayer1B 1.03: continued training of |
| [Slayer1B 1.02](https://huggingface.co/SlayerLab/Slayer1B-1.02) from its final checkpoint |
| ([`ckpt_5720.pt`](https://huggingface.co/SlayerLab/Slayer1B-1.02-checkpoints), sha256 `6b2a9b11…`, constant learning |
| rate). Each file holds the model weights **and the optimizer state**. |
|
|
| **Status (27 October 2026): training finished at step 7150** (05:14:58 UTC, train loss 2.3955 at step 7150). |
| All checkpoints of this run, up to the final `ckpt_7150.pt`, are in this repository; their SHA-256 are listed below. |
|
|
| | | | |
| |---|---| |
| | Start | 1.02 `ckpt_5720.pt` (step 5720, constant lr 6e-4) | |
| | Steps | 7150 in total × 1 048 576 tokens = 7 497 318 400 tokens (+1430 steps = +1 499 463 680 tokens over 1.02) | |
| | Schedule | **constant lr 6e-4 to the end, no final decay** — the decay is deferred to the end of the whole Slayer1B line (pre-training continues on the whole prepared corpus) | |
| | Data | continues the pack-03 stream of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) (revision `d50df04a`) used by 1.02 from step 5390: steps 5721–7150 | |
| | Hardware | steps 5721–6750: 4 × NVIDIA RTX 4090 (24 GB); steps 6751–7150: 4 × NVIDIA L40S (48 GB), resumed from `ckpt_6750.pt` (sha256 `970518a1…`) with the same trainer, recipe, data order and global batch. The data order of the resume was verified against the original run (batch hash of zero-based iterations 6751 and 7149, i.e. native steps 6752 and 7150, identical). DDP; global batch 256 × 4096 tokens as in 1.00–1.02 (effective: 4 ranks × micro-batch 2 × gradient accumulation 32) | |
| | Checkpoints | every 150 steps from 5850 and the final step 7150; all are constant-lr states | |
| | Trainer | `train_gpt_ref_r24_1.py`: the 1.02 trainer, releasing the checkpoint tensors from GPU memory after resuming (needed on 24 GB cards); the model definition is unchanged | |
| | Training curve | steps up to 7150 (6860–7150 from the L40S resume): https://track.fabryka.ai/run/7ca4f84f-3b07-4525-8cde-f3a936dc3425 | |
| | Benchmarks 1.00–1.03 (lm-eval 0.4.13, 0-shot) | https://track.fabryka.ai/run/1426c0a8-2f07-5a03-80fc-92f7e922aa4b | |
|
|
| Training continues from `ckpt_7150.pt` on the rest of the prepared corpus (track: |
| https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5); those checkpoints belong to the next versions. |
|
|
| ## Benchmark caveat |
|
|
| A leak scan of the whole training superset (SlayerLab/slayer-pl-8x3b@d50df04a, 22 603 122 documents) found n-gram |
| overlap with BoolQ, MMLU and ARC test items in the English shared part (`shared/en`: 1428 / 1429 / 1345 documents). |
| Overlap is not a measured contamination rate; the assessment is in progress. Read the BoolQ, MMLU and ARC numbers with |
| this caveat. From pack-04 on (first filtered update: zero-based 7192, native step 7193), the 5133 matching documents |
| are excluded from new training data; earlier steps, including 1.00–1.03, used the unfiltered data. |
|
|
| ## File format |
|
|
| `torch.save` dictionary of the trainer: `model`, `opt` (Muon + AdamW), `step`, `config`. Load with |
| `torch.load(path, map_location="cpu", weights_only=True)`. Model class: `train_gpt_ref_r23.py` in the 1.00–1.02 model |
| repositories. |
|
|
| ## SHA-256 |
|
|
| Recorded off the training process at the moment each checkpoint is written, before upload; files are downloaded back |
| and checked against these sums. |
|
|
| ``` |
| 17fd4f09df786f1764a1033dc9dbba50e970197c2a87cffe3434f32729d8776a ckpt_5850.pt |
| 9a801f2363abe7ce68064d891f99d6c3ec7c7098943412e736db668c89790d12 ckpt_6000.pt |
| 4b082cf3b67b95eac50f2379e35b9b0c7ed75bf75260345f77bb4a82803bf580 ckpt_6150.pt |
| 833e1c9baec710e8aab43be0f8623467d66d1498da25764c9fd5f94b118c194c ckpt_6300.pt |
| 065458042ee08ea4c3154256d4405fce33aa4511a48265a11eed6494ed2c849b ckpt_6450.pt |
| 0c290af8dc9892d48d795219b120ab83c4e4e5294338494ab668dc11125410b3 ckpt_6600.pt |
| 970518a13efa13f65bfb6bc17db16b6b9daa010d7bfc615a799d07284d6fa38e ckpt_6750.pt |
| d094bef832864563e52371c681316220d41ce7415e9c0912cecbfbe81adcd0fe ckpt_6900.pt |
| f1fbdcfdf124ce961bed5c1ba3949fc61b52bbe1cfee9918655a313482a1b561 ckpt_7050.pt |
| d7cad9673f3ee706f3022ad6a97fa7593604806d4d25ba936c27885a8f115d97 ckpt_7150.pt |
| ``` |
|
|
| ## License |
|
|
| Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (`shared/core_sa`, `shared/legal_sa` of |
| slayer-pl-8x3b); see the dataset card for the licenses of the data. |
|
|
