File size: 4,545 Bytes
393ef98 8a24937 b8ac4d4 8a24937 393ef98 8a24937 393ef98 5e9303d 393ef98 a4d6156 d020584 8a24937 d020584 8a24937 393ef98 8a24937 5e9303d 393ef98 8a24937 393ef98 5930f7b 4463a8d 5a65acb 3ba51a4 cb9da23 0a509bc b8ac4d4 393ef98 8a24937 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | ---
license: apache-2.0
library_name: pytorch
tags:
- polish
- checkpoints
- training-history
---
# Slayer1B 1.03 — training checkpoints
Full trainer checkpoints of the run that produces Slayer1B 1.03: continued training of
[Slayer1B 1.02](https://huggingface.co/SlayerLab/Slayer1B-1.02) from its final checkpoint
([`ckpt_5720.pt`](https://huggingface.co/SlayerLab/Slayer1B-1.02-checkpoints), sha256 `6b2a9b11…`, constant learning
rate). Each file holds the model weights **and the optimizer state**.
**Status (27 October 2026): training finished at step 7150** (05:14:58 UTC, train loss 2.3955 at step 7150).
All checkpoints of this run, up to the final `ckpt_7150.pt`, are in this repository; their SHA-256 are listed below.
| | |
|---|---|
| Start | 1.02 `ckpt_5720.pt` (step 5720, constant lr 6e-4) |
| Steps | 7150 in total × 1 048 576 tokens = 7 497 318 400 tokens (+1430 steps = +1 499 463 680 tokens over 1.02) |
| Schedule | **constant lr 6e-4 to the end, no final decay** — the decay is deferred to the end of the whole Slayer1B line (pre-training continues on the whole prepared corpus) |
| Data | continues the pack-03 stream of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) (revision `d50df04a`) used by 1.02 from step 5390: steps 5721–7150 |
| Hardware | steps 5721–6750: 4 × NVIDIA RTX 4090 (24 GB); steps 6751–7150: 4 × NVIDIA L40S (48 GB), resumed from `ckpt_6750.pt` (sha256 `970518a1…`) with the same trainer, recipe, data order and global batch. The data order of the resume was verified against the original run (batch hash of zero-based iterations 6751 and 7149, i.e. native steps 6752 and 7150, identical). DDP; global batch 256 × 4096 tokens as in 1.00–1.02 (effective: 4 ranks × micro-batch 2 × gradient accumulation 32) |
| Checkpoints | every 150 steps from 5850 and the final step 7150; all are constant-lr states |
| Trainer | `train_gpt_ref_r24_1.py`: the 1.02 trainer, releasing the checkpoint tensors from GPU memory after resuming (needed on 24 GB cards); the model definition is unchanged |
| Training curve | steps up to 7150 (6860–7150 from the L40S resume): https://track.fabryka.ai/run/7ca4f84f-3b07-4525-8cde-f3a936dc3425 |
| Benchmarks 1.00–1.03 (lm-eval 0.4.13, 0-shot) | https://track.fabryka.ai/run/1426c0a8-2f07-5a03-80fc-92f7e922aa4b |
Training continues from `ckpt_7150.pt` on the rest of the prepared corpus (track:
https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5); those checkpoints belong to the next versions.
## Benchmark caveat
A leak scan of the whole training superset (SlayerLab/slayer-pl-8x3b@d50df04a, 22 603 122 documents) found n-gram
overlap with BoolQ, MMLU and ARC test items in the English shared part (`shared/en`: 1428 / 1429 / 1345 documents).
Overlap is not a measured contamination rate; the assessment is in progress. Read the BoolQ, MMLU and ARC numbers with
this caveat. From pack-04 on (first filtered update: zero-based 7192, native step 7193), the 5133 matching documents
are excluded from new training data; earlier steps, including 1.00–1.03, used the unfiltered data.
## File format
`torch.save` dictionary of the trainer: `model`, `opt` (Muon + AdamW), `step`, `config`. Load with
`torch.load(path, map_location="cpu", weights_only=True)`. Model class: `train_gpt_ref_r23.py` in the 1.00–1.02 model
repositories.
## SHA-256
Recorded off the training process at the moment each checkpoint is written, before upload; files are downloaded back
and checked against these sums.
```
17fd4f09df786f1764a1033dc9dbba50e970197c2a87cffe3434f32729d8776a ckpt_5850.pt
9a801f2363abe7ce68064d891f99d6c3ec7c7098943412e736db668c89790d12 ckpt_6000.pt
4b082cf3b67b95eac50f2379e35b9b0c7ed75bf75260345f77bb4a82803bf580 ckpt_6150.pt
833e1c9baec710e8aab43be0f8623467d66d1498da25764c9fd5f94b118c194c ckpt_6300.pt
065458042ee08ea4c3154256d4405fce33aa4511a48265a11eed6494ed2c849b ckpt_6450.pt
0c290af8dc9892d48d795219b120ab83c4e4e5294338494ab668dc11125410b3 ckpt_6600.pt
970518a13efa13f65bfb6bc17db16b6b9daa010d7bfc615a799d07284d6fa38e ckpt_6750.pt
d094bef832864563e52371c681316220d41ce7415e9c0912cecbfbe81adcd0fe ckpt_6900.pt
f1fbdcfdf124ce961bed5c1ba3949fc61b52bbe1cfee9918655a313482a1b561 ckpt_7050.pt
d7cad9673f3ee706f3022ad6a97fa7593604806d4d25ba936c27885a8f115d97 ckpt_7150.pt
```
## License
Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (`shared/core_sa`, `shared/legal_sa` of
slayer-pl-8x3b); see the dataset card for the licenses of the data.
|