Blame for README.md

---
license: apache-2.0
library_name: pytorch
tags:
- polish
- checkpoints
- training-history
---

# Slayer1B 1.07 — training checkpoints

Native training checkpoints of the run that produced [Slayer1B 1.07](https://huggingface.co/SlayerLab/Slayer1B-1.07),
for resuming or studying training (full trainer state, not a Transformers/safetensors package): pretraining continued
from the final 1.06 checkpoint `ckpt_13848.pt` (sha256 `1fc7396c…`, published in
[Slayer1B-1.06-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.06-checkpoints)). Each file holds the model
weights **and the optimizer state**.

**Status: training complete at step 22327 (6 November 2026).** The table below lists all 23 saved checkpoints of the
run, up to the final step 22327. The converted weights of the final
checkpoint (`model.safetensors` and the files needed to load it) are in this repository as well, see below, and in the
[Slayer1B 1.07 model repository](https://huggingface.co/SlayerLab/Slayer1B-1.07).
Every file listed was compared after upload with the checksum list written by the trainer: the LFS object id (sha256)
and the size match.

| | |
|---|---|
| Start | `ckpt_13848.pt` (final state of 1.06, constant lr 6e-4) |
| Steps | 13849 → 22327 (8 479 steps × 1 048 576 tokens), 256 sequences × 4096 tokens per step |
| Schedule | **constant lr 6e-4**, no decay |
| Data | [SlayerLab/slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain), 8 891 005 707 tokens, one pass |
| Hardware | 4 × NVIDIA RTX 4090, DDP, gradient accumulation 32; global batch identical to the 8-GPU runs |
| Interruptions | stopped on 4 November at step 17130 and resumed on 5 November from `ckpt_17000.pt`; steps 17001–17130 were trained twice, and the loss at step 17010 was identical in both runs. Stopped again on 5 November at step 19330 and resumed from the rolling checkpoint at step 19250 (not published); steps 19251–19330 were trained twice, and the loss at step 19260 was identical in both runs. Checkpoints up to step 17000 were saved before the first interruption, 18000–19000 between the two, later ones after the second resume |
| Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (unchanged) |
| Training curve and evaluations | https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925 |

## SHA-256

| step | sha256 |
|---:|---|
| 13998 | `f1ec1380a956f7fa002446902464df26314901d9cf2afe98ecbd00dec34d3529` |
| 14000 | `752d9d8ec48a6473be78dbccdebfd61621d3b24d78932b1b5f4616da415d28b8` |
| 15000 | `f89d7f3c6c223ec00ac4d85cb85546d08194808e419dcda3821ebd7ba4915ae1` |
| 15730 | `dd7170909294dbdec1282617a12d8c215037c27be5075e421a68ae6b1245e70b` |
| 16000 | `c18be246a5ae83be67a61fddf479fe6f906936ccd31f93dccf52fb4f03ca2b48` |
| 17000 | `c4961c9ced633fef2efd8d28cc16217154f82b25f4a178ce31865b4523eed95b` |
| 18000 | `fd6e2360badc1ed79e931ab36864440235d67d86c835592b6c9d788a9b9f0980` |
| 18590 | `6779c0bbfe84af003e5a708c1e9a02914e492a40c3f24278bbd096635a254467` |
| 19000 | `9ae730323e29b929f2ba0f753ee75be43626b804f1a5c452d5cee16d0dacfaf7` |
| 20000 | `33146051461e6102cb97f72b6d5a0913e228ca55f165fa9dc1db8eaa057aeefe` |
| 21000 | `05fc22d03e934f39bf70d57ac8f7cf20186df5aa4458250338f76d650c5a5233` |
| 21450 | `7c1dcd9e3d0d4bc03f296788c196eadf1e3855e76f414bb2ea2b96d5e9aff10d` |
| 21877 | `2fa5e3e0919b4d8a33b5fac717f593b6615866db1a090f538fc2f2e6bda03c68` |
| 21927 | `03b30237dcfabecc7f99fd1703835f7b3a03ce19fed2f3a44ad8578538621bc7` |
| 21977 | `49768faad2a36bdb20beab807a7a44e4c5660e7047644fa2826cc6f3b0c7326c` |
| 22000 | `8e5b5f33cedbe2b3523f1fe87470eb522d165926303e2eeb59fdd5852fd89b06` |
| 22027 | `a12862b21bde36b7a3a3ec47df5f27891f0b055ef3636400af99320d6a33051a` |
| 22077 | `ff2e2ca16028fb66c607b0696b34beee1e9e1089624056741b82d8099c461338` |
| 22127 | `793962376892ed6ea8ebb5add94b98b14487ef742cc78cfb907303cdf2b250b7` |
| 22177 | `f95c4a7df9262eee3b7121944583251e1c72c3bca5adaf2d1d9647977198d661` |
| 22227 | `ab4dedce91acdef64e689ef699bda59e548490b4a96b52b4d9faebbd37b4d61c` |
| 22277 | `f1d7fe0397d809cff8aade1c2e3f5668dff8e7ea778e9266afdcd0339adda257` |
| 22327 | `48cdb5eb16b69e0ee9a7a2ed72895ff47c0ba57aca0f4516b1f7e9cba286685c` |

Available training-time evaluations are in the
[Slayer1B 1.07 model card](https://huggingface.co/SlayerLab/Slayer1B-1.07).

## Converted weights of the final checkpoint (step 22327)

`model.safetensors` (float32 weights only, no optimizer state, 4 295 670 676 bytes), `config.json`, `tokenizer.json`,
`load_slayer.py` (loader), `train_gpt_ref_r23.py` (model definition) and `SHA256SUMS` for these five files. They are
the same files as in the [Slayer1B 1.07 model repository](https://huggingface.co/SlayerLab/Slayer1B-1.07). `config.json`
records the sha256 of the source checkpoint `ckpt_22327.pt` (`48cdb5eb…`, listed above).

```python
import load_slayer
model = load_slayer.load(".")  # float32, eval mode, on CPU
```

## File format

The files are PyTorch pickles (`torch.save` dictionary of the trainer: `model`, `opt` (Muon + AdamW), `step`,
`config`); they contain only dictionaries, numbers, strings and tensors. Load with
`torch.load(path, map_location="cpu", weights_only=True)` (checked with torch 2.13 on `ckpt_17000.pt`; `mmap=True`
avoids reading the whole file into memory). Model class: `train_gpt_ref_r23.py` in the model repositories
(1.00–1.06). Every file is 8 860 287 151 bytes.

## License

Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (the share-alike parts of the Polish core
and legal texts, repeated from slayer-pl-8x3b); see the dataset cards of
[slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) and
[slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain) for the licenses of the data.