File size: 3,449 Bytes
187ab77 fd7e55d 187ab77 e9f0f81 187ab77 275794b 187ab77 e9f0f81 187ab77 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: apache-2.0
library_name: pytorch
tags:
- polish
- checkpoints
- training-history
---
# Slayer1B 1.01 — training checkpoints
Full trainer checkpoints of the run that produced Slayer1B 1.01: continued training of
[Slayer1B 1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00) from its last checkpoint before the learning-rate decay
([`ckpt_1050.pt`](https://huggingface.co/SlayerLab/Slayer1B-1.00-checkpoints)), with a new total budget of twice the
tokens per parameter of 1.00. Each file holds the model weights **and the optimizer state**.
| | |
|---|---|
| Start | 1.00 `ckpt_1050.pt` (step 1050, constant lr 6e-4) |
| Steps | 2860 in total × 1 048 576 tokens = 2 998 927 360 tokens (≈ 2.8 tokens per parameter; 1.00: 1430 steps) |
| Schedule | constant lr 6e-4 through step 2289, then 1 − sqrt decay over steps 2290–2860 to lr / 10 |
| Data | steps 1051–1801: the same pack-00 stream as 1.00 (same windows); steps 1802–2860: pack-01 of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) (fresh web slice; shared components disjoint from those already used) |
| Last checkpoint before the decay | **`ckpt_2250.pt`** — starting point for further continued training |
| Training curve and evaluations during training | https://track.fabryka.ai/run/4934429a-d393-42e0-8aac-03eae6b71ff7 |
Checkpoints every 150 steps from 1200 to 2850, plus the final step 2860. Released model (weights in safetensors) and
model card: [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01).
## File format
`torch.save` dictionary of the trainer `train_gpt_ref_r23.py` (in the 1.00 model repository): `model`, `opt` (Muon +
AdamW), `step`, `config`. Load with `torch.load(path, map_location="cpu", weights_only=True)`.
## Run record (`run/`)
`metrics.jsonl` (training loss and learning rate every 10 steps), `train.log`, `train.out`, `args.txt` (trainer
arguments), `checkpoints_SHA256SUMS` (written on the node at save time), data manifests and sums of both token streams
(`wsad_v2_*`: pack-00 stream, `wsad_p01_*`: pack-01 stream), `13gram_p01.json` (13-gram overlap of the pack-01 stream
with the evaluation sets), `pip_freeze.txt` (software environment).
## SHA-256
Recorded off the training node at the moment each checkpoint was written, before upload; files are downloaded back and
checked against these sums.
```
3637f1d90f60c2151724ad88b4b7c52925c87b7330a7680974b9093e8c1b4021 ckpt_1200.pt
d6d75584ffbd7261ed23f489ed4927d3e33fc3a6fef3bd9d3ad46b309b63daf4 ckpt_1350.pt
3b2af30f8f4331f7bf549b1e7f0b2643ac2ec8ad3369a42b579d578930ba2934 ckpt_1500.pt
db254bfb77e302fc618f9cc7e3c4e23f1e3d6923386e54ad5fb930ff78a04bc4 ckpt_1650.pt
9591dfa0d41f68b9e5cc40db2caf6f601b102c31b37c76d2ddeffa9f26cec49e ckpt_1800.pt
6b8037e19b3fc42bb1c70599754749cdbb6a507a27fa4e2c3e316d676e969555 ckpt_1950.pt
b46f5e02f74dab2db1055fa009668c784411f403e01198dfff36deae9a0200f5 ckpt_2100.pt
aeddfb023abb11c8654b47c728c9680c232d1b6568697e3a1eee8128efd37054 ckpt_2250.pt
616df6070fbc6e338e106d4b535030d06767063168e80494b67f508005904b73 ckpt_2400.pt
82fd3d07914308411a2ddec965ce6881f296b8bb6956369096534bdc2c45b0c8 ckpt_2550.pt
e2b5b29f05d3113c0bc95a5a4d74fe07fee057031068bc82844f129a341c09f0 ckpt_2700.pt
decd0f02487fa15c65aaaaf9e2c2320f196571efa0337c8c588705915f7fd315 ckpt_2850.pt
b0a288798ae614c38e37ff8176a293f6650cec2221d59e46d5116afac7c31eba ckpt_2860.pt
```
## License
Apache-2.0.
|