Add model card (contents, checkpoint format, SHA-256)
Browse files
README.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- polish
|
| 6 |
+
- checkpoints
|
| 7 |
+
- training-history
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# Slayer1B 1.00 — training checkpoints
|
| 11 |
+
|
| 12 |
+
Full trainer checkpoints from the training run of [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00):
|
| 13 |
+
every 150 steps, 10 files. Each file holds the model weights **and the optimizer state**, so training can be resumed
|
| 14 |
+
from any of them.
|
| 15 |
+
|
| 16 |
+
| | |
|
| 17 |
+
|---|---|
|
| 18 |
+
| Model | 1 073 912 851 parameters, 20 layers × d_model 2048 × 16 heads (details in the model card) |
|
| 19 |
+
| Steps | 1430 × 1 048 576 tokens = 1 499 463 680 tokens |
|
| 20 |
+
| Schedule | warmup 150 steps, constant lr 6e-4, then 1 − sqrt decay from step 1144 to lr / 10 |
|
| 21 |
+
| Last checkpoint before the decay | **`ckpt_1050.pt`** — used as the starting point of [Slayer1B 1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01-checkpoints) |
|
| 22 |
+
| Training curve | https://track.fabryka.ai/run/48744ed2-3155-4294-b899-a6fda1f851da |
|
| 23 |
+
|
| 24 |
+
## File format
|
| 25 |
+
|
| 26 |
+
`torch.save` dictionary of the trainer `train_gpt_ref_r23.py` (in the model repository):
|
| 27 |
+
`model` (state dict), `opt` (Muon for hidden matrices + AdamW for the rest), `step`, `config`.
|
| 28 |
+
Load with `torch.load(path, map_location="cpu", weights_only=True)`; resume with the trainer's `--resume`.
|
| 29 |
+
Each file is 8 860 285 999 bytes.
|
| 30 |
+
|
| 31 |
+
## SHA-256
|
| 32 |
+
|
| 33 |
+
The sums below were recorded on the training node before the files were copied; every file on this repository was
|
| 34 |
+
downloaded back and checked against them.
|
| 35 |
+
|
| 36 |
+
```
|
| 37 |
+
8f113dd02835bf3cc6737980f1664464d24c26168c0b245e9b8ed365ce4fb6c9 ckpt_150.pt
|
| 38 |
+
2765fc1b71989ac85cb44084e8083bd86cf6a33ba67b7df1453afdaabd355f16 ckpt_300.pt
|
| 39 |
+
d940b909d2d0db48e4e0184405c96a472ec44f6d676798e358dd44a22c49f85f ckpt_450.pt
|
| 40 |
+
13b5e60a27e4732cb1f48d144d2651408d0a3bb4b6c6dabf5d165a664622df56 ckpt_600.pt
|
| 41 |
+
3e5b03e6d546b139060503b932419c5ee04f1241822bb36b29951769f1a7e1d5 ckpt_750.pt
|
| 42 |
+
6fa366b77f4d3460cd4ecaf5bf86f299eadf2bd3d38ac1728e7667095756099b ckpt_900.pt
|
| 43 |
+
236108e8f204995f4adcfea7d9a99437d230c1af12708c64c68c74afbc2910fd ckpt_1050.pt
|
| 44 |
+
2fe897999f7fc76a3805c9234e69f56df8743b03f13c0e0f12752d575fbc5102 ckpt_1200.pt
|
| 45 |
+
551728562d7121f94980be587ed03dfb81619f968a91de1f29bbab9b42646cf7 ckpt_1350.pt
|
| 46 |
+
7d092d4d0bf838da1115a5369bf0a6729fe4c170e0d5ad303873f7320d6c1866 ckpt_1430.pt
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
## License
|
| 50 |
+
|
| 51 |
+
Apache-2.0, as the model. Training data: see the model card.
|