| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - polish |
| - checkpoints |
| - training-history |
| --- |
| |
| # Slayer1B 1.00 — training checkpoints |
|
|
| Full trainer checkpoints from the training run of [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00): |
| every 150 steps, 10 files. Each file holds the model weights **and the optimizer state**, so training can be resumed |
| from any of them. |
|
|
| | | | |
| |---|---| |
| | Model | 1 073 912 851 parameters, 20 layers × d_model 2048 × 16 heads (details in the model card) | |
| | Steps | 1430 × 1 048 576 tokens = 1 499 463 680 tokens | |
| | Schedule | warmup 150 steps, constant lr 6e-4, then 1 − sqrt decay from step 1144 to lr / 10 | |
| | Last checkpoint before the decay | **`ckpt_1050.pt`** — used as the starting point of [Slayer1B 1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01-checkpoints) | |
| | Training curve | https://track.fabryka.ai/run/48744ed2-3155-4294-b899-a6fda1f851da | |
| |
| ## File format |
| |
| `torch.save` dictionary of the trainer `train_gpt_ref_r23.py` (in the model repository): |
| `model` (state dict), `opt` (Muon for hidden matrices + AdamW for the rest), `step`, `config`. |
| Load with `torch.load(path, map_location="cpu", weights_only=True)`; resume with the trainer's `--resume`. |
| Each file is 8 860 285 999 bytes. |
|
|
| ## SHA-256 |
|
|
| A full SHA-256 recorded off the machine before the copy exists for steps 150, 300, 750, 1050 and 1430. For 450, 600, |
| 900, 1200 and 1350 only the first 8 hex characters were recorded beforehand; the full sums in the table for these steps |
| were computed from the files downloaded back from this repository and match those prefixes. |
|
|
| ``` |
| 8f113dd02835bf3cc6737980f1664464d24c26168c0b245e9b8ed365ce4fb6c9 ckpt_150.pt |
| 2765fc1b71989ac85cb44084e8083bd86cf6a33ba67b7df1453afdaabd355f16 ckpt_300.pt |
| d940b909d2d0db48e4e0184405c96a472ec44f6d676798e358dd44a22c49f85f ckpt_450.pt |
| 13b5e60a27e4732cb1f48d144d2651408d0a3bb4b6c6dabf5d165a664622df56 ckpt_600.pt |
| 3e5b03e6d546b139060503b932419c5ee04f1241822bb36b29951769f1a7e1d5 ckpt_750.pt |
| 6fa366b77f4d3460cd4ecaf5bf86f299eadf2bd3d38ac1728e7667095756099b ckpt_900.pt |
| 236108e8f204995f4adcfea7d9a99437d230c1af12708c64c68c74afbc2910fd ckpt_1050.pt |
| 2fe897999f7fc76a3805c9234e69f56df8743b03f13c0e0f12752d575fbc5102 ckpt_1200.pt |
| 551728562d7121f94980be587ed03dfb81619f968a91de1f29bbab9b42646cf7 ckpt_1350.pt |
| 7d092d4d0bf838da1115a5369bf0a6729fe4c170e0d5ad303873f7320d6c1866 ckpt_1430.pt |
| ``` |
|
|
| ## License |
|
|
| Apache-2.0, as the model. Training data: see the model card. |
|
|
