Card: Slayer1B 1.04 checkpoints (in progress)

README.md

@@ -0,0 +1,48 @@
+---
+license: apache-2.0
+library_name: pytorch
+tags:
+- polish
+- checkpoints
+- training-history
+---
+
+# Slayer1B 1.04 — training checkpoints (in progress)
+
+Full trainer checkpoints of the run that produces Slayer1B 1.04: continued pre-training of
+[Slayer1B 1.03](https://huggingface.co/SlayerLab/Slayer1B-1.03) from its final checkpoint
+([`ckpt_7150.pt`](https://huggingface.co/SlayerLab/Slayer1B-1.03-checkpoints), sha256 `d7cad967…`, constant learning
+rate) on the rest of the prepared corpus. Each file holds the model weights **and the optimizer state**. Release 1.04 is
+the state at native step 10010; this repository collects every checkpoint of the steps 7151–10010.
+
+| | |
+|---|---|
+| Start | 1.03 `ckpt_7150.pt` (step 7150, constant lr 6e-4) |
+| Steps | 7151 → 10010, 1 048 576 tokens per step (4 ranks × micro-batch 2 × gradient accumulation 32 × 4096 tokens) |
+| Schedule | **constant lr 6e-4**, no decay; the decay is applied once, at the end of the whole Slayer1B line |
+| Data | [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) revision `d50df04a`: tail of pack-03 (native 7151–7192), then pack-04 from native step 7193 (first filtered update: zero-based 7192) and pack-05 from native 8991 |
+| Leak-scan exclusion | from pack-04 on, the 5133 documents with n-gram overlap with BoolQ / MMLU / ARC test items are excluded (list sha256 `84caed78…`); steps up to 7192 used the unfiltered data |
+| Hardware | 4 × NVIDIA L40S (48 GB), DDP |
+| Checkpoints | every 50 steps; all are constant-lr states. Training paused (budget) at step 8300 and resumed from `ckpt_8300.pt`; loss at the first resumed steps equal to the pre-pause values to four decimals |
+| Trainer | `train_gpt_ref_r24_1.py` (sha256 `881cb055…`), model definition unchanged (`train_gpt_ref_r23.py`) |
+| Training curve and evaluations | https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5 |
+
+Checkpoint files are added as they are archived; each upload is verified by a full anonymous download and SHA-256
+comparison against the sum recorded when the checkpoint was written.
+
+## Evaluations
+
+Leaderboard protocol (ARC-Easy, BLiMP, WikiText-2 byte perplexity) is computed automatically for every checkpoint and
+published on the training-curve page above. BoolQ, MMLU and ARC numbers of earlier releases carry the leak-scan caveat
+described in the [1.03 card](https://huggingface.co/SlayerLab/Slayer1B-1.03).
+
+## File format
+
+`torch.save` dictionary of the trainer: `model`, `opt` (Muon + AdamW), `step`, `config`. Load with
+`torch.load(path, map_location="cpu", weights_only=True)`. Model class: `train_gpt_ref_r23.py` in the 1.00–1.03 model
+repositories.
+
+## License
+
+Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (`shared/core_sa`, `shared/legal_sa` of
+slayer-pl-8x3b); see the dataset card for the licenses of the data.