Model card: checkpoints listed so far (up to ckpt_17000.pt)

README.md

@@ -0,0 +1,56 @@
+---
+license: apache-2.0
+library_name: pytorch
+tags:
+- polish
+- checkpoints
+- training-history
+---
+
+# Slayer1B 1.07 — training checkpoints
+
+Native training checkpoints of the run that is producing [Slayer1B 1.07](https://huggingface.co/SlayerLab/Slayer1B-1.07),
+for resuming or studying training (full trainer state, not a Transformers/safetensors package): pretraining continued
+from the final 1.06 checkpoint `ckpt_13848.pt` (sha256 `1fc7396c…`, published in
+[Slayer1B-1.06-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.06-checkpoints)). Each file holds the model
+weights **and the optimizer state**.
+
+**Status: training in progress.** The table below lists only the files that are in this repository now; further
+checkpoints are added as training saves them (the planned end is step 22327, which will also be released as the final
+1.07 weights in the model repository). Every file listed was compared after upload with the checksum list written by
+the trainer: the LFS object id (sha256) and the size match.
+
+| | |
+|---|---|
+| Start | `ckpt_13848.pt` (final state of 1.06, constant lr 6e-4) |
+| Steps | 13849 → 22327 planned (8 479 steps × 1 048 576 tokens), 256 sequences × 4096 tokens per step |
+| Schedule | **constant lr 6e-4**, no decay |
+| Data | [SlayerLab/slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain), 8 891 005 707 tokens, one pass |
+| Hardware | 4 × NVIDIA RTX 4090, DDP, gradient accumulation 32; global batch identical to the 8-GPU runs |
+| Interruption | stopped on 4 November at step 17130 and resumed on 5 November from `ckpt_17000.pt`; steps 17001–17130 were trained twice, and the loss at step 17010 was identical in both runs. Checkpoints up to step 17000 were saved before the interruption (the resume started from `ckpt_17000.pt`); later checkpoints were saved after the resume |
+| Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (unchanged) |
+| Training curve and evaluations | https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925 |
+
+## SHA-256
+
+| step | sha256 |
+|---:|---|
+| 17000 | `c4961c9ced633fef2efd8d28cc16217154f82b25f4a178ce31865b4523eed95b` |
+
+Available training-time evaluations are in the
+[Slayer1B 1.07 model card](https://huggingface.co/SlayerLab/Slayer1B-1.07).
+
+## File format
+
+The files are PyTorch pickles (`torch.save` dictionary of the trainer: `model`, `opt` (Muon + AdamW), `step`,
+`config`); they contain only dictionaries, numbers, strings and tensors. Load with
+`torch.load(path, map_location="cpu", weights_only=True)` (checked with torch 2.13 on `ckpt_17000.pt`; `mmap=True`
+avoids reading the whole file into memory). Model class: `train_gpt_ref_r23.py` in the model repositories
+(1.00–1.06). Every file is 8 860 287 151 bytes.
+
+## License
+
+Apache-2.0 for the checkpoints. The training data includes CC BY-SA sources (the share-alike parts of the Polish core
+and legal texts, repeated from slayer-pl-8x3b); see the dataset cards of
+[slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) and
+[slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain) for the licenses of the data.