File size: 9,224 Bytes
b1e7883 9134eb9 b1e7883 a78f425 db4857f b1e7883 9134eb9 b1e7883 db4857f b1e7883 ed13d53 b1e7883 3add7eb ed13d53 64bc745 b1e7883 a78f425 a090b54 a78f425 a090b54 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | ---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
- SlayerLab/slayer-1.07-pretrain
---
# Slayer1B 1.07
Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
replayed from the 1.06 data), at the same constant learning rate and global batch.
**Status: training complete at step 22327 (6 November 2026). This repository contains the converted weights of the
final checkpoint (step 22327)**; see Files below. All 15 evaluated checkpoints of the run, from the 1.06 starting point
(13848) to the final checkpoint (22327), are in the training-time evaluation table below. Native training checkpoints
(raw PyTorch training state `ckpt_N.pt`, not a ready-to-use model) are published file by file in
[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
- Previous version: [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06) (step 13848,
14 520 680 448 tokens, one full pass over [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b))
- Training data of 1.07: [SlayerLab/slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain)
- Training curve and training-time evaluation: https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925
[Full-size chart](https://huggingface.co/SlayerLab/Slayer1B-1.07/resolve/main/slayer1b_107_training.png)
## Training
- Start: Slayer1B 1.06 checkpoint `ckpt_13848.pt`
(sha256 `1fc7396ca1790805128244b2f827e02b9fe88e352534cd887d0c488f121d1e4d`).
- Same model, tokenizer (v6, 32 768 tokens, sha256 `0640d3bd…`) and global batch as 1.06: 256 sequences × 4096 tokens
= 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
- Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
- Length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
(2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
resumed within two minutes from the checkpoint at step 19250; the loss at step 19260 was again identical (2.3105).
## Training data of 1.07
8 891 005 707 tokens in 9 files, after removing documents that overlap evaluation sets (13-gram scan) and after
masking personal data (e-mail addresses, phone numbers, national ID numbers) with the same procedure as in 1.06.
| source | language, type | documents (built) | tokens (built) | tokens in training | license |
|---|---|---:|---:|---:|---|
| HPLT (Polish web, parts not used in 1.06) | PL, general web | 2 448 525 | 3 119 999 086 | 3 110 430 927 | compilation CC0; texts used under the text-and-data-mining exception (Art. 4, Directive 2019/790) |
| Polish core texts, repeated from 1.06 | PL, curated texts | 3 165 531 | 999 785 312 | 982 781 646 | per source, as in slayer-pl-8x3b |
| Polish legal texts, repeated from 1.06 | PL, legal | 261 590 | 1 004 151 322 | 911 514 295 | per source, as in slayer-pl-8x3b |
| FineWeb-Edu (from minimal-en-corpus-5b) | EN, educational web | 840 133 | 978 166 858 | one file, 1 650 618 814 together | ODC-By |
| FineMath | EN, mathematics | 152 623 | 237 458 786 | (same file) | ODC-By |
| open-web-math | EN, mathematics from the web | 86 621 | 225 412 736 | (same file) | ODC-By |
| DCLM-baseline (subset) | EN, general web | 177 946 | 244 996 577 | (same file) | CC BY 4.0 |
| FineWeb-Edu `sample/10BT` | EN, educational web | 2 035 410 | 2 254 202 363 | 2 235 660 025 | ODC-By |
Documents in the published training files (after leak removal): Polish web 2 445 601, Polish core 3 151 339,
Polish legal 258 328, English mix 1 249 258, FineWeb-Edu `sample/10BT` 2 029 345 (9 133 871 in total).
Share of training tokens: Polish web 35.0 %, Polish core 11.1 %, Polish legal 10.3 %, English 43.7 %.
The Polish core and legal parts are a random (hash-ordered) subset of what 1.06 already saw; about 36 % of the 1.06
core tokens and 68 % of the 1.06 legal tokens are seen a second time, the rest of the data once. English documents
whose exact normalized text was already in the 1.06 English data were excluded.
## Training-time evaluation
Same automatic evaluator and the same zero-shot protocol as the 1.06 card (BLiMP, ARC-Easy, WikiText-2, MultiBLiMP-PL,
ARC-Easy-PL eu20). Checkpoints 13848–19000 and 22327 were evaluated on CPU, checkpoints 20000–22000 on an NVIDIA RTX 4090
(rented). On both devices the 1.06 checkpoint (step 13848) reproduces the 1.06 card values at card rounding, and on the
GPU the published Slayer1B 1.02 also reproduces its card values, so all rows are on the same scale. These are **not** the
lm-evaluation-harness benchmark used in the Results table of 1.00–1.03.
| step | BLiMP (EN), % | ARC-Easy (EN), % | WikiText-2 byte PPL ↓ | WikiText-2 token PPL ↓ | MultiBLiMP (PL), % | ARC-Easy (PL, eu20), % | device |
|---:|---:|---:|---:|---:|---:|---:|---|
| 13848 (1.06) | 76.12 | 52.99 | 2.2849 | 30.03 | 98.41 | 45.03 | CPU |
| 13998 | 76.29 | 54.34 | 2.2618 | 28.80 | 98.66 | 45.37 | CPU |
| 14000 | 76.33 | 54.21 | 2.2601 | 28.71 | 98.59 | 44.91 | CPU |
| 15000 | 77.94 | 54.59 | 2.2321 | 27.28 | 98.62 | 45.92 | CPU |
| 15730 | 77.32 | 55.64 | 2.2237 | 26.85 | 98.62 | 45.75 | CPU |
| 16000 | 77.13 | 56.73 | 2.2205 | 26.70 | 98.81 | 46.25 | CPU |
| 17000 | 77.61 | 56.48 | 2.2138 | 26.37 | 98.87 | 45.41 | CPU |
| 18000 | 77.88 | 56.52 | 2.2042 | 25.90 | 98.84 | 46.21 | CPU |
| 18590 | 77.46 | 56.52 | 2.2145 | 26.40 | 98.62 | 47.14 | CPU |
| 19000 | 77.74 | 56.65 | 2.2029 | 25.84 | 98.53 | 46.80 | CPU |
| 20000 | 77.68 | 56.40 | 2.1940 | 25.41 | 98.66 | 46.55 | RTX 4090 |
| 21000 | 77.53 | 56.69 | 2.1813 | 24.81 | 98.62 | 47.01 | RTX 4090 |
| 21450 | 77.44 | 58.29 | 2.1867 | 25.06 | 98.75 | 47.31 | RTX 4090 |
| 22000 | 77.16 | 57.28 | 2.1794 | 24.72 | 98.75 | 46.97 | RTX 4090 |
| **22327 (final)** | **77.90** | **58.67** | **2.1837** | **24.92** | **98.84** | **46.93** | CPU |
ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
checkpoints are within noise. From 21000 on, WikiText-2 byte perplexity stays between 2.179 and 2.187 instead of
falling steadily (lowest 2.1794 at 22000; 2.1837 at the final step): the run used a constant learning rate without decay,
so the last checkpoint is where training stopped, not a converged minimum. The final checkpoint has the highest English
ARC-Easy accuracy of the run (58.67, +2.0 points over 19000). Each point is a single measurement. The training loss in the figure is measured
on the 1.07 data mixture (44 % English), so it is not directly comparable with the training loss of 1.00–1.06.
The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files
were scanned for 13-gram overlap with the test items of the evaluation suite before training, and documents with any
hit were removed. Exact matches of short answers (below 13 words) were counted but not removed, and one evaluation set
(EWoK, distributed as a password-protected archive) was not part of the scan. The Slayer 1.06 core includes the source
passage corpora of PolQA and PoQuAD; passages with a 13-gram hit were removed and the rest remain in the replay.
## Files
`model.safetensors` (weights, float32, unchanged from the checkpoint), `config.json`, `tokenizer.json`, `load_slayer.py`
(loader), `train_gpt_ref_r23.py` and `SHA256SUMS`. The `config.json` records the sha256 of the source checkpoint
`ckpt_22327.pt` (`48cdb5eb16b69e0ee9a7a2ed72895ff47c0ba57aca0f4516b1f7e9cba286685c`, also published in the checkpoints
repository). The conversion checks that the model built by `load_slayer.py` gives the same logits as the model loaded
from the checkpoint (maximum absolute difference 0).
The file layout and the loader are the same as in [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06):
```python
import load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".") # float32, eval mode, on CPU
tok = Tokenizer.from_file("tokenizer.json") # id 0 is end-of-text
```
`train_gpt_ref_r23.py` is the model definition (`GPT` class) imported by `load_slayer.py`; `trainer_sha256` in
`config.json` refers to this file. 1.07 was trained with `train_gpt_ref` r24.2
(sha256 `46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc`); the architecture is unchanged from 1.00–1.06.
|