Slayer1B 1.07 card: Files - logits check, trainer r24.2

a090b54 verified about 5 hours ago

9.22 kB

---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
- SlayerLab/slayer-1.07-pretrain
---
# Slayer1B 1.07

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
replayed from the 1.06 data), at the same constant learning rate and global batch.

**Status: training complete at step 22327 (6 November 2026). This repository contains the converted weights of the
final checkpoint (step 22327)**; see Files below. All 15 evaluated checkpoints of the run, from the 1.06 starting point
(13848) to the final checkpoint (22327), are in the training-time evaluation table below. Native training checkpoints
(raw PyTorch training state `ckpt_N.pt`, not a ready-to-use model) are published file by file in
[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.

- Previous version: [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06) (step 13848,
  14 520 680 448 tokens, one full pass over [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b))
- Training data of 1.07: [SlayerLab/slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain)
- Training curve and training-time evaluation: https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925

![Slayer1B 1.07 training loss and training-time evaluation](slayer1b_107_training.png)

[Full-size chart](https://huggingface.co/SlayerLab/Slayer1B-1.07/resolve/main/slayer1b_107_training.png)

## Training

- Start: Slayer1B 1.06 checkpoint `ckpt_13848.pt`
  (sha256 `1fc7396ca1790805128244b2f827e02b9fe88e352534cd887d0c488f121d1e4d`).
- Same model, tokenizer (v6, 32 768 tokens, sha256 `0640d3bd…`) and global batch as 1.06: 256 sequences × 4096 tokens
  = 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
- Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
  about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
- Length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
  from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
  (2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
  resumed within two minutes from the checkpoint at step 19250; the loss at step 19260 was again identical (2.3105).

## Training data of 1.07

8 891 005 707 tokens in 9 files, after removing documents that overlap evaluation sets (13-gram scan) and after
masking personal data (e-mail addresses, phone numbers, national ID numbers) with the same procedure as in 1.06.

| source | language, type | documents (built) | tokens (built) | tokens in training | license |
|---|---|---:|---:|---:|---|
| HPLT (Polish web, parts not used in 1.06) | PL, general web | 2 448 525 | 3 119 999 086 | 3 110 430 927 | compilation CC0; texts used under the text-and-data-mining exception (Art. 4, Directive 2019/790) |
| Polish core texts, repeated from 1.06 | PL, curated texts | 3 165 531 | 999 785 312 | 982 781 646 | per source, as in slayer-pl-8x3b |
| Polish legal texts, repeated from 1.06 | PL, legal | 261 590 | 1 004 151 322 | 911 514 295 | per source, as in slayer-pl-8x3b |
| FineWeb-Edu (from minimal-en-corpus-5b) | EN, educational web | 840 133 | 978 166 858 | one file, 1 650 618 814 together | ODC-By |
| FineMath | EN, mathematics | 152 623 | 237 458 786 | (same file) | ODC-By |
| open-web-math | EN, mathematics from the web | 86 621 | 225 412 736 | (same file) | ODC-By |
| DCLM-baseline (subset) | EN, general web | 177 946 | 244 996 577 | (same file) | CC BY 4.0 |
| FineWeb-Edu `sample/10BT` | EN, educational web | 2 035 410 | 2 254 202 363 | 2 235 660 025 | ODC-By |

Documents in the published training files (after leak removal): Polish web 2 445 601, Polish core 3 151 339,
Polish legal 258 328, English mix 1 249 258, FineWeb-Edu `sample/10BT` 2 029 345 (9 133 871 in total).

Share of training tokens: Polish web 35.0 %, Polish core 11.1 %, Polish legal 10.3 %, English 43.7 %.
The Polish core and legal parts are a random (hash-ordered) subset of what 1.06 already saw; about 36 % of the 1.06
core tokens and 68 % of the 1.06 legal tokens are seen a second time, the rest of the data once. English documents
whose exact normalized text was already in the 1.06 English data were excluded.

## Training-time evaluation

Same automatic evaluator and the same zero-shot protocol as the 1.06 card (BLiMP, ARC-Easy, WikiText-2, MultiBLiMP-PL,
ARC-Easy-PL eu20). Checkpoints 13848–19000 and 22327 were evaluated on CPU, checkpoints 20000–22000 on an NVIDIA RTX 4090
(rented). On both devices the 1.06 checkpoint (step 13848) reproduces the 1.06 card values at card rounding, and on the
GPU the published Slayer1B 1.02 also reproduces its card values, so all rows are on the same scale. These are **not** the
lm-evaluation-harness benchmark used in the Results table of 1.00–1.03.

| step | BLiMP (EN), % | ARC-Easy (EN), % | WikiText-2 byte PPL ↓ | WikiText-2 token PPL ↓ | MultiBLiMP (PL), % | ARC-Easy (PL, eu20), % | device |
|---:|---:|---:|---:|---:|---:|---:|---|
| 13848 (1.06) | 76.12 | 52.99 | 2.2849 | 30.03 | 98.41 | 45.03 | CPU |
| 13998 | 76.29 | 54.34 | 2.2618 | 28.80 | 98.66 | 45.37 | CPU |
| 14000 | 76.33 | 54.21 | 2.2601 | 28.71 | 98.59 | 44.91 | CPU |
| 15000 | 77.94 | 54.59 | 2.2321 | 27.28 | 98.62 | 45.92 | CPU |
| 15730 | 77.32 | 55.64 | 2.2237 | 26.85 | 98.62 | 45.75 | CPU |
| 16000 | 77.13 | 56.73 | 2.2205 | 26.70 | 98.81 | 46.25 | CPU |
| 17000 | 77.61 | 56.48 | 2.2138 | 26.37 | 98.87 | 45.41 | CPU |
| 18000 | 77.88 | 56.52 | 2.2042 | 25.90 | 98.84 | 46.21 | CPU |
| 18590 | 77.46 | 56.52 | 2.2145 | 26.40 | 98.62 | 47.14 | CPU |
| 19000 | 77.74 | 56.65 | 2.2029 | 25.84 | 98.53 | 46.80 | CPU |
| 20000 | 77.68 | 56.40 | 2.1940 | 25.41 | 98.66 | 46.55 | RTX 4090 |
| 21000 | 77.53 | 56.69 | 2.1813 | 24.81 | 98.62 | 47.01 | RTX 4090 |
| 21450 | 77.44 | 58.29 | 2.1867 | 25.06 | 98.75 | 47.31 | RTX 4090 |
| 22000 | 77.16 | 57.28 | 2.1794 | 24.72 | 98.75 | 46.97 | RTX 4090 |
| **22327 (final)** | **77.90** | **58.67** | **2.1837** | **24.92** | **98.84** | **46.93** | CPU |

ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
checkpoints are within noise. From 21000 on, WikiText-2 byte perplexity stays between 2.179 and 2.187 instead of
falling steadily (lowest 2.1794 at 22000; 2.1837 at the final step): the run used a constant learning rate without decay,
so the last checkpoint is where training stopped, not a converged minimum. The final checkpoint has the highest English
ARC-Easy accuracy of the run (58.67, +2.0 points over 19000). Each point is a single measurement. The training loss in the figure is measured
on the 1.07 data mixture (44 % English), so it is not directly comparable with the training loss of 1.00–1.06.

The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files
were scanned for 13-gram overlap with the test items of the evaluation suite before training, and documents with any
hit were removed. Exact matches of short answers (below 13 words) were counted but not removed, and one evaluation set
(EWoK, distributed as a password-protected archive) was not part of the scan. The Slayer 1.06 core includes the source
passage corpora of PolQA and PoQuAD; passages with a 13-gram hit were removed and the rest remain in the replay.

## Files

`model.safetensors` (weights, float32, unchanged from the checkpoint), `config.json`, `tokenizer.json`, `load_slayer.py`
(loader), `train_gpt_ref_r23.py` and `SHA256SUMS`. The `config.json` records the sha256 of the source checkpoint
`ckpt_22327.pt` (`48cdb5eb16b69e0ee9a7a2ed72895ff47c0ba57aca0f4516b1f7e9cba286685c`, also published in the checkpoints
repository). The conversion checks that the model built by `load_slayer.py` gives the same logits as the model loaded
from the checkpoint (maximum absolute difference 0).
The file layout and the loader are the same as in [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06):

```python
import load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")            # float32, eval mode, on CPU
tok = Tokenizer.from_file("tokenizer.json")  # id 0 is end-of-text
```

`train_gpt_ref_r23.py` is the model definition (`GPT` class) imported by `load_slayer.py`; `trainer_sha256` in
`config.json` refers to this file. 1.07 was trained with `train_gpt_ref` r24.2
(sha256 `46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc`); the architecture is unchanged from 1.00–1.06.

	---
	language:
	- pl
	- en
	license: apache-2.0
	library_name: pytorch
	pipeline_tag: text-generation
	tags:
	- polish
	- causal-lm
	- from-scratch
	datasets:
	- SlayerLab/slayer-pl-8x3b
	- SlayerLab/slayer-1.07-pretrain
	---

	# Slayer1B 1.07

	Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
	Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
	replayed from the 1.06 data), at the same constant learning rate and global batch.

	**Status: training complete at step 22327 (6 November 2026). This repository contains the converted weights of the
	final checkpoint (step 22327)**; see Files below. All 15 evaluated checkpoints of the run, from the 1.06 starting point
	(13848) to the final checkpoint (22327), are in the training-time evaluation table below. Native training checkpoints
	(raw PyTorch training state `ckpt_N.pt`, not a ready-to-use model) are published file by file in
	[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.

	- Previous version: [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06) (step 13848,
	14 520 680 448 tokens, one full pass over [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b))
	- Training data of 1.07: [SlayerLab/slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain)
	- Training curve and training-time evaluation: https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925

	![Slayer1B 1.07 training loss and training-time evaluation](slayer1b_107_training.png)

	[Full-size chart](https://huggingface.co/SlayerLab/Slayer1B-1.07/resolve/main/slayer1b_107_training.png)

	## Training

	- Start: Slayer1B 1.06 checkpoint `ckpt_13848.pt`
	(sha256 `1fc7396ca1790805128244b2f827e02b9fe88e352534cd887d0c488f121d1e4d`).
	- Same model, tokenizer (v6, 32 768 tokens, sha256 `0640d3bd…`) and global batch as 1.06: 256 sequences × 4096 tokens
	= 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
	- Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
	about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
	- Length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
	- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
	from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
	(2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was
	resumed within two minutes from the checkpoint at step 19250; the loss at step 19260 was again identical (2.3105).

	## Training data of 1.07

	8 891 005 707 tokens in 9 files, after removing documents that overlap evaluation sets (13-gram scan) and after
	masking personal data (e-mail addresses, phone numbers, national ID numbers) with the same procedure as in 1.06.

	\| source \| language, type \| documents (built) \| tokens (built) \| tokens in training \| license \|
	\|---\|---\|---:\|---:\|---:\|---\|
	\| HPLT (Polish web, parts not used in 1.06) \| PL, general web \| 2 448 525 \| 3 119 999 086 \| 3 110 430 927 \| compilation CC0; texts used under the text-and-data-mining exception (Art. 4, Directive 2019/790) \|
	\| Polish core texts, repeated from 1.06 \| PL, curated texts \| 3 165 531 \| 999 785 312 \| 982 781 646 \| per source, as in slayer-pl-8x3b \|
	\| Polish legal texts, repeated from 1.06 \| PL, legal \| 261 590 \| 1 004 151 322 \| 911 514 295 \| per source, as in slayer-pl-8x3b \|
	\| FineWeb-Edu (from minimal-en-corpus-5b) \| EN, educational web \| 840 133 \| 978 166 858 \| one file, 1 650 618 814 together \| ODC-By \|
	\| FineMath \| EN, mathematics \| 152 623 \| 237 458 786 \| (same file) \| ODC-By \|
	\| open-web-math \| EN, mathematics from the web \| 86 621 \| 225 412 736 \| (same file) \| ODC-By \|
	\| DCLM-baseline (subset) \| EN, general web \| 177 946 \| 244 996 577 \| (same file) \| CC BY 4.0 \|
	\| FineWeb-Edu `sample/10BT` \| EN, educational web \| 2 035 410 \| 2 254 202 363 \| 2 235 660 025 \| ODC-By \|

	Documents in the published training files (after leak removal): Polish web 2 445 601, Polish core 3 151 339,
	Polish legal 258 328, English mix 1 249 258, FineWeb-Edu `sample/10BT` 2 029 345 (9 133 871 in total).

	Share of training tokens: Polish web 35.0 %, Polish core 11.1 %, Polish legal 10.3 %, English 43.7 %.
	The Polish core and legal parts are a random (hash-ordered) subset of what 1.06 already saw; about 36 % of the 1.06
	core tokens and 68 % of the 1.06 legal tokens are seen a second time, the rest of the data once. English documents
	whose exact normalized text was already in the 1.06 English data were excluded.

	## Training-time evaluation

	Same automatic evaluator and the same zero-shot protocol as the 1.06 card (BLiMP, ARC-Easy, WikiText-2, MultiBLiMP-PL,
	ARC-Easy-PL eu20). Checkpoints 13848–19000 and 22327 were evaluated on CPU, checkpoints 20000–22000 on an NVIDIA RTX 4090
	(rented). On both devices the 1.06 checkpoint (step 13848) reproduces the 1.06 card values at card rounding, and on the
	GPU the published Slayer1B 1.02 also reproduces its card values, so all rows are on the same scale. These are not the
	lm-evaluation-harness benchmark used in the Results table of 1.00–1.03.

	\| step \| BLiMP (EN), % \| ARC-Easy (EN), % \| WikiText-2 byte PPL ↓ \| WikiText-2 token PPL ↓ \| MultiBLiMP (PL), % \| ARC-Easy (PL, eu20), % \| device \|
	\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|---\|
	\| 13848 (1.06) \| 76.12 \| 52.99 \| 2.2849 \| 30.03 \| 98.41 \| 45.03 \| CPU \|
	\| 13998 \| 76.29 \| 54.34 \| 2.2618 \| 28.80 \| 98.66 \| 45.37 \| CPU \|
	\| 14000 \| 76.33 \| 54.21 \| 2.2601 \| 28.71 \| 98.59 \| 44.91 \| CPU \|
	\| 15000 \| 77.94 \| 54.59 \| 2.2321 \| 27.28 \| 98.62 \| 45.92 \| CPU \|
	\| 15730 \| 77.32 \| 55.64 \| 2.2237 \| 26.85 \| 98.62 \| 45.75 \| CPU \|
	\| 16000 \| 77.13 \| 56.73 \| 2.2205 \| 26.70 \| 98.81 \| 46.25 \| CPU \|
	\| 17000 \| 77.61 \| 56.48 \| 2.2138 \| 26.37 \| 98.87 \| 45.41 \| CPU \|
	\| 18000 \| 77.88 \| 56.52 \| 2.2042 \| 25.90 \| 98.84 \| 46.21 \| CPU \|
	\| 18590 \| 77.46 \| 56.52 \| 2.2145 \| 26.40 \| 98.62 \| 47.14 \| CPU \|
	\| 19000 \| 77.74 \| 56.65 \| 2.2029 \| 25.84 \| 98.53 \| 46.80 \| CPU \|
	\| 20000 \| 77.68 \| 56.40 \| 2.1940 \| 25.41 \| 98.66 \| 46.55 \| RTX 4090 \|
	\| 21000 \| 77.53 \| 56.69 \| 2.1813 \| 24.81 \| 98.62 \| 47.01 \| RTX 4090 \|
	\| 21450 \| 77.44 \| 58.29 \| 2.1867 \| 25.06 \| 98.75 \| 47.31 \| RTX 4090 \|
	\| 22000 \| 77.16 \| 57.28 \| 2.1794 \| 24.72 \| 98.75 \| 46.97 \| RTX 4090 \|
	\| 22327 (final) \| 77.90 \| 58.67 \| 2.1837 \| 24.92 \| 98.84 \| 46.93 \| CPU \|

	ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
	checkpoints are within noise. From 21000 on, WikiText-2 byte perplexity stays between 2.179 and 2.187 instead of
	falling steadily (lowest 2.1794 at 22000; 2.1837 at the final step): the run used a constant learning rate without decay,
	so the last checkpoint is where training stopped, not a converged minimum. The final checkpoint has the highest English
	ARC-Easy accuracy of the run (58.67, +2.0 points over 19000). Each point is a single measurement. The training loss in the figure is measured
	on the 1.07 data mixture (44 % English), so it is not directly comparable with the training loss of 1.00–1.06.

	The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files
	were scanned for 13-gram overlap with the test items of the evaluation suite before training, and documents with any
	hit were removed. Exact matches of short answers (below 13 words) were counted but not removed, and one evaluation set
	(EWoK, distributed as a password-protected archive) was not part of the scan. The Slayer 1.06 core includes the source
	passage corpora of PolQA and PoQuAD; passages with a 13-gram hit were removed and the rest remain in the replay.

	## Files

	`model.safetensors` (weights, float32, unchanged from the checkpoint), `config.json`, `tokenizer.json`, `load_slayer.py`
	(loader), `train_gpt_ref_r23.py` and `SHA256SUMS`. The `config.json` records the sha256 of the source checkpoint
	`ckpt_22327.pt` (`48cdb5eb16b69e0ee9a7a2ed72895ff47c0ba57aca0f4516b1f7e9cba286685c`, also published in the checkpoints
	repository). The conversion checks that the model built by `load_slayer.py` gives the same logits as the model loaded
	from the checkpoint (maximum absolute difference 0).
	The file layout and the loader are the same as in [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06):

	```python
	import load_slayer
	from tokenizers import Tokenizer

	model = load_slayer.load(".") # float32, eval mode, on CPU
	tok = Tokenizer.from_file("tokenizer.json") # id 0 is end-of-text
	```

	`train_gpt_ref_r23.py` is the model definition (`GPT` class) imported by `load_slayer.py`; `trainer_sha256` in
	`config.json` refers to this file. 1.07 was trained with `train_gpt_ref` r24.2
	(sha256 `46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc`); the architecture is unchanged from 1.00–1.06.