README.md

metadata

language:
  - pl
  - en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
  - polish
  - causal-lm
  - from-scratch
datasets:
  - SlayerLab/slayer-pl-8x3b
  - SlayerLab/slayer-1.07-pretrain

Slayer1B 1.07

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. Slayer1B 1.07 continues pretraining of Slayer1B 1.06 from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B replayed from the 1.06 data), at the same constant learning rate and global batch.

Status: training complete at step 22327 (6 November 2026). This repository contains the converted weights of the final checkpoint (step 22327); see Files below. All 15 evaluated checkpoints of the run, from the 1.06 starting point (13848) to the final checkpoint (22327), are in the training-time evaluation table below. Native training checkpoints (raw PyTorch training state ckpt_N.pt, not a ready-to-use model) are published file by file in SlayerLab/Slayer1B-1.07-checkpoints, with sha256 sums.

• Previous version: SlayerLab/Slayer1B-1.06 (step 13848, 14 520 680 448 tokens, one full pass over SlayerLab/slayer-pl-8x3b)
• Training data of 1.07: SlayerLab/slayer-1.07-pretrain
• Training curve and training-time evaluation: https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925

Full-size chart

Training

• Start: Slayer1B 1.06 checkpoint ckpt_13848.pt (sha256 1fc7396ca1790805128244b2f827e02b9fe88e352534cd887d0c488f121d1e4d).
• Same model, tokenizer (v6, 32 768 tokens, sha256 0640d3bd…) and global batch as 1.06: 256 sequences × 4096 tokens = 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
• Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs), about 45 000 tokens/s. Trainer train_gpt_ref r24.2 (sha256 46f83b51…), unchanged since 1.05.
• Length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
• Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run (2.3177), so the resumed run reproduces the original one. A second external stop on 5 November at step 19330 was resumed within two minutes from the checkpoint at step 19250; the loss at step 19260 was again identical (2.3105).

Training data of 1.07

8 891 005 707 tokens in 9 files, after removing documents that overlap evaluation sets (13-gram scan) and after masking personal data (e-mail addresses, phone numbers, national ID numbers) with the same procedure as in 1.06.

source	language, type	documents (built)	tokens (built)	tokens in training	license
HPLT (Polish web, parts not used in 1.06)	PL, general web	2 448 525	3 119 999 086	3 110 430 927	compilation CC0; texts used under the text-and-data-mining exception (Art. 4, Directive 2019/790)
Polish core texts, repeated from 1.06	PL, curated texts	3 165 531	999 785 312	982 781 646	per source, as in slayer-pl-8x3b
Polish legal texts, repeated from 1.06	PL, legal	261 590	1 004 151 322	911 514 295	per source, as in slayer-pl-8x3b
FineWeb-Edu (from minimal-en-corpus-5b)	EN, educational web	840 133	978 166 858	one file, 1 650 618 814 together	ODC-By
FineMath	EN, mathematics	152 623	237 458 786	(same file)	ODC-By
open-web-math	EN, mathematics from the web	86 621	225 412 736	(same file)	ODC-By
DCLM-baseline (subset)	EN, general web	177 946	244 996 577	(same file)	CC BY 4.0
FineWeb-Edu sample/10BT	EN, educational web	2 035 410	2 254 202 363	2 235 660 025	ODC-By

Documents in the published training files (after leak removal): Polish web 2 445 601, Polish core 3 151 339, Polish legal 258 328, English mix 1 249 258, FineWeb-Edu sample/10BT 2 029 345 (9 133 871 in total).

Share of training tokens: Polish web 35.0 %, Polish core 11.1 %, Polish legal 10.3 %, English 43.7 %. The Polish core and legal parts are a random (hash-ordered) subset of what 1.06 already saw; about 36 % of the 1.06 core tokens and 68 % of the 1.06 legal tokens are seen a second time, the rest of the data once. English documents whose exact normalized text was already in the 1.06 English data were excluded.

Training-time evaluation

Same automatic evaluator and the same zero-shot protocol as the 1.06 card (BLiMP, ARC-Easy, WikiText-2, MultiBLiMP-PL, ARC-Easy-PL eu20). Checkpoints 13848–19000 and 22327 were evaluated on CPU, checkpoints 20000–22000 on an NVIDIA RTX 4090 (rented). On both devices the 1.06 checkpoint (step 13848) reproduces the 1.06 card values at card rounding, and on the GPU the published Slayer1B 1.02 also reproduces its card values, so all rows are on the same scale. These are not the lm-evaluation-harness benchmark used in the Results table of 1.00–1.03.

step	BLiMP (EN), %	ARC-Easy (EN), %	WikiText-2 byte PPL ↓	WikiText-2 token PPL ↓	MultiBLiMP (PL), %	ARC-Easy (PL, eu20), %	device
13848 (1.06)	76.12	52.99	2.2849	30.03	98.41	45.03	CPU
13998	76.29	54.34	2.2618	28.80	98.66	45.37	CPU
14000	76.33	54.21	2.2601	28.71	98.59	44.91	CPU
15000	77.94	54.59	2.2321	27.28	98.62	45.92	CPU
15730	77.32	55.64	2.2237	26.85	98.62	45.75	CPU
16000	77.13	56.73	2.2205	26.70	98.81	46.25	CPU
17000	77.61	56.48	2.2138	26.37	98.87	45.41	CPU
18000	77.88	56.52	2.2042	25.90	98.84	46.21	CPU
18590	77.46	56.52	2.2145	26.40	98.62	47.14	CPU
19000	77.74	56.65	2.2029	25.84	98.53	46.80	CPU
20000	77.68	56.40	2.1940	25.41	98.66	46.55	RTX 4090
21000	77.53	56.69	2.1813	24.81	98.62	47.01	RTX 4090
21450	77.44	58.29	2.1867	25.06	98.75	47.31	RTX 4090
22000	77.16	57.28	2.1794	24.72	98.75	46.97	RTX 4090
22327 (final)	77.90	58.67	2.1837	24.92	98.84	46.93	CPU

ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring checkpoints are within noise. From 21000 on, WikiText-2 byte perplexity stays between 2.179 and 2.187 instead of falling steadily (lowest 2.1794 at 22000; 2.1837 at the final step): the run used a constant learning rate without decay, so the last checkpoint is where training stopped, not a converged minimum. The final checkpoint has the highest English ARC-Easy accuracy of the run (58.67, +2.0 points over 19000). Each point is a single measurement. The training loss in the figure is measured on the 1.07 data mixture (44 % English), so it is not directly comparable with the training loss of 1.00–1.06.

The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files were scanned for 13-gram overlap with the test items of the evaluation suite before training, and documents with any hit were removed. Exact matches of short answers (below 13 words) were counted but not removed, and one evaluation set (EWoK, distributed as a password-protected archive) was not part of the scan. The Slayer 1.06 core includes the source passage corpora of PolQA and PoQuAD; passages with a 13-gram hit were removed and the rest remain in the replay.

Files

model.safetensors (weights, float32, unchanged from the checkpoint), config.json, tokenizer.json, load_slayer.py (loader), train_gpt_ref_r23.py and SHA256SUMS. The config.json records the sha256 of the source checkpoint ckpt_22327.pt (48cdb5eb16b69e0ee9a7a2ed72895ff47c0ba57aca0f4516b1f7e9cba286685c, also published in the checkpoints repository). The conversion checks that the model built by load_slayer.py gives the same logits as the model loaded from the checkpoint (maximum absolute difference 0). The file layout and the loader are the same as in SlayerLab/Slayer1B-1.06:

import load_slayer
from tokenizers import Tokenizer

model = load_slayer.load(".")            # float32, eval mode, on CPU
tok = Tokenizer.from_file("tokenizer.json")  # id 0 is end-of-text

train_gpt_ref_r23.py is the model definition (GPT class) imported by load_slayer.py; trainer_sha256 in config.json refers to this file. 1.07 was trained with train_gpt_ref r24.2 (sha256 46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc); the architecture is unchanged from 1.00–1.06.