File size: 7,527 Bytes

---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
---

# Slayer1B 1.00

Decoder-only language model, 1 073 912 851 parameters, trained from scratch on Polish-dominant text
(pack-00 of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b)).

Training curve: https://track.fabryka.ai/run/48744ed2-3155-4294-b899-a6fda1f851da.

## Results

Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
checkpoint (step 1430, trainer checkpoint sha256 `7d092d4d…`). Computed twice with two independent loaders (these
weights via `load_slayer.py`, and the trainer checkpoint directly); both gave the same scores (to the precision shown).

| task | metric | n | score |
|---|---|---:|---:|
| BLiMP (67 phenomena) | accuracy | 67 000 | 71.51 % (47 912 correct; an evaluation on another GPU counted 47 913, cause not determined: no per-item record from that run) |
| ARC-Easy (test) | accuracy | 2 376 | 42.89 % (1 019 correct) |
| WikiText-2 (raw, test) | token perplexity (this tokenizer) | — | 49.9734 |
| WikiText-2 (raw, test) | byte perplexity | 1 292 008 bytes | 2.5857 |
| eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity) | leaderboard formula | — | 69.50 |

Polish columns of the same leaderboard (also computed twice with two independent implementations; identical item by
item):

| task | metric | n | score |
|---|---|---:|---:|
| MultiBLiMP, Polish (`jumelet/multiblimp` pol) | accuracy, summed log-probability, leading-space prefix | 3 272 | 97.89 % (3 203 correct) |
| ARC-Easy-PL (`Eurolingua/arcx` EU20, test) | accuracy, LL(q + choice) − LL(q) | 2 376 | 36.91 % (877 correct) |

## Usage

The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
is `GPT` in `train_gpt_ref_r23.py` (included).

```python
import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]
```

## Model

| | |
|---|---|
| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
| Context | 4096 tokens |
| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |

## Training data

Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`, pack-00
(its own disjoint web slice + the shared core, legal and English components), sampled per document with the pack's
mixing weights (defined in cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10.

| component | share of tokens (this tokenizer) | license |
|---|---|---|
| web (HPLT v3 PL, disjoint slice) | 51.8 % | CC0-1.0 |
| core (register-diverse; includes CC-BY-SA partition) | 22.2 % | mixed, incl. CC-BY-SA |
| legal / public administration (includes CC-BY-SA partition) | 8.5 % | CC-BY-4.0 / PD / CC-BY-SA |
| English | 17.6 % | ODC-BY |

English is 10 % of the mixture in cl100k tokens and 17.6 % in this tokenizer's tokens: the tokenizer is trained on
Polish and splits English into more pieces.

Available training stream: 1 889 519 627 tokens (`train.bin`, sha256 `760ef0cc…`), of which 1 499 463 680 were consumed
(1430 steps × 1 048 576; each window at most once). Held-out
validation from the same pack, disjoint by document id: 1 562 647 PL tokens, 1 643 997 EN tokens.

Personal data: before tokenization, e-mail addresses, phone numbers (Polish and international), PESEL, NIP, REGON and
Polish ID-card numbers were replaced with tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`
(rule-based; counts in the data manifest: e-mail 19 080, phone 21 606, NIP 1 602, REGON 1 421, ID card 259,
PESEL 131). The source data also contains upstream placeholders such as `[PII]`. Rule-based masking does not guarantee
removal of every personal datum.

## Training

| | |
|---|---|
| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
| Batch | 256 × 4096 = 1 048 576 tokens per step |
| Steps | 1430 (02:35:48 → 06:47:13 UTC, about 4 h 11 min; ~100 000 tokens/s) |
| Tokens consumed | 1430 × 1 048 576 = 1 499 463 680 (about 1.4 tokens per parameter) |
| Final training loss | 2.6969 (step 1430, nats per token) |
| Checkpoints | every 150 steps (10 in total), each copied off the training node with sha256 |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
| Schedule | warmup 150 steps, constant, then 1 − sqrt decay over the last 20 % of steps (from step 1144) to lr / 10 |
| Data order | each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023 |
| Precision | bf16 autocast, torch.compile, activation checkpointing |

## Training-loss scenarios (not a forecast)

![Training loss and conditional scenarios](analysis/scaling-training-loss.png)

`analysis/scaling-training-loss.png` shows the measured training loss and conditional scenarios for a total budget of
3, 6 and 12 billion tokens. Training continues from checkpoint 1050 (the last one before the learning-rate decay that
starts at step 1144) at constant learning rate, and the learning rate is decayed only over the last 20 % of the new
budget. Up to step 1050 this is the same schedule as a new warmup-stable-decay run with that budget. The scenarios come
from the learning-rate-annealing scaling law (Tissue et al., arXiv:2408.11029) fitted to steps 200–1430, for
λ = 0.99 / 0.995 / 0.999. Checkpoint 1050 and the other intermediate checkpoints (with optimizer state) are in
[SlayerLab/Slayer1B-1.00-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.00-checkpoints).

- **The backtest of this method failed** on this run: fitted on earlier steps, it missed the measured loss by
  0.077–0.088 nats per token (threshold 0.03). The bars of ±0.09 show that historical error; they are not a confidence
  or prediction interval.
- The spread between λ values (up to 0.6 nats at 12 B tokens) is larger than the gain being estimated. One 1.5 B-token
  run does not determine it.
- Assumptions: training loss (not validation loss), new data from the same distribution beyond the 1.89 B tokens that
  exist, one seed. Benchmarks are not projected.
- Numbers: `analysis/scaling-training-loss.json`; input loss log: `analysis/training_metrics.jsonl`. Two independent
  implementations agree to 3e-7 nats.

## Limitations

- 1 073 912 851 parameters trained on 1 499 463 680 tokens (about 1.4 tokens per parameter). Compare with the token
  counts of other models when reading the scores.
- Polish-dominant data; English is a minority of the mixture.
- Base model: no instruction tuning, no safety tuning.
- Masking of personal data is rule-based (see above).

## License

Weights: Apache-2.0. Training data components carry their own licenses (table above), including CC-BY-SA partitions.