Card: BLiMP count note — cause of the 47 913 count not determined

43e7ca2 verified 5 days ago

7.53 kB

---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
---
# Slayer1B 1.00

Decoder-only language model, 1 073 912 851 parameters, trained from scratch on Polish-dominant text
(pack-00 of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b)).

Training curve: https://track.fabryka.ai/run/48744ed2-3155-4294-b899-a6fda1f851da.

## Results

Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
checkpoint (step 1430, trainer checkpoint sha256 `7d092d4d…`). Computed twice with two independent loaders (these
weights via `load_slayer.py`, and the trainer checkpoint directly); both gave the same scores (to the precision shown).

| task | metric | n | score |
|---|---|---:|---:|
| BLiMP (67 phenomena) | accuracy | 67 000 | 71.51 % (47 912 correct; an evaluation on another GPU counted 47 913, cause not determined: no per-item record from that run) |
| ARC-Easy (test) | accuracy | 2 376 | 42.89 % (1 019 correct) |
| WikiText-2 (raw, test) | token perplexity (this tokenizer) | — | 49.9734 |
| WikiText-2 (raw, test) | byte perplexity | 1 292 008 bytes | 2.5857 |
| eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity) | leaderboard formula | — | 69.50 |

Polish columns of the same leaderboard (also computed twice with two independent implementations; identical item by
item):

| task | metric | n | score |
|---|---|---:|---:|
| MultiBLiMP, Polish (`jumelet/multiblimp` pol) | accuracy, summed log-probability, leading-space prefix | 3 272 | 97.89 % (3 203 correct) |
| ARC-Easy-PL (`Eurolingua/arcx` EU20, test) | accuracy, LL(q + choice) − LL(q) | 2 376 | 36.91 % (877 correct) |

## Usage

The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
is `GPT` in `train_gpt_ref_r23.py` (included).

```python
import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]
```

## Model

| | |
|---|---|
| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
| Context | 4096 tokens |
| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |
## Training data
Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`, pack-00
(its own disjoint web slice + the shared core, legal and English components), sampled per document with the pack's
mixing weights (defined in cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10.
| component | share of tokens (this tokenizer) | license |
|---|---|---|
| web (HPLT v3 PL, disjoint slice) | 51.8 % | CC0-1.0 |
| core (register-diverse; includes CC-BY-SA partition) | 22.2 % | mixed, incl. CC-BY-SA |
| legal / public administration (includes CC-BY-SA partition) | 8.5 % | CC-BY-4.0 / PD / CC-BY-SA |
| English | 17.6 % | ODC-BY |
English is 10 % of the mixture in cl100k tokens and 17.6 % in this tokenizer's tokens: the tokenizer is trained on
Polish and splits English into more pieces.
Available training stream: 1 889 519 627 tokens (`train.bin`, sha256 `760ef0cc…`), of which 1 499 463 680 were consumed
(1430 steps × 1 048 576; each window at most once). Held-out
validation from the same pack, disjoint by document id: 1 562 647 PL tokens, 1 643 997 EN tokens.
Personal data: before tokenization, e-mail addresses, phone numbers (Polish and international), PESEL, NIP, REGON and
Polish ID-card numbers were replaced with tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`
(rule-based; counts in the data manifest: e-mail 19 080, phone 21 606, NIP 1 602, REGON 1 421, ID card 259,
PESEL 131). The source data also contains upstream placeholders such as `[PII]`. Rule-based masking does not guarantee
removal of every personal datum.
## Training
| | |
|---|---|
| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
| Batch | 256 × 4096 = 1 048 576 tokens per step |
| Steps | 1430 (02:35:48 → 06:47:13 UTC, about 4 h 11 min; ~100 000 tokens/s) |
| Tokens consumed | 1430 × 1 048 576 = 1 499 463 680 (about 1.4 tokens per parameter) |
| Final training loss | 2.6969 (step 1430, nats per token) |
| Checkpoints | every 150 steps (10 in total), each copied off the training node with sha256 |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
| Schedule | warmup 150 steps, constant, then 1 − sqrt decay over the last 20 % of steps (from step 1144) to lr / 10 |
| Data order | each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023 |
| Precision | bf16 autocast, torch.compile, activation checkpointing |
## Training-loss scenarios (not a forecast)
![Training loss and conditional scenarios](analysis/scaling-training-loss.png)
`analysis/scaling-training-loss.png` shows the measured training loss and conditional scenarios for a total budget of
3, 6 and 12 billion tokens. Training continues from checkpoint 1050 (the last one before the learning-rate decay that
starts at step 1144) at constant learning rate, and the learning rate is decayed only over the last 20 % of the new
budget. Up to step 1050 this is the same schedule as a new warmup-stable-decay run with that budget. The scenarios come
from the learning-rate-annealing scaling law (Tissue et al., arXiv:2408.11029) fitted to steps 200–1430, for
λ = 0.99 / 0.995 / 0.999. Checkpoint 1050 and the other intermediate checkpoints (with optimizer state) are in
[SlayerLab/Slayer1B-1.00-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.00-checkpoints).
- **The backtest of this method failed** on this run: fitted on earlier steps, it missed the measured loss by
  0.077–0.088 nats per token (threshold 0.03). The bars of ±0.09 show that historical error; they are not a confidence
  or prediction interval.
- The spread between λ values (up to 0.6 nats at 12 B tokens) is larger than the gain being estimated. One 1.5 B-token
  run does not determine it.
- Assumptions: training loss (not validation loss), new data from the same distribution beyond the 1.89 B tokens that
  exist, one seed. Benchmarks are not projected.
- Numbers: `analysis/scaling-training-loss.json`; input loss log: `analysis/training_metrics.jsonl`. Two independent
  implementations agree to 3e-7 nats.

## Limitations

- 1 073 912 851 parameters trained on 1 499 463 680 tokens (about 1.4 tokens per parameter). Compare with the token
  counts of other models when reading the scores.
- Polish-dominant data; English is a minority of the mixture.
- Base model: no instruction tuning, no safety tuning.
- Masking of personal data is rule-based (see above).

## License

Weights: Apache-2.0. Training data components carry their own licenses (table above), including CC-BY-SA partitions.

	---
	language:
	- pl
	- en
	license: apache-2.0
	library_name: pytorch
	pipeline_tag: text-generation
	tags:
	- polish
	- causal-lm
	- from-scratch
	datasets:
	- SlayerLab/slayer-pl-8x3b
	---

	# Slayer1B 1.00

	Decoder-only language model, 1 073 912 851 parameters, trained from scratch on Polish-dominant text
	(pack-00 of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b)).

	Training curve: https://track.fabryka.ai/run/48744ed2-3155-4294-b899-a6fda1f851da.

	## Results

	Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
	BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
	checkpoint (step 1430, trainer checkpoint sha256 `7d092d4d…`). Computed twice with two independent loaders (these
	weights via `load_slayer.py`, and the trainer checkpoint directly); both gave the same scores (to the precision shown).

	\| task \| metric \| n \| score \|
	\|---\|---\|---:\|---:\|
	\| BLiMP (67 phenomena) \| accuracy \| 67 000 \| 71.51 % (47 912 correct; an evaluation on another GPU counted 47 913, cause not determined: no per-item record from that run) \|
	\| ARC-Easy (test) \| accuracy \| 2 376 \| 42.89 % (1 019 correct) \|
	\| WikiText-2 (raw, test) \| token perplexity (this tokenizer) \| — \| 49.9734 \|
	\| WikiText-2 (raw, test) \| byte perplexity \| 1 292 008 bytes \| 2.5857 \|
	\| eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity) \| leaderboard formula \| — \| 69.50 \|

	Polish columns of the same leaderboard (also computed twice with two independent implementations; identical item by
	item):

	\| task \| metric \| n \| score \|
	\|---\|---\|---:\|---:\|
	\| MultiBLiMP, Polish (`jumelet/multiblimp` pol) \| accuracy, summed log-probability, leading-space prefix \| 3 272 \| 97.89 % (3 203 correct) \|
	\| ARC-Easy-PL (`Eurolingua/arcx` EU20, test) \| accuracy, LL(q + choice) − LL(q) \| 2 376 \| 36.91 % (877 correct) \|

	## Usage

	The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
	is `GPT` in `train_gpt_ref_r23.py` (included).

	```python
	import torch, load_slayer
	from tokenizers import Tokenizer
	model = load_slayer.load(".") # directory with the files of this repository
	tok = Tokenizer.from_file("tokenizer.json")
	ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
	logits, _ = model(ids) # [batch, seq, 32768]
	```

	## Model

	\| \| \|
	\|---\|---\|
	\| Parameters \| 1 073 912 851 (input and output embeddings tied, counted once) \|
	\| Architecture \| decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) \|
	\| Blocks \| pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases \|
	\| Context \| 4096 tokens \|
	\| Tokenizer \| byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 \|

	## Training data

	Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`, pack-00
	(its own disjoint web slice + the shared core, legal and English components), sampled per document with the pack's
	mixing weights (defined in cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10.

	\| component \| share of tokens (this tokenizer) \| license \|
	\|---\|---\|---\|
	\| web (HPLT v3 PL, disjoint slice) \| 51.8 % \| CC0-1.0 \|
	\| core (register-diverse; includes CC-BY-SA partition) \| 22.2 % \| mixed, incl. CC-BY-SA \|
	\| legal / public administration (includes CC-BY-SA partition) \| 8.5 % \| CC-BY-4.0 / PD / CC-BY-SA \|
	\| English \| 17.6 % \| ODC-BY \|

	English is 10 % of the mixture in cl100k tokens and 17.6 % in this tokenizer's tokens: the tokenizer is trained on
	Polish and splits English into more pieces.

	Available training stream: 1 889 519 627 tokens (`train.bin`, sha256 `760ef0cc…`), of which 1 499 463 680 were consumed
	(1430 steps × 1 048 576; each window at most once). Held-out
	validation from the same pack, disjoint by document id: 1 562 647 PL tokens, 1 643 997 EN tokens.

	Personal data: before tokenization, e-mail addresses, phone numbers (Polish and international), PESEL, NIP, REGON and
	Polish ID-card numbers were replaced with tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`
	(rule-based; counts in the data manifest: e-mail 19 080, phone 21 606, NIP 1 602, REGON 1 421, ID card 259,
	PESEL 131). The source data also contains upstream placeholders such as `[PII]`. Rule-based masking does not guarantee
	removal of every personal datum.

	## Training

	\| \| \|
	\|---\|---\|
	\| Hardware \| 8 × NVIDIA RTX 5090 (32 GB), DDP \|
	\| Batch \| 256 × 4096 = 1 048 576 tokens per step \|
	\| Steps \| 1430 (02:35:48 → 06:47:13 UTC, about 4 h 11 min; ~100 000 tokens/s) \|
	\| Tokens consumed \| 1430 × 1 048 576 = 1 499 463 680 (about 1.4 tokens per parameter) \|
	\| Final training loss \| 2.6969 (step 1430, nats per token) \|
	\| Checkpoints \| every 150 steps (10 in total), each copied off the training node with sha256 \|
	\| Optimizer \| Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest \|
	\| Schedule \| warmup 150 steps, constant, then 1 − sqrt decay over the last 20 % of steps (from step 1144) to lr / 10 \|
	\| Data order \| each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023 \|
	\| Precision \| bf16 autocast, torch.compile, activation checkpointing \|

	## Training-loss scenarios (not a forecast)

	![Training loss and conditional scenarios](analysis/scaling-training-loss.png)

	`analysis/scaling-training-loss.png` shows the measured training loss and conditional scenarios for a total budget of
	3, 6 and 12 billion tokens. Training continues from checkpoint 1050 (the last one before the learning-rate decay that
	starts at step 1144) at constant learning rate, and the learning rate is decayed only over the last 20 % of the new
	budget. Up to step 1050 this is the same schedule as a new warmup-stable-decay run with that budget. The scenarios come
	from the learning-rate-annealing scaling law (Tissue et al., arXiv:2408.11029) fitted to steps 200–1430, for
	λ = 0.99 / 0.995 / 0.999. Checkpoint 1050 and the other intermediate checkpoints (with optimizer state) are in
	[SlayerLab/Slayer1B-1.00-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.00-checkpoints).

	- The backtest of this method failed on this run: fitted on earlier steps, it missed the measured loss by
	0.077–0.088 nats per token (threshold 0.03). The bars of ±0.09 show that historical error; they are not a confidence
	or prediction interval.
	- The spread between λ values (up to 0.6 nats at 12 B tokens) is larger than the gain being estimated. One 1.5 B-token
	run does not determine it.
	- Assumptions: training loss (not validation loss), new data from the same distribution beyond the 1.89 B tokens that
	exist, one seed. Benchmarks are not projected.
	- Numbers: `analysis/scaling-training-loss.json`; input loss log: `analysis/training_metrics.jsonl`. Two independent
	implementations agree to 3e-7 nats.

	## Limitations

	- 1 073 912 851 parameters trained on 1 499 463 680 tokens (about 1.4 tokens per parameter). Compare with the token
	counts of other models when reading the scores.
	- Polish-dominant data; English is a minority of the mixture.
	- Base model: no instruction tuning, no safety tuning.
	- Masking of personal data is rule-based (see above).

	## License

	Weights: Apache-2.0. Training data components carry their own licenses (table above), including CC-BY-SA partitions.