Versions

Results

One instrument for the versions that have values (1.00–1.02 and 1.04–1.06): leaderboard protocol of track.fabryka.ai/models, zero-shot, on the released checkpoints. Column 1.03 is a dash on purpose: its run (1.03) was measured with the older lm-eval task set (hellaswag, piqa, lambada, arc_challenge, boolq, mmlu — see the additional history section), so there are no values in this instrument; the dash is not a missing measurement. 1.00 and 1.01 ended with a learning-rate decay; 1.02 and 1.04–1.06 did not, so comparisons across that boundary are not clean. Neighbouring checkpoints differ by up to about a point on single tests, so small differences between 1.05 and 1.06 are within run noise.

Additional benchmark history

16 checkpoints from 1.00–1.03, through final 7150 have separate lm-eval 0.4.13, zero-shot, fp32 results: 13 historical points plus 6900, 7050 and 7150. These results use a different benchmark procedure from the table above.

Accuracy and normalized accuracy are distinct series, in fractions (0–1); LAMBADA perplexity is separate. The scan found n-gram overlap in shared/en for BoolQ, MMLU and ARC; its significance is still being assessed. Overlap is not by itself a measured contamination rate. Open Charts and filter by eval/. This is an imported result series, not a native automatic-benchmark job.

The three new points use the same runner and task procedure, but a different runtime: Torch 2.8.0+cu128, tokenizers 0.23.2 and NumPy 2.2.6, versus 2.14.1+cu126, 0.23.3 and 2.5.3 for the historical points. The 6750 → 6900 boundary is not an identical-environment checkpoint comparison.

Score curves: 1.00–1.06

Measured values only, no fits, no trend lines. 1.00–1.02: leaderboard protocol, one checkpoint every 150 steps; shaded bands mark the learning-rate decay of 1.00 and 1.01. 1.03: training loss only — it was evaluated with an older lm-eval task set, so it has no points in the score panels. 1.04–1.06: one continuation from step 7150 at a constant learning rate; diamonds mark the releases 1.04 (10010), 1.05 (12870) and 1.06 (13848, end of the corpus). The in-run evaluator measured every 50 steps up to 9100, then control points 9400/9700/10000, then every 200 steps; breaks in the line are gaps in that schedule, not missing training. eff = (BLiMP + ARC + WikiScore) / 3, the board formula, which reproduces every published 1.00–1.02 value within 0.01. Data: scores-history-106.csv.

Release points: 1.04 → 1.06

Measured on the released checkpoints by the run's automatic evaluator (same instrument as the results table), read from the public continuation run. The whole continuation used a constant learning rate; a final decay has not been applied, so these are not end-of-training numbers for the model line.

How we do it

1. From scratch. A transformer (20 layers, width 2048), our own Polish tokenizer, Polish data with an English minority (slayer-pl-8x3b).
2. Checkpointed continuation. Earlier decayed versions branched from checkpoints before their final decay. Version names mark milestones in one global data trajectory, not mandatory process restarts. Retention maintenance resumed from verified checkpoint 7750 without resetting the data order. The final learning-rate decay is planned once, at the end.
3. Source and model units. The source corpus is counted as ~24 B cl100k tokens. Our v6/32k tokenizer produces separate uint16 train/validation files. Actual model-token counts and stream boundaries come from pinned manifests. Next packs are prepared while the current stage trains.
4. We verify everything. Checkpoint SHA-256 records are preserved outside the training machine. Before local pruning, a full copy on a separate archive is verified and its receipt is persisted atomically. From version 1.01 on, a loss forecast is written down before each run and evaluated afterwards.

Technical report: Slayer1B: Continuing a Polish-Dominant Base Model.

What's next

Corpus continuation 1.04–1.06 — complete. The global trajectory ran from native step 7150 to 13848, the end of the prepared corpus: 14.5 B training tokens, one full pass, about 13.5 tokens per parameter. Releases 1.04 (10010), 1.05 (12870) and 1.06 (13848) are public, each with its checkpoint archive. Training run.

The linked runs belong to the author's Track account. The completed 1.03 run includes its resumed steps; corpus continuation and benchmark history are separate records of the same model lineage.

Beyond 13848 — roadmap candidates, not scheduled runs: continuing to about 20 tokens per parameter with fresh data, a final learning-rate decay, and extending the context window in stages (first 32k tokens, up to 256k as the target), each stage checked with long-context tests. Each follow-up needs its own plan, cost estimate and go-ahead.

These are base models: they predict the continuation of a text, they were not tuned for chat or for following instructions, and they can be wrong.