---
language: [en]
license: apache-2.0
library_name: pytorch
tags: [slayer, small-language-model, glint-tiny-ml, fabryka-ai]
datasets: [SlayerLab/gollem-v5-arcmix-9b, SlayerLab/minimal-en-corpus-5b, HuggingFaceFW/fineweb-edu, mlfoundations/dclm-baseline-1.0-parquet]
---

# Slayer139 base

A small decoder-only English language model (139.3M parameters) trained from scratch by Fabryka AI. This is the **pretrained base** of [Slayer139 1.01](https://huggingface.co/SlayerLab/Slayer139-1.01) and [Slayer139 1.02](https://huggingface.co/SlayerLab/Slayer139-1.02): the end of the 8× RTX 5090 pretraining run (step 90,310), with **no fine-tuning** of any kind. Slayer139 1.01 is this model fine-tuned on ARC-Easy train; Slayer139 1.02 is this model after continued pretraining and a fine-tune. [Slayer139 1.01b](https://huggingface.co/SlayerLab/Slayer139-1.01b) is a separate run with the same recipe on a smaller machine, not this checkpoint.

**Base model, no ARC fine-tuning.** No ARC dataset (questions or answers, any split) was added to the pretraining data, and this base was never fine-tuned. ARC-Easy train was used only later, in the separate fine-tunes that produced Slayer139 1.01 and 1.02. The pretraining mix does give extra weight to web documents on science topics related to ARC (see Training data and Limitations); web documents overlapping ARC validation/test were removed.

**Author:** Arkadiusz Słota (Fabryka AI / SlayerLab). Training, evaluation and safety gates were run by the Fabryka AI agent team under his direction.

## Result

lm-evaluation-harness 0.4.13, board protocol (C2), our measurement on 1× RTX 5090; the same checkpoint measured on a second machine (AMD Radeon) gave the same scores.

| measure | Slayer139 base | 95% CI |
|---|---|---|
| BLiMP (67 tasks) | 79.41 | [79.14; 79.69] |
| ARC-Easy (test, 2,376) | 57.58 | [55.60; 59.55] |
| WikiText-2 score | 99.86 | |
| **Overall** | **78.95** | [78.28; 79.62] |

Overall is the mean of the three measures. Confidence intervals: item-level bootstrap (ARC-Easy and each of the 67 BLiMP tasks resampled separately, 10,000 draws). They cover test-set sampling only, not seed-to-seed variation; the run is a single seed. These are the same numbers as the "pretrained base" column in the Slayer139 1.01 and 1.01b cards.

Provenance (sha256 prefixes): lm-eval `results.json` RTX 5090 `fabe34ce`, AMD Radeon `719bd11c`; per-item samples RTX 5090 `19f78648`, AMD Radeon `00db3761`.

Numbers are our measurements with the board's protocol, not official scores.

## Model

- Architecture: 17 layers × 768, 12 heads (12 KV heads), SwiGLU, decoder-only; vocabulary 24,576 (BPE), context 1,024.
- Parameters: 139,279,760 unique parameters (input and output embeddings tied: one shared tensor, counted once; counting the shared tensor twice gives 158,154,128).
- Checkpoint: the final step 90,310 of the run (sha256 `ba223162…`), the end of the run rather than a selected step (rule written before any test number). `model.safetensors` holds the same weights as that checkpoint (205 of 205 tensors identical; the output head is the tied input embedding).
- Code: `modeling_slayer139.py` (inference only, plain PyTorch), the same file as in Slayer139 1.01b.

## Training

- Data: ~23.67B tokens, no repeated epoch (sources and licences below). First the ARC-MIX pool, then a top-up from the same source families in the base corpus's proportions, mixed at document level. Both stages were scanned against WikiText-2 (test, validation) and ARC-Easy/Challenge (validation, test) for shared normalized 13-grams, with short ARC questions matched exactly, and against BLiMP sentences of ≥ 6 words; matching documents were removed. ARC train was not filtered.
- Optimiser and schedule: Muon (hidden matrices, LR 0.0566) + AdamW (other parameters, LR 1.6971e-3), both ×√8 from the single-node recipe; WSD schedule: 250 warm-up steps, constant to step 72,248, then 1−√ decay to 10% of peak at step 90,310; batch 256 × 1,024 tokens; seed 1337.
- Same recipe as our single-node final (batch 32, 4× RTX 5090, ~191k tokens/s), scaled to batch 256 with LR ×√8 and the identical data order — 4.4× faster. At equal tokens the 8-GPU run had lower held-out DCLM bits-per-byte at every compared checkpoint (−0.014 at 1.1B tokens, −0.032 at 19.4B tokens).
- Hardware and time: 8× NVIDIA RTX 5090 (data parallel, bf16 all-reduce), median ~832k tokens/s, about 8 hours of training.
- Training curves: https://track.fabryka.ai/run/5a4d174e-4a9d-4cc1-9602-838a3ffbb917

## Training data

Aggregate corpus; each source keeps its upstream terms.

- **ARC-MIX pool** (`SlayerLab/gollem-v5-arcmix-9b`, 9.39B tokens before retokenization): the 15-source base `SlayerLab/minimal-en-corpus-5b` (FineWeb-Edu, DCLM-baseline, open-web-math, FineMath, StarCoderData, StackExchange, LoC public-domain books, Wikipedia, Project Gutenberg, scientific papers, UltraChat, WildChat, CC-News, tiny-textbooks, OpenSubtitles) + a FineWeb-Edu expansion + OpenStax textbooks (×4) + extra copies of FineWeb-Edu documents scored as related to ARC science topics (+2 copies for the highest score, +1 for the next). Removed before training: 4,362 documents by the WikiText-2 / ARC validation+test scan and 1,175 documents marked CC BY-NC-SA.
- **Top-up** (11.4M documents, 57.45 GB of text, build manifest `b73f45f5…`): FineWeb-Edu expansion (37.1%, ODC-By 1.0, plus the same dose of ARC-related copies as the pool), FineWeb-Edu (14.9%, ODC-By 1.0), DCLM-baseline (10.9%, CC BY 4.0), StackExchange via RedPajama (6.1%, CC BY-SA content), LoC public-domain books (5.4%, CC0), StarCoderData (5.1%, The Stack terms), Wikipedia 20231101.en (4.6%, CC BY-SA 3.0 / GFDL), Project Gutenberg (2.9%, US public domain), PMC Open Access commercial-use subset (2.8%, CC0 / CC BY / CC BY-SA per article; ND and NC excluded), open-web-math (2.7%, ODC-By), FineMath (2.6%, ODC-By), WildChat-1M (2.0%, ODC-By; English, non-toxic), CC-News (1.4%, licence unknown), UltraChat 200k (0.7%, MIT), tiny-textbooks (0.7%, Apache 2.0). The top-up scan removed 25,808 documents (0.23%), each together with its ARC-related copies.
- Licences upstream include ODC-By, CC BY 4.0, CC BY-SA (Wikipedia, StackExchange, some PMC articles), CC0 / public domain, and sources with unknown or restrictive terms. Commercial use: review upstream terms, in particular CC-News, scientific papers, StarCoderData and model-generated chat data (UltraChat, WildChat, tiny-textbooks).
- OpenStax textbooks (55 titles) are used under CC BY 4.0; attribution: OpenStax, Rice University (see [OpenStax attribution](#openstax-attribution) below).

## Limitations

- English only; small model; one seed.
- The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC validation and test sets; no ARC questions were added); ARC-Easy is therefore an in-domain benchmark for this model.
- Not instruction-tuned and not fine-tuned on any benchmark.

## OpenStax attribution

The training data of this model includes text extracted from the following OpenStax textbooks, each licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), © Rice University. Download for free at https://openstax.org. The texts were obtained from the Hugging Face dataset crumb/openstax-text (revision 8f502ca45f9f05cb5673eae445b7a97a4e8c4349). Modified: text extracted, chunked and filtered (chunks matching benchmark test/validation sets were removed).

| Title (from source file name) | © year |
|---|---|
| APBiology | 2018 |
| APCollege Physics | 2017 |
| APMacroeconomics 2e | 2017 |
| APMicroeconomics 2e | 2017 |
| Algebra and Trigonometry 2e | 2021 |
| American Government 3e | 2021 |
| Anatomy and Physiology 2e | 2022 |
| Anatomyand Physiology | 2017 |
| Astronomy 2e | 2022 |
| Astronomy | 2018 |
| Biology 2e | 2020 |
| Business Ethics | 2018 |
| Chemistry 2e | 2019 |
| Chemistry Atoms First 2e | 2019 |
| College Algebra 2e | 2021 |
| College Algebra Corequisite Support 2e | 2021 |
| College Physics | 2020 |
| College Physics 2e | 2022 |
| College Physics for AP Courses 2e | 2022 |
| College Success | 2020 |
| College Success | 2023 |
| Concepts Biology | 2017 |
| Contemporary Mathematics | 2023 |
| Economics 2e | 2018 |
| Economics 3e | 2022 |
| Elementary Algebra 2e | 2020 |
| Entrepreneurship | 2020 |
| Intermediate Algebra 2e | 2020 |
| Introduction to Intellectual Property | n/a |
| Introduction to Philosophy | 2022 |
| Introduction to Political Science | 2022 |
| Introductionto Anthropology | 2022 |
| Introductionto Sociology 3e | 2021 |
| Introductory Business Statistics | 2018 |
| Introductory Statistics | 2018 |
| Macroeconomics 2e | 2018 |
| Macroeconomics 3e | 2022 |
| Microbiology | 2021 |
| Microeconomics 2e | 2018 |
| Microeconomics 3e | 2022 |
| Physics | n/a |
| Prealgebra 2e | 2020 |
| Precalculus 2e | 2021 |
| Preparing for College Success | 2023 |
| Principles Marketing | 2023 |
| Principlesof Finance | 2022 |
| Psychology 2e | 2020 |
| Statistics | n/a |
| USHistory | 2021 |
| University Physics Vol 1 | 2021 |
| University Physics Volume 2 | 2021 |
| University Physics Volume 3 | 2021 |
| World History Volume 1 | 2023 |
| World History Volume 2 | 2022 |
| Writing Guide | 2021 |

Titles licensed CC BY-NC-SA 4.0, non-English titles, and three CC BY 4.0 titles whose text contains elements marked ‘CC BY-NC-SA’ (Introduction to Business, Organizational Behavior, Principles of Management) were not used.

## Reproduce

Evaluation: lm-evaluation-harness 0.4.13, tasks `blimp`, `arc_easy`, `wikitext` (board protocol C2); per-item samples are kept for paired bootstrap. Method and full run history: paper „Same recipe, 4.4× faster on 8× RTX 5090” (Fabryka AI, in preparation).