| --- |
| language: [en] |
| license: apache-2.0 |
| library_name: pytorch |
| tags: [slayer, small-language-model, glint-tiny-ml, fabryka-ai] |
| datasets: [SlayerLab/gollem-v5-arcmix-9b, SlayerLab/minimal-en-corpus-5b, HuggingFaceFW/fineweb-edu, mlfoundations/dclm-baseline-1.0-parquet, allenai/ai2_arc, Salesforce/wikitext] |
| --- |
| |
| # Slayer139 1.02 |
|
|
| A small decoder-only English language model (139.3M parameters) by Fabryka AI, built for the Glint Tiny-ML leaderboard (≤150M parameters). It is Slayer139 1.01's pretrained base, given a short continued pretraining and a new ARC fine-tune. |
|
|
| **Author:** Arkadiusz Słota (Fabryka AI / SlayerLab) |
|
|
| **Identity:** Slayer139 1.02 is the training checkpoint with sha256 `50f07cab0e1eedb4b3c1e65119a53fb6105f9c6e1fe7984b2b604c77c0c5ec7c` (internal run name POL4-CEP s1491). No other checkpoint is published under this name. |
|
|
| ## Result |
|
|
| lm-evaluation-harness 0.4.13, 0-shot, board protocol (C2) — the same protocol and setup as the Slayer139 1.01 row. |
|
|
| | measure | Slayer139 1.02 | Slayer139 1.01 | Δ | |
| |---|---|---|---| |
| | BLiMP (67 tasks) | 79.22 | 79.13 | +0.09 | |
| | ARC-Easy (test, 2,376, acc) | 65.91 | 63.09 | +2.82 | |
| | WikiText-2 byte perplexity | 1.8620 | 1.8782 | −0.0162 | |
| | WikiText-2 score | 99.98 | 99.83 | +0.15 | |
| | **Overall** | **81.70** | **80.68** | **+1.02** | |
|
|
| Overall is the mean of BLiMP, ARC-Easy and the WikiText-2 score. Board size multiplier 1.003100 (efficiency score 81.96). |
|
|
| Provenance (sha256 prefix of the lm-eval `results.json`): `8b21f803`. |
|
|
| Numbers are our measurements with the board's protocol, not official scores until the leaderboard lists them. |
|
|
| ### How this checkpoint was selected (read this before comparing) |
|
|
| - 15 fine-tune variants were compared on ARC-Easy **validation** (one half, 285 questions) plus 500 ARC-Easy **train** questions held out from all fine-tuning. The other validation half was read twice in the series (once per test candidate), only to forecast the test score. |
| - The test split was read only after a selection rule written in advance was met. For this model the test set was read **once**. Across the whole series the test set was read **twice**: an earlier candidate scored Overall 81.14 and was **not** published. |
| - Within the chosen variant the last epoch was taken (fixed in advance, not selected). |
| - Our Slayer149-ARC model scores Overall 81.50 with the same protocol, so the lead of this model over it is **only +0.20 Overall, within about one standard error**; treat the two as tied. This checkpoint has not yet been re-benchmarked by the leaderboard maintainers. |
|
|
| ## Model |
|
|
| - Architecture: 17 layers × 768, 12 heads (12 KV heads), SwiGLU, decoder-only; vocabulary 24,576 (BPE), context 1,024. Identical to Slayer139 1.01. |
| - Parameters: 139,279,760 unique parameters (input and output embeddings tied: one shared tensor, counted once; counting the shared tensor twice gives 158,154,128). |
| - Checkpoint lineage: pretrained base (step 90,310, sha256 `ba223162…`, the base of Slayer139 1.01) → continued pretraining (1,145 steps, sha256 `05815bf3…`) → ARC fine-tune → **this model** (sha256 `50f07cab…`). |
|
|
| ## Training |
|
|
| - **Pretraining:** identical to Slayer139 1.01 (~23.67B tokens, no repeated epoch; see the Slayer139 1.01 card for the optimiser, schedule and hardware). |
| - **Continued pretraining:** 1,145 steps, ~303M tokens: 85% the same data mix as pretraining, 15% WikiText-103 train. |
| - **ARC fine-tune** (after continued pretraining): listwise cross-entropy over the summed log-probabilities of the answer tokens, prompt `Question: {q}\nAnswer:` / ` {a}` (the lm-evaluation-harness format). Data: ARC-Easy train and ARC-Challenge train, after removing questions that overlap the ARC-Easy test set; 500 ARC-Easy train questions held out for selection. ARC-Challenge questions were used in epochs 1–2 only. AdamW, constant LR 2e-5, betas (0.9, 0.95), weight decay 0.01, 8 questions per step, 5 epochs (last epoch taken), seed 1491. Replay: the loss is 0.15 × answer-choice loss + 0.85 × next-token loss on pretraining text windows, to limit forgetting. |
|
|
| ## Training data |
|
|
| Aggregate corpus; each source keeps its upstream terms. |
|
|
| - **ARC-MIX pool** (`SlayerLab/gollem-v5-arcmix-9b`, 9.39B tokens before retokenization): the 15-source base `SlayerLab/minimal-en-corpus-5b` (FineWeb-Edu, DCLM-baseline, open-web-math, FineMath, StarCoderData, StackExchange, LoC public-domain books, Wikipedia, Project Gutenberg, scientific papers, UltraChat, WildChat, CC-News, tiny-textbooks, OpenSubtitles) + a FineWeb-Edu expansion + OpenStax textbooks (×4) + extra copies of FineWeb-Edu documents scored as related to ARC science topics (+2 copies for the highest score, +1 for the next). Removed before training: 4,362 documents by the WikiText-2 / ARC validation+test scan and 1,175 documents marked CC BY-NC-SA. |
| - **Top-up** (11.4M documents, 57.45 GB of text, build manifest `b73f45f5…`): FineWeb-Edu expansion (37.1%, ODC-By 1.0, plus the same dose of ARC-related copies as the pool), FineWeb-Edu (14.9%, ODC-By 1.0), DCLM-baseline (10.9%, CC BY 4.0), StackExchange via RedPajama (6.1%, CC BY-SA content), LoC public-domain books (5.4%, CC0), StarCoderData (5.1%, The Stack terms), Wikipedia 20231101.en (4.6%, CC BY-SA 3.0 / GFDL), Project Gutenberg (2.9%, US public domain), PMC Open Access commercial-use subset (2.8%, CC0 / CC BY / CC BY-SA per article; ND and NC excluded), open-web-math (2.7%, ODC-By), FineMath (2.6%, ODC-By), WildChat-1M (2.0%, ODC-By; English, non-toxic), CC-News (1.4%, licence unknown), UltraChat 200k (0.7%, MIT), tiny-textbooks (0.7%, Apache 2.0). The top-up scan removed 25,808 documents (0.23%), each together with its ARC-related copies. |
| - Licences upstream include ODC-By, CC BY 4.0, CC BY-SA (Wikipedia, StackExchange, some PMC articles), CC0 / public domain, and sources with unknown or restrictive terms. Commercial use: review upstream terms, in particular CC-News, scientific papers, StarCoderData and model-generated chat data (UltraChat, WildChat, tiny-textbooks). |
| - OpenStax textbooks (55 titles) are used under CC BY 4.0; attribution: OpenStax, Rice University (see [OpenStax attribution](#openstax-attribution) below). |
| - **Continued-pretraining data:** 85% the same mix as the pretraining above, 15% WikiText-103 train (Merity et al., 2016, Salesforce; `Salesforce/wikitext`, CC BY-SA 3.0 — text from English Wikipedia). WikiText-103 train shares its source (Wikipedia Good/Featured articles) with the WikiText-2 benchmark but not its test articles; see Limitations. |
| - **Fine-tune data:** ARC-Easy train and ARC-Challenge train from `allenai/ai2_arc` (Clark et al., 2018, Allen Institute for AI), CC BY-SA 4.0. Questions overlapping the ARC-Easy test set were removed from both. ARC-Easy validation and 500 held-out ARC-Easy train questions (never trained on) were used only for selection; the test split was not used for training or selection. |
|
|
| ## Limitations |
|
|
| - English only; small model; one seed. |
| - The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC test set); ARC-Easy is therefore an in-domain benchmark for this model, and it was fine-tuned on ARC-Easy and ARC-Challenge **train**. |
| - Continued pretraining includes WikiText-103 train, which comes from the same Wikipedia article pool as the WikiText-2 benchmark (different articles); the WikiText-2 score (99.98 vs 99.83) should be read with that in mind. |
| - The improvement over Slayer139 1.01 was measured with one test read after selection on validation and held-out train questions; the margin over our Slayer149-ARC (+0.20 Overall) is within noise. |
|
|
| ## OpenStax attribution |
|
|
| The training data of this model includes text extracted from the following OpenStax textbooks, each licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), © Rice University. Download for free at https://openstax.org. The texts were obtained from the Hugging Face dataset crumb/openstax-text (revision 8f502ca45f9f05cb5673eae445b7a97a4e8c4349). Modified: text extracted, chunked and filtered (chunks matching benchmark test/validation sets were removed). |
|
|
| | Title (from source file name) | © year | |
| |---|---| |
| | APBiology | 2018 | |
| | APCollege Physics | 2017 | |
| | APMacroeconomics 2e | 2017 | |
| | APMicroeconomics 2e | 2017 | |
| | Algebra and Trigonometry 2e | 2021 | |
| | American Government 3e | 2021 | |
| | Anatomy and Physiology 2e | 2022 | |
| | Anatomyand Physiology | 2017 | |
| | Astronomy 2e | 2022 | |
| | Astronomy | 2018 | |
| | Biology 2e | 2020 | |
| | Business Ethics | 2018 | |
| | Chemistry 2e | 2019 | |
| | Chemistry Atoms First 2e | 2019 | |
| | College Algebra 2e | 2021 | |
| | College Algebra Corequisite Support 2e | 2021 | |
| | College Physics | 2020 | |
| | College Physics 2e | 2022 | |
| | College Physics for AP Courses 2e | 2022 | |
| | College Success | 2020 | |
| | College Success | 2023 | |
| | Concepts Biology | 2017 | |
| | Contemporary Mathematics | 2023 | |
| | Economics 2e | 2018 | |
| | Economics 3e | 2022 | |
| | Elementary Algebra 2e | 2020 | |
| | Entrepreneurship | 2020 | |
| | Intermediate Algebra 2e | 2020 | |
| | Introduction to Intellectual Property | n/a | |
| | Introduction to Philosophy | 2022 | |
| | Introduction to Political Science | 2022 | |
| | Introductionto Anthropology | 2022 | |
| | Introductionto Sociology 3e | 2021 | |
| | Introductory Business Statistics | 2018 | |
| | Introductory Statistics | 2018 | |
| | Macroeconomics 2e | 2018 | |
| | Macroeconomics 3e | 2022 | |
| | Microbiology | 2021 | |
| | Microeconomics 2e | 2018 | |
| | Microeconomics 3e | 2022 | |
| | Physics | n/a | |
| | Prealgebra 2e | 2020 | |
| | Precalculus 2e | 2021 | |
| | Preparing for College Success | 2023 | |
| | Principles Marketing | 2023 | |
| | Principlesof Finance | 2022 | |
| | Psychology 2e | 2020 | |
| | Statistics | n/a | |
| | USHistory | 2021 | |
| | University Physics Vol 1 | 2021 | |
| | University Physics Volume 2 | 2021 | |
| | University Physics Volume 3 | 2021 | |
| | World History Volume 1 | 2023 | |
| | World History Volume 2 | 2022 | |
| | Writing Guide | 2021 | |
|
|
| Titles licensed CC BY-NC-SA 4.0, non-English titles, and three CC BY 4.0 titles whose text contains elements marked ‘CC BY-NC-SA’ (Introduction to Business, Organizational Behavior, Principles of Management) were not used. |
|
|
| ## How to load |
|
|
| ```python |
| from modeling_slayer139 import load_slayer139, generate |
| model, tok = load_slayer139(".") # directory with config.json, model.safetensors, tokenizer.json |
| print(generate(model, tok, "The capital of France is", max_new_tokens=40)) |
| ``` |
|
|
| Requirements: torch, safetensors, tokenizers. The output head is tied to the token embedding (stored once in `model.safetensors`, re-tied on load). |
|
|
| ## Reproduce |
|
|
| Evaluation: lm-evaluation-harness 0.4.13, tasks `blimp`, `arc_easy`, `wikitext` (board protocol C2), 0-shot. Method and full run history: paper „Same recipe, 4.4× faster on 8× RTX 5090” (Fabryka AI, in preparation). |
|
|
