| --- |
| license: cc-by-sa-4.0 |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - tiny-lm |
| - arc-easy |
| - fine-tuned |
| datasets: |
| - allenai/qasc |
| - allenai/cosmos_qa |
| - allenai/quoref |
| - allenai/ropes |
| - tau/commonsense_qa |
| - rajpurkar/squad |
| - ucinlp/drop |
| - google/boolq |
| - allenai/ai2_arc |
| --- |
| |
| # Slayer9-DQA-9M |
|
|
| An 8,786,185-parameter decoder-only language model by Fabryka AI (Arkadiusz Słota), trained from scratch, annealed |
| with a question-answering mix and fine-tuned on ARC train. It is a research artifact for the Glint Tiny-ML leaderboard, |
| not a general-purpose or instruction-following model. |
|
|
| **Name.** *Slayer* is the SlayerLab model line; *9* is the ~9M-parameter family of this line; *DQA* stands for |
| decay-phase QA: question-answering data (reading passages with their questions, and short multiple-choice questions) |
| placed in the decay (annealing) phase of training, which is what distinguishes this recipe from our baseline; this |
| release uses the licence-clean QA pool listed below. *9M* is the size (8.79M unique parameters, the tied input/output |
| embedding counted once). |
|
|
| ## Evaluation |
|
|
| **Status: test split read once for the selected checkpoint (seed 1373, see below). Weights are not uploaded yet.** |
|
|
| ### Selected checkpoint (test, read once) |
|
|
| | metric | Glint-1.3 protocol | lm-evaluation-harness 0.4.13 (second column) | |
| |---|---|---| |
| | BLiMP (67 tasks, 67,000 pairs) | 73.83 | not measured | |
| | ARC-Easy test (2,376 questions) | 44.19 | not measured | |
| | WikiText-2 test, byte perplexity | 2.3971 | not measured | |
| | Overall / Efficiency (leaderboard formulas) | 71.16 / 79.60 | — | |
|
|
| For comparison, the leaderboard row of Qielbas Tiny 9M M2 (as of 2026-10-12) is BLiMP 75.68, ARC-Easy 47.43, |
| WikiText-2 2.3367, Efficiency 81.42; this model does not beat it. |
|
|
| **Validation overestimated this model.** Its validation Efficiency was 81.88; on test it is 79.60. Two causes, both |
| visible in the numbers: ARC-Easy 46.84 (validation) vs 44.19 (test), expected from choosing the best of fourteen |
| variants on validation; and BLiMP 77.24 on the validation half (first 34 tasks) vs 73.83 on all 67 tasks. For Qielbas |
| M2 the same two BLiMP numbers are 77.65 on our validation half A and 75.68 on the leaderboard (all 67 tasks), so the |
| half-A validation axis overestimates this recipe more than it overestimates M2. |
|
|
| Glint-1.3: zero-shot raw summed log-likelihood of the answer, first 256 tokens, raw accuracy; the protocol adds no |
| BOS of its own (the tokenizer's template does, see Model). lm-evaluation-harness uses a different prompt |
| (`Question: …\nAnswer:`) and reports `acc`; the two columns are not comparable as identical measurements. The |
| leaderboard row uses the Glint-1.3 column. |
|
|
| ### Runs so far (validation split, descriptive) |
|
|
| Validation axis: ARC-Easy validation (570 questions), BLiMP half A (first 34 of the 67 BLiMP tasks, 34,000 pairs) and |
| WikiText-2 validation byte perplexity; the test axis is ARC-Easy test, full BLiMP and WikiText-2 test. Mean ± SD over |
| seeds. The baseline recipe is identical except that its annealing contains no question-answering pool. |
|
|
| | stage | n seeds | ARC-E val | BLiMP-A | WikiText-2 val byte ppl | Efficiency (val) | |
| |---|---|---|---|---|---| |
| | baseline recipe, pretrained | 7 | 40.37 ± 1.21 | 77.22 ± 0.60 | 2.3660 ± 0.0038 | 79.53 ± 0.56 | |
| | this recipe (QA annealing), pretrained | 7 | 40.40 ± 2.05 | 78.58 ± 0.63 | 2.3662 ± 0.0043 | 80.05 ± 0.82 | |
| | baseline recipe + ARC fine-tune | 7 | 42.96 ± 1.68 | 76.83 ± 0.57 | 2.3803 ± 0.0041 | 80.31 ± 0.74 | |
| | this recipe + ARC fine-tune | 7 | 43.58 ± 1.59 | 78.10 ± 0.57 | 2.3811 ± 0.0026 | 81.01 ± 0.67 | |
| | **Δ (this − baseline) after ARC fine-tune, paired by seed** | 7 | +0.62 ± 2.72 (5/7 > 0) | +1.27 ± 0.99 (7/7) | +0.001 ± 0.004 | **+0.70 ± 1.15 (5/7)** | |
| | this recipe + ARC fine-tune + 1 extra ARC-Easy epoch | 7 | 44.61 ± 1.64 | 77.95 ± 0.53 | 2.3874 ± 0.0037 | 81.32 ± 0.61 | |
| | **Δ (extra ARC-Easy epoch − ARC fine-tune), paired by seed** | 7 | | | | **+0.31 ± 0.39 (5/7)** | |
|
|
| Validation Efficiency of the seven QA-annealed models after the ARC fine-tune (seeds 1369–1375): 80.93 / 81.07 / |
| 79.80 / 80.62 / 81.20 / 81.76 / 81.72; after the extra ARC-Easy epoch: 81.69 / 81.56 / 80.26 / 80.69 / 81.88 / |
| 81.75 / 81.44. |
|
|
| **How the checkpoint was chosen.** Seven training seeds (1369–1375) were run with the same recipe. Rule (1) was fixed |
| before any result was known: take the seed with the highest leaderboard Efficiency on the validation axis (Efficiency |
| formula on ARC-Easy validation, BLiMP half A and WikiText-2 validation byte perplexity) after the ARC fine-tune |
| (ties → lower seed). Rule (2) was added at the owner's request after the pre-fine-tune validation results were known, |
| but before any fine-tuned or extra-epoch result was known: one extra epoch on ARC-Easy train only replaces the |
| fine-tune for all seeds if it improves validation Efficiency by at least +0.20 on average and for at least 5 of 7 |
| seeds, in which case the highest of the seven extended models is taken. Rule (2) was met (+0.31, 5/7), so the selected |
| checkpoint is seed 1373 with the extra epoch (validation Efficiency 81.88). The test-axis evaluation was run **once**, |
| for that checkpoint only; the test splits were never used for training, fine-tuning, epoch choice or seed choice. The |
| test split was read without a separate confirmation run on fresh seeds (owner's decision). Choosing the best of |
| fourteen variants on validation inflated its validation number; the test numbers above are the result. |
|
|
| ## Model |
|
|
| - 10 layers, d_model 224, 4 heads, RoPE (θ 100,000, interleaved), SwiGLU (×2.667), RMSNorm, QK-norm, |
| value residual; output head tied to the token embedding (stored once). Context 1,024 tokens. |
| - Tokenizer: BPE, 12,288 tokens (`tokenizer.json`, sha256 `235abda6ba9591b7…`). Its template prepends `<|endoftext|>` |
| (id 12285) to every encoded text: in training every document was preceded by this separator, so evaluation starts |
| the same way. Training data were tokenized without the template (the separator came from the document stream); all |
| reported scores use the tokenizer as shipped (default `encode()`). |
| - Files (after selection): `model.safetensors` (fp32 weights only), `config.json`, `tokenizer.json`, inference code, |
| `README.md`. Loading reproduces the logits of the training checkpoint exactly. |
| |
| ## Training |
| |
| 1. **Pretraining + annealing, 2.0B tokens total** (8,720 steps × 229,376 tokens). Pretraining mix: a subset of |
| `SlayerLab/minimal-en-corpus-5b` (by Dawid Majewski) restricted to FineWeb-Edu, DCLM-baseline and Wikipedia |
| documents (about 47 / 36 / 16 percent of tokens; 2.29B tokens available, ~0.87 epoch used); documents overlapping |
| WikiText-2 test/validation or ARC-Easy/Challenge test/validation/train were removed before selection (13-gram scan; |
| BLiMP was not scanned). |
| 2. **QA annealing** (final 1,744 steps, 400M tokens): 19.0 percent from a question-answering pool (each passage once, |
| followed by all of its questions and answers; 4 passes over 18.99M unique tokens), 15.0 percent WikiText-103 train |
| (documents overlapping the union of evaluation test sets removed by a 13-gram filter), 66.0 percent the pretraining |
| mix. Pool sources (train splits only): QASC, Cosmos QA, Quoref, ROPES (CC BY 4.0), CommonsenseQA (MIT), SQuAD v1.1, |
| DROP (CC BY-SA 4.0), BoolQ (CC BY-SA 3.0). Questions that appear in ARC train were removed; 72 documents overlapping |
| WikiText-2 test/validation or ARC-Easy/Challenge test/validation/train were removed by a 13-gram scan (BLiMP was |
| not scanned). |
| 3. **ARC fine-tune**: ARC-Easy + ARC-Challenge train (CC BY-SA 4.0; questions overlapping ARC test/validation |
| removed), listwise cross-entropy over raw answer likelihoods, 2 epochs (fixed, not selected), AdamW lr 5e-5 |
| (constant), 15 percent language-model replay from the annealing mix; then **one extra epoch on ARC-Easy train only** |
| with the same settings (adopted by rule (2) above). |
| |
| ## Licence and attribution |
| |
| Weights: CC BY-SA 4.0 (share-alike sources in training: Wikipedia and WikiText-103 (CC BY-SA 3.0 + GFDL), ARC, |
| SQuAD, DROP (CC BY-SA 4.0), BoolQ (CC BY-SA 3.0)). Other sources: FineWeb-Edu (ODC-By), DCLM-baseline (CC BY 4.0), |
| QASC, Cosmos QA, Quoref, ROPES (CC BY 4.0), CommonsenseQA (MIT). The pretraining corpus keeps each document's upstream |
| source and terms. |
| |
| Sources (repository @ revision, licence): QASC `allenai/qasc`@a34ba204 CC BY 4.0 · Cosmos QA `allenai/cosmos_qa`@ed50fe83 |
| (parquet conversion) CC BY 4.0 · Quoref `allenai/quoref`@41fed5bc CC BY 4.0 · ROPES `allenai/ropes`@d59f1e2e CC BY 4.0 · |
| CommonsenseQA `tau/commonsense_qa`@94630fe3 MIT · SQuAD `rajpurkar/squad`@7b6d24c4 CC BY-SA 4.0 · DROP |
| `ucinlp/drop`@95cda593 CC BY-SA 4.0 · BoolQ `google/boolq`@35b264d0 CC BY-SA 3.0 · ARC `allenai/ai2_arc`@210d026f |
| CC BY-SA 4.0 · pretraining corpus `SlayerLab/minimal-en-corpus-5b`@bb3aec23, built from FineWeb-Edu |
| `HuggingFaceFW/fineweb-edu`@87f09149 ODC-By, DCLM-baseline `mlfoundations/dclm-baseline-1.0-parquet`@817d6752 CC BY 4.0 |
| and Wikipedia `wikimedia/wikipedia`@b04c8d1c (20231101.en) CC BY-SA 3.0 + GFDL · WikiText-103 |
| `Salesforce/wikitext`@b08601e0 CC BY-SA 3.0 + GFDL. |
|
|
| ## Limitations |
|
|
| Very small model (8.8M parameters); trained on the leaderboard's ARC task (in-domain fine-tune, described above); |
| one selected seed out of seven. |
|
|
