Slayer139 base (pretrained, step 90,310)

README.md

@@ -0,0 +1,128 @@
+---
+language: [en]
+license: apache-2.0
+library_name: pytorch
+tags: [slayer, small-language-model, glint-tiny-ml, fabryka-ai]
+datasets: [SlayerLab/gollem-v5-arcmix-9b, SlayerLab/minimal-en-corpus-5b, HuggingFaceFW/fineweb-edu, mlfoundations/dclm-baseline-1.0-parquet]
+---
+
+# Slayer139 base
+
+A small decoder-only English language model (139.3M parameters) trained from scratch by Fabryka AI. This is the **pretrained base** of [Slayer139 1.01](https://huggingface.co/SlayerLab/Slayer139-1.01) and [Slayer139 1.02](https://huggingface.co/SlayerLab/Slayer139-1.02): the end of the 8× RTX 5090 pretraining run (step 90,310), with **no fine-tuning** of any kind. Slayer139 1.01 is this model fine-tuned on ARC-Easy train; Slayer139 1.02 is this model after continued pretraining and a fine-tune. [Slayer139 1.01b](https://huggingface.co/SlayerLab/Slayer139-1.01b) is a separate run with the same recipe on a smaller machine, not this checkpoint.
+
+**Author:** Arkadiusz Słota (Fabryka AI / SlayerLab). Training, evaluation and safety gates were run by the Fabryka AI agent team under his direction.
+
+## Result
+
+lm-evaluation-harness 0.4.13, board protocol (C2), our measurement on 1× RTX 5090; the same checkpoint measured on a second machine (AMD Radeon) gave the same scores.
+
+| measure | Slayer139 base | 95% CI |
+|---|---|---|
+| BLiMP (67 tasks) | 79.41 | [79.14; 79.69] |
+| ARC-Easy (test, 2,376) | 57.58 | [55.60; 59.55] |
+| WikiText-2 score | 99.86 | |
+| **Overall** | **78.95** | [78.28; 79.62] |
+
+Overall is the mean of the three measures. Confidence intervals: item-level bootstrap (ARC-Easy and each of the 67 BLiMP tasks resampled separately, 10,000 draws). They cover test-set sampling only, not seed-to-seed variation; the run is a single seed. These are the same numbers as the "pretrained base" column in the Slayer139 1.01 and 1.01b cards.
+
+Provenance (sha256 prefixes): lm-eval `results.json` RTX 5090 `fabe34ce`, AMD Radeon `719bd11c`; per-item samples RTX 5090 `19f78648`, AMD Radeon `00db3761`.
+
+Numbers are our measurements with the board's protocol, not official scores.
+
+## Model
+
+- Architecture: 17 layers × 768, 12 heads (12 KV heads), SwiGLU, decoder-only; vocabulary 24,576 (BPE), context 1,024.
+- Parameters: 139,279,760 unique parameters (input and output embeddings tied: one shared tensor, counted once; counting the shared tensor twice gives 158,154,128).
+- Checkpoint: the final step 90,310 of the run (sha256 `ba223162…`), the end of the run rather than a selected step (rule written before any test number). `model.safetensors` holds the same weights as that checkpoint (205 of 205 tensors identical; the output head is the tied input embedding).
+- Code: `modeling_slayer139.py` (inference only, plain PyTorch), the same file as in Slayer139 1.01b.
+
+## Training
+
+- Data: ~23.67B tokens, no repeated epoch (sources and licences below). First the ARC-MIX pool, then a top-up from the same source families in the base corpus's proportions, mixed at document level. Both stages were scanned against WikiText-2 (test, validation) and ARC-Easy/Challenge (validation, test) for shared normalized 13-grams, with short ARC questions matched exactly, and against BLiMP sentences of ≥ 6 words; matching documents were removed. ARC train was not filtered.
+- Optimiser and schedule: Muon (hidden matrices, LR 0.0566) + AdamW (other parameters, LR 1.6971e-3), both ×√8 from the single-node recipe; WSD schedule: 250 warm-up steps, constant to step 72,248, then 1−√ decay to 10% of peak at step 90,310; batch 256 × 1,024 tokens; seed 1337.
+- Same recipe as our single-node final (batch 32, 4× RTX 5090, ~191k tokens/s), scaled to batch 256 with LR ×√8 and the identical data order — 4.4× faster. At equal tokens the 8-GPU run had lower held-out DCLM bits-per-byte at every compared checkpoint (−0.014 at 1.1B tokens, −0.032 at 19.4B tokens).
+- Hardware and time: 8× NVIDIA RTX 5090 (data parallel, bf16 all-reduce), median ~832k tokens/s, about 8 hours of training.
+- Training curves: https://track.fabryka.ai/run/5a4d174e-4a9d-4cc1-9602-838a3ffbb917
+
+## Training data
+
+Aggregate corpus; each source keeps its upstream terms.
+
+- **ARC-MIX pool** (`SlayerLab/gollem-v5-arcmix-9b`, 9.39B tokens before retokenization): the 15-source base `SlayerLab/minimal-en-corpus-5b` (FineWeb-Edu, DCLM-baseline, open-web-math, FineMath, StarCoderData, StackExchange, LoC public-domain books, Wikipedia, Project Gutenberg, scientific papers, UltraChat, WildChat, CC-News, tiny-textbooks, OpenSubtitles) + a FineWeb-Edu expansion + OpenStax textbooks (×4) + extra copies of FineWeb-Edu documents scored as related to ARC science topics (+2 copies for the highest score, +1 for the next). Removed before training: 4,362 documents by the WikiText-2 / ARC validation+test scan and 1,175 documents marked CC BY-NC-SA.
+- **Top-up** (11.4M documents, 57.45 GB of text, build manifest `b73f45f5…`): FineWeb-Edu expansion (37.1%, ODC-By 1.0, plus the same dose of ARC-related copies as the pool), FineWeb-Edu (14.9%, ODC-By 1.0), DCLM-baseline (10.9%, CC BY 4.0), StackExchange via RedPajama (6.1%, CC BY-SA content), LoC public-domain books (5.4%, CC0), StarCoderData (5.1%, The Stack terms), Wikipedia 20231101.en (4.6%, CC BY-SA 3.0 / GFDL), Project Gutenberg (2.9%, US public domain), PMC Open Access commercial-use subset (2.8%, CC0 / CC BY / CC BY-SA per article; ND and NC excluded), open-web-math (2.7%, ODC-By), FineMath (2.6%, ODC-By), WildChat-1M (2.0%, ODC-By; English, non-toxic), CC-News (1.4%, licence unknown), UltraChat 200k (0.7%, MIT), tiny-textbooks (0.7%, Apache 2.0). The top-up scan removed 25,808 documents (0.23%), each together with its ARC-related copies.
+- Licences upstream include ODC-By, CC BY 4.0, CC BY-SA (Wikipedia, StackExchange, some PMC articles), CC0 / public domain, and sources with unknown or restrictive terms. Commercial use: review upstream terms, in particular CC-News, scientific papers, StarCoderData and model-generated chat data (UltraChat, WildChat, tiny-textbooks).
+- OpenStax textbooks (55 titles) are used under CC BY 4.0; attribution: OpenStax, Rice University (see [OpenStax attribution](#openstax-attribution) below).
+
+## Limitations
+
+- English only; small model; one seed.
+- The pretraining mix deliberately upweights web documents related to ARC science topics (decontaminated against the ARC test set); ARC-Easy is therefore an in-domain benchmark for this model.
+- Not instruction-tuned and not fine-tuned on any benchmark.
+
+## OpenStax attribution
+
+The training data of this model includes text extracted from the following OpenStax textbooks, each licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0), © Rice University. Download for free at https://openstax.org. The texts were obtained from the Hugging Face dataset crumb/openstax-text (revision 8f502ca45f9f05cb5673eae445b7a97a4e8c4349). Modified: text extracted, chunked and filtered (chunks matching benchmark test/validation sets were removed).
+
+| Title (from source file name) | © year |
+|---|---|
+| APBiology | 2018 |
+| APCollege Physics | 2017 |
+| APMacroeconomics 2e | 2017 |
+| APMicroeconomics 2e | 2017 |
+| Algebra and Trigonometry 2e | 2021 |
+| American Government 3e | 2021 |
+| Anatomy and Physiology 2e | 2022 |
+| Anatomyand Physiology | 2017 |
+| Astronomy 2e | 2022 |
+| Astronomy | 2018 |
+| Biology 2e | 2020 |
+| Business Ethics | 2018 |
+| Chemistry 2e | 2019 |
+| Chemistry Atoms First 2e | 2019 |
+| College Algebra 2e | 2021 |
+| College Algebra Corequisite Support 2e | 2021 |
+| College Physics | 2020 |
+| College Physics 2e | 2022 |
+| College Physics for AP Courses 2e | 2022 |
+| College Success | 2020 |
+| College Success | 2023 |
+| Concepts Biology | 2017 |
+| Contemporary Mathematics | 2023 |
+| Economics 2e | 2018 |
+| Economics 3e | 2022 |
+| Elementary Algebra 2e | 2020 |
+| Entrepreneurship | 2020 |
+| Intermediate Algebra 2e | 2020 |
+| Introduction to Intellectual Property | n/a |
+| Introduction to Philosophy | 2022 |
+| Introduction to Political Science | 2022 |
+| Introductionto Anthropology | 2022 |
+| Introductionto Sociology 3e | 2021 |
+| Introductory Business Statistics | 2018 |
+| Introductory Statistics | 2018 |
+| Macroeconomics 2e | 2018 |
+| Macroeconomics 3e | 2022 |
+| Microbiology | 2021 |
+| Microeconomics 2e | 2018 |
+| Microeconomics 3e | 2022 |
+| Physics | n/a |
+| Prealgebra 2e | 2020 |
+| Precalculus 2e | 2021 |
+| Preparing for College Success | 2023 |
+| Principles Marketing | 2023 |
+| Principlesof Finance | 2022 |
+| Psychology 2e | 2020 |
+| Statistics | n/a |
+| USHistory | 2021 |
+| University Physics Vol 1 | 2021 |
+| University Physics Volume 2 | 2021 |
+| University Physics Volume 3 | 2021 |
+| World History Volume 1 | 2023 |
+| World History Volume 2 | 2022 |
+| Writing Guide | 2021 |
+
+Titles licensed CC BY-NC-SA 4.0, non-English titles, and three CC BY 4.0 titles whose text contains elements marked ‘CC BY-NC-SA’ (Introduction to Business, Organizational Behavior, Principles of Management) were not used.
+
+## Reproduce
+
+Evaluation: lm-evaluation-harness 0.4.13, tasks `blimp`, `arc_easy`, `wikitext` (board protocol C2); per-item samples are kept for paired bootstrap. Method and full run history: paper „Same recipe, 4.4× faster on 8× RTX 5090” (Fabryka AI, in preparation).

config.json

@@ -0,0 +1,19 @@
+{
+  "block": 1024,
+  "ffn": "swiglu",
+  "ffn_mult": 2.667,
+  "logit_cap": 0.0,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_layer": 17,
+  "norm": "rmsnorm",
+  "norm_eps": 1e-06,
+  "pos": "rope",
+  "qk_norm": true,
+  "rope_theta": 100000.0,
+  "tied_weights": {
+    "tok.weight": "head.weight"
+  },
+  "value_residual": true,
+  "vocab": 24576
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f011c9b842640f23bc8530c29f223ecce9ba224b3c6433083d7dd6005b45e772
+size 557137920

modeling_slayer139.py

@@ -0,0 +1,174 @@
+"""Slayer139 1.01 model code (inference only), extracted from the training script used for this run.
+
+The module definitions are those of the training script; only the weight initialisation and the training loss were
+removed and comments translated. Weights load with strict=True. Requirements: torch, safetensors, tokenizers.
+
+    from modeling_slayer139 import load_slayer139, generate
+    model, tok = load_slayer139(".")          # directory with config.json, model.safetensors, tokenizer.json
+    print(generate(model, tok, "The capital of France is", max_new_tokens=40))
+
+Notes: RoPE uses the interleaved convention (pairs of adjacent channels), not rotate_half; the output head is tied to
+the token embedding; context length is 1,024 tokens.
+"""
+from __future__ import annotations
+
+import json
+import os
+import types
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class RMSNorm(nn.Module):
+    """RMSNorm with fp32 compute for stability."""
+    def __init__(self, d, eps=1e-6):
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(d))
+        self.eps = eps
+
+    def forward(self, x):
+        return x * torch.rsqrt(x.float().pow(2).mean(-1, keepdim=True) + self.eps).to(x.dtype) * self.weight
+
+
+def make_norm(d, cfg):
+    return RMSNorm(d, cfg.norm_eps) if cfg.norm == "rmsnorm" else nn.LayerNorm(d)
+
+
+def apply_rope(x, base=100000.0):
+    """Parameter-free RoPE on [B, H, T, D], interleaved convention (even/odd channel pairs)."""
+    _, _, T, dim = x.shape
+    pos = torch.arange(T, device=x.device, dtype=torch.float32)
+    freq = 1.0 / (base ** (torch.arange(0, dim, 2, device=x.device, dtype=torch.float32) / dim))
+    ang = torch.outer(pos, freq)
+    cos, sin = ang.cos().to(x.dtype)[None, None], ang.sin().to(x.dtype)[None, None]
+    even, odd = x[..., ::2], x[..., 1::2]
+    return torch.stack((even * cos - odd * sin, even * sin + odd * cos), dim=-1).flatten(-2)
+
+
+class SwiGLU(nn.Module):
+    """Gated MLP: down(silu(gate(x)) * up(x))."""
+    def __init__(self, d, hidden):
+        super().__init__()
+        self.gate = nn.Linear(d, hidden, bias=False)
+        self.up = nn.Linear(d, hidden, bias=False)
+        self.down = nn.Linear(hidden, d, bias=False)
+
+    def forward(self, x):
+        return self.down(F.silu(self.gate(x)) * self.up(x))
+
+
+class Block(nn.Module):
+    def __init__(self, d, nh, block, cfg, is_first=False):
+        super().__init__()
+        self.ln1 = make_norm(d, cfg)
+        self.ln2 = make_norm(d, cfg)
+        self.qkv = nn.Linear(d, 3 * d)
+        self.proj = nn.Linear(d, d)
+        if cfg.ffn == "swiglu":
+            self.mlp = SwiGLU(d, int(round(cfg.ffn_mult * d)))
+        else:
+            self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
+        self.nh = nh
+        self.d = d
+        self.cfg = cfg
+        self.is_first = is_first
+        if cfg.value_residual and not is_first:
+            self.vr_lambda = nn.Parameter(torch.zeros(1))
+        if cfg.qk_norm:
+            hd = d // nh
+            self.q_norm = RMSNorm(hd, cfg.norm_eps)
+            self.k_norm = RMSNorm(hd, cfg.norm_eps)
+
+    def forward(self, x, v0=None):
+        B, T, D = x.size()
+        h = self.ln1(x)
+        q, k, v = self.qkv(h).split(self.d, dim=2)
+        hd = D // self.nh
+        q = q.view(B, T, self.nh, hd).transpose(1, 2)
+        k = k.view(B, T, self.nh, hd).transpose(1, 2)
+        v = v.view(B, T, self.nh, hd).transpose(1, 2)
+        if self.cfg.qk_norm:
+            q = self.q_norm(q)
+            k = self.k_norm(k)
+        if self.cfg.pos == "rope":
+            q = apply_rope(q, self.cfg.rope_theta)
+            k = apply_rope(k, self.cfg.rope_theta)
+        if self.cfg.value_residual:
+            if self.is_first:
+                v0 = v
+            else:
+                v = v + self.vr_lambda * v0
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        y = y.transpose(1, 2).contiguous().view(B, T, D)
+        x = x + self.proj(y)
+        x = x + self.mlp(self.ln2(x))
+        return x, v0
+
+
+class GPT(nn.Module):
+    def __init__(self, vocab, n_layer, n_embd, n_head, block, cfg):
+        super().__init__()
+        self.cfg = cfg
+        self.tok = nn.Embedding(vocab, n_embd)
+        self.use_rope = cfg.pos == "rope"
+        if not self.use_rope:
+            self.pos = nn.Embedding(block, n_embd)
+        self.blocks = nn.ModuleList([Block(n_embd, n_head, block, cfg, is_first=(i == 0)) for i in range(n_layer)])
+        self.lnf = make_norm(n_embd, cfg)
+        self.head = nn.Linear(n_embd, vocab, bias=False)
+        self.head.weight = self.tok.weight  # tied
+        self.block = block
+
+    def forward(self, idx):
+        B, T = idx.size()
+        x = self.tok(idx)
+        if not self.use_rope:
+            pos = torch.arange(T, device=idx.device)
+            x = x + self.pos(pos)[None]
+        v0 = None
+        for b in self.blocks:
+            x, v0 = b(x, v0)
+        logits = self.head(self.lnf(x))
+        cap = getattr(self.cfg, "logit_cap", 0.0)
+        if cap and cap > 0:
+            logits = cap * torch.tanh(logits / cap)
+        return logits
+
+
+def load_slayer139(path, device="cpu"):
+    """Model (eval mode) and tokenizer from a directory with config.json, model.safetensors and tokenizer.json."""
+    from safetensors.torch import load_file
+    from tokenizers import Tokenizer
+
+    with open(os.path.join(path, "config.json"), encoding="utf-8") as f:
+        cfg = types.SimpleNamespace(**json.load(f))
+    model = GPT(cfg.vocab, cfg.n_layer, cfg.n_embd, cfg.n_head, cfg.block, cfg)
+    state = load_file(os.path.join(path, "model.safetensors"))
+    # the output head is tied to the token embedding and stored once (tok.weight); restore the tied key for strict loading
+    if "head.weight" in state:
+        raise ValueError("model.safetensors must not contain head.weight (tied to tok.weight, stored once)")
+    state["head.weight"] = state["tok.weight"]
+    model.load_state_dict(state, strict=True)
+    return model.to(device).eval(), Tokenizer.from_file(os.path.join(path, "tokenizer.json"))
+
+
+@torch.no_grad()
+def generate(model, tok, prompt, max_new_tokens=64, temperature=0.0, top_k=None, seed=None):
+    """Continue `prompt`. temperature 0 = greedy; otherwise sampling, optionally top-k."""
+    g = torch.Generator().manual_seed(seed) if seed is not None else None
+    dev = next(model.parameters()).device
+    x = torch.tensor([tok.encode(prompt).ids], device=dev)
+    for _ in range(max_new_tokens):
+        logits = model(x[:, -model.block:])[:, -1].float().cpu()
+        if temperature <= 0:
+            nxt = logits.argmax(-1, keepdim=True)
+        else:
+            logits = logits / temperature
+            if top_k:
+                kth = torch.topk(logits, top_k).values[:, -1:]
+                logits = logits.masked_fill(logits < kth, float("-inf"))
+            nxt = torch.multinomial(torch.softmax(logits, -1), 1, generator=g)
+        x = torch.cat([x, nxt.to(dev)], dim=1)
+    return tok.decode(x[0].tolist())