pretty_name: Slayer1B 1.08 training-data composition, reference v2
language:
- pl
- en
task_categories:
- text-generation
tags:
- continued-pretraining
- prepared-reference
Slayer1B 1.08 — training-data composition
Arkadiusz Słota, Anna Sieczka, Piotr Styła / Fabryka AI (SlayerLab)
Prepared reference v2, 9 November 2026. Target reconstruction must match the approved digests before training. This documents the new 1.08 composition, not a separately hosted mixed-corpus release; the earlier 1.08b experiment is distinct.
Prepared training reference
The composition combines analytical/reasoning-oriented text, English technical material and general-language replay. Retained counts come from the reference assembly manifest; they describe available input positions, not tokens consumed by a training run. Component names identify sources, not an audited per-document language distribution.
| Retained component | Native token positions including EOS |
|---|---|
| PL Logika | 1,631,862,622 |
| English technical | 293,943,541 |
Replay pl_web |
435,000,183 |
Replay en_fineweb_edu |
220,000,324 |
Replay en_mix |
220,000,212 |
| TRAIN total | 2,800,806,882 |
Reference manifest SHA256: c0f1e7771104f60b92ec38d3576fcfe34ed125a072da529d8b56fe2da0f87e7b. TRAIN SHA256: b73a69eaf399bb594300e7d1914a967a38b80b5bc35505fbc6459a6a27be496f.
| Separate partition | Documents | Token positions including EOS |
|---|---|---|
| Polish-logic held-out | 3,158 | 18,294,311 |
| Trainer validation, a subset of that held-out | 360 | 2,002,010 |
| Eligible English VALID held-out | 5,218 | 17,988,388 |
These are partition totals, not domain-evaluation sample sizes.
Published inputs before filtering
The following source-published counts use their stated split and EOS units.
| Source / revision | Split | Documents | Native Slayer tokens |
|---|---|---|---|
PL Logika, e24c8b71526ca1dc56a280100f88c2f09a79ecdd |
published train | 327,219 | 1,911,918,958, including EOS |
English Technical, 3b833d36bd3ba9aeecf46f39605c9ac849bb96a3 |
train | 188,904 | 899,001,323, before pack-added EOS |
| Same English release | valid | 10,370 | 49,886,904, before pack-added EOS |
| Same English release | test | 10,665 | 51,226,632, before pack-added EOS |
The English all-split total, 1,000,114,859 tokens, is not training size. Its pack manifest adds one EOS per document: source TRAIN becomes 899,190,227 token positions. Proxy cl100k counts are a different unit. Source counts, retained reference positions and observed training exposure are distinct quantities.
Replay comes from Slayer 1.07 pretraining data, revision 6a337bbfba518d88f14eb54f7d0a9b9f297df0d1: pl_web, en_fineweb_edu and en_mix. These texts were seen by 1.07; retained quotas are shown above.
Composition and rights
Licenses differ by source; compilation terms do not grant rights to every underlying text. Reference selection excludes GPL/GFDL sources and mixed GPL/LGPL English sources. Retained components still carry their individual terms, not a blanket permissive-license grant.
Polish corpus
Required conditions remain source-specific:
- FinePDFs: retain the notice “Contains information from FinePDFs, which is made available under the ODC Attribution License,” applicable Common Crawl terms and annotation attribution. PDF-content rights remain unresolved; upstream relies on a TDM rationale and did not recheck domain-level opt-outs.
- Dolci: preserve original Dolci, translation and component notices, including NVIDIA Nemotron CC BY 4.0. The translation license does not replace underlying obligations. Attribution identifies Gemma translation and mixed upstream reasoning teachers; embedded conversation-history provenance remains uncertain.
- DOAB: preserve each book's creator/editor, title, publisher, license, URL and modification notices. Selected material includes English books, so the corpus label is not a per-document language guarantee.
- Tax material: preserve Romuald Członkowski and Eureka MF/KIS attribution. Upstream did not locate the portal terms; absence of known restrictions is not confirmed permission.
The pinned attribution inventory carries the full notices; this summary does not replace them.
English technical corpus and replay
English sources span systems, machine learning, compilers, CNC/CAM, simulation and graphics. Row licenses and matching notices govern selection: excluded GPL sources include Linux, Blender, OpenFOAM and some CNC material; retained entries use MIT, Apache-2.0, Apache-2.0 WITH LLVM-exception, BSD-3-Clause, dual Apache/MIT or the FreeCAD repository's CC0 terms. The source-license inventory records the notices. The retained technical count above reflects all filters, not just license exclusion.
FreeCAD documentation: FreeCAD wiki contributors, via FreeCAD/FreeCAD-documentation at 0499378a, repository license CC0-1.0. This identifies the conversion repository's license; the underlying wiki attribution is retained.
Replay includes ODC-By FineWeb-Edu/FineMath/open-web-math, CC BY DCLM and HPLT. HPLT's compilation is CC0, not its underlying web texts; the replay publisher relies on Directive (EU) 2019/790 Article 4. Redistribution requires separate rights review. Required notices follow the pinned replay card.
Preparation and split isolation
Preparation uses the same Slayer v6 tokenizer and little-endian uint16 token format, with EOS 0 after each document.
Reference TRAIN excludes whole documents flagged by the contact/identifier filter or containing literal special-token IDs in their text. Assembly metadata records removal counts and overlapping filters.
English valid/test remain outside TRAIN; eligible VALID uses the TRAIN license exclusions and removes documents also present in TRAIN. Polish held-out sections are grouped by source-book identity rather than splitting chapters across train and evaluation.
Target reconstruction uses the pinned preparation recipe, exclusion lists and the same input revisions. TRAIN, validation and held-out digests must match the approved reference artifacts before use.
Quality, privacy and limitations
The reference uses recorded per-row identifier/overlap exclusions for new sources and inherits the replay release's previous checks. Legacy replay did not detect near-duplicates, omitted EWoK from its scan and retained some short overlaps. These limitations and exposure inherited through the base model remain relevant to evaluation.
Whole-document removal reduces FinePDFs coverage, including academic text with author contact details. Residual names, addresses, translation artifacts, PDF-layout damage and tax-text fragmentation can remain. Polish metadata reports no toxicity classifier and masking exceptions in programming/math contexts. Technical text can contain unsafe instructions; reasoning traces can be wrong. Domain labels and source counts establish neither model capability nor safe professional advice.
Source metadata
Further pinned evidence: Polish aggregate manifest; English aggregate manifest and EOS-inclusive pack manifest. This documentation was prepared from public cards and aggregate/license metadata, not raw records or token binaries.