pretty_name: Slayer1B 1.09 staged training-data composition
language:
- pl
- en
task_categories:
- text-generation
tags:
- pretraining
- staged-training-data
Slayer1B 1.09 — staged training data
Arkadiusz Słota, Anna Sieczka, Piotr Styła / Fabryka AI (SlayerLab)
Historical CPT data snapshot, 10 November 2026, 19:23 UTC: the continuation from Slayer1B 1.08 was paused by the cost guard after the on-node update-11,600 checkpoint. At that snapshot, weights-only and full recovery checkpoints at update 10,000 had been verified on the trusted local workstation; this does not assert a local update-11,600 copy. The c01–c06 queue was byte-approved and reproduced on the training host, providing data through update 30,000. Input capacity is distinct from consumed training exposure. This dated snapshot does not describe current hardware availability, a completed future corpus or achieved over-100B content-unique history; later execution status belongs to the linked CPT Track history and operator reports.
Target, units and exposure
Funding and planning update — 11 November 2026, 15:29–15:41 UTC: an additional USD 500 was reported received for continued main-base training. This is not the current account balance; the earlier USD 425 envelope is a dated planning record, not a new remaining-budget calculation. The goal remains maximum useful affordable exposure with constant learning rates after warmup and no in-run terminal decay, retaining cost guards and the operational reserve. The approved operator recipe selects KROKI=120000 with decay from update 96,000; at the 15:41 UTC snapshot the running process had not yet adopted it and still used 80,000. Track's separate 80,000-update plan is an estimate, not a funded endpoint or evidence of execution. Review the schedule before its decay boundary becomes reachable; do not bring decay forward by shortening the scheduling argument. Source allocation targets 20% PL / 80% EN, not the cumulative model-language ratio.
Prepared-data inventory — 11 November 2026, 14:32 UTC, v6 token positions: the dataset operator reported the following aggregate counts. Prepared input capacity, reported training progress and unfinished material are different states; rows overlap as labelled and must not all be summed.
| Inventory measure | Positions | Scope |
|---|---|---|
| Built shard inputs c01–c10 | 53,477,709,553 | Total prepared inputs, including positions already used |
| Reported training progress at update 29,040 | 30,450,647,040 | Logical progress telemetry, not proof of the newest saved checkpoint |
| Remaining prepared input positions at that snapshot | 23,027,062,513 | Part of the built input total, not an additional corpus |
| Processed material not yet assembled into shards | 5,618,606,566 | Not a ready base shard; composition, gold exclusions and byte approval still apply |
| Gap from built inputs to 100B prepared positions | 46,522,290,447 | More is required to exceed 100B; this target was not yet achieved |
The requirement is over 100B prepared base-data positions, not merely a scheduling argument, available upstream text or inherited exposure. At 5,000 updates per slot and 1,048,576 positions per update, ten slots process 52,428,800,000 positions through update 50,000, less than the 53,477,709,553-position input capacity. Extra input tails do not automatically extend the existing schedule to update 51,000. A checkpoint recovery can repeat work already present in earlier telemetry; that repeated work is not new distinct data. Source/file-disjoint ancestry counts do not establish a globally content-unique corpus.
Separate ongoing gold-data curation — not a finalized training corpus
The 11 November, 12:07 and 14:29 UTC decisions call for continuous curation of approximately 10B tokens of quality-selected data for a separate finishing/pretraining run at context 4,096. The dataset team curates candidates, including books selected with Arkadiusz Słota only after title-level license approval. Gold is isolated from main-base inputs and evaluation material, with its own provenance, manifest and licensing/admission gate; training it requires a separate run and budget decision.
The operator's 14:26 UTC report identified approximately 0.26B v6 English positions from s17 reserved by exclusion lists, not a complete packed gold corpus. Gold English candidates come from sources not used by c01–c10; those base shards remain unchanged. Each later base shard must exclude the relevant gold-selected rows. Source pools may be curated separately, but proposed training slots mix approximately 80% EN / 20% PL rather than training whole English-only and Polish-only slots in succession. The main-base inventory above must not also count reserved gold as available base data.
No complete 10B gold corpus or training release is claimed, and no separate training budget has been approved. The reported additional USD 500 supports the main base, not a gold-training allocation. “Gold” describes intended quality-based selection, not a measured performance gain. Published upstream capacities and unfinished candidate material do not establish retained size or readiness.
Historical byte-approved queue snapshot c01–c06
| Segment | Input positions | Selected Polish source | TRAIN SHA256 |
|---|---|---|---|
| c01 | 5,347,737,900 | GoLLeM web | 3341080241aeabd1e03c77530a6d735cb9501ecd788fba60ea065a944cd09ee1 |
| c02 v3 | 5,347,740,964 | GoLLeM web | 0b064039733dc7d61684a8def5859fc166f7d34331ee2659a97cbad96950f23b |
| c03 v4 | 5,347,740,627 | FinePDFs-Edu PL | cf477bd496fcb430d7e5e9980edfab3f43899b11bb246fcf6258f04d91003821 |
| c04 | 5,347,852,265 | FinePDFs-Edu PL | f2eaf21d7482a9646c32d580244ef581af5e080c86bd646e096a3a8cf4d444f7 |
| c05 v2 | 5,347,778,178 | FinePDFs-Edu PL | 6e457d8a084f7eef24cd95be52456f920a81376283c39b6e3a55aa63a1f9b9e6 |
| c06 | 5,347,899,736 | FinePDFs-Edu PL and GoLLeM web | 9d5df696f146dd0546858d635c4ed2518dc8d5ebc002d3fc80b9f7ef849dbd9e |
English-source input is FineWeb-Edu. This historical snapshot has 32,086,749,670 positions of approved input capacity, including EOS, and queue SHA256 5b1c2755aa79db7fb6d5dcb40b17bb65465dae36fcddfe3db3f9d8e44d51cc9a. It contains 5,000 updates per segment and contiguous logical ranges. Six completed stages process 31,457,280,000 positions, less than their input capacity. This digest does not identify the later growing queue; each appended segment still requires its own approval.
The source selection changed at the c02–c03 boundary. Training-loss values across this boundary do not measure the same input distribution. The historical 10 November snapshot records a verified transfer receipt for full local recovery state at update 10,000; neither that receipt nor a lower training loss establishes final model quality.
Byte-approved c01 reference — input capacity, not completed exposure
| Source category | Native input positions |
|---|---|
| English educational web: FineWeb-Edu | 4,278,190,159 |
| Polish web: GoLLeM, after ancestral exclusions | 1,069,547,741 |
| Total c01 capacity | 5,347,737,900 |
The source allocation is approximately 80% EN / 20% PL; this is not per-document language verification or the cumulative language ratio of inherited exposure. Each completed 5,000-update stage processes 5,242,880,000 positions. The c01 receipt identifies only the initial segment; the later queue snapshot above has separate approvals.
TRAIN SHA256: 3341080241aeabd1e03c77530a6d735cb9501ecd788fba60ea065a944cd09ee1; one-slot queue SHA256: 48144a407bcd3b2da8ad9106e0f8d94eba4335bf08dcbb0492f99acbdce91d98; construction manifest SHA256: e924f775a9b471e50a9cc2185d7da117a508200c72f22c257550dba0472ec4e8.
Runtime validation is a separate fresh CL split: expected val.bin SHA256 4c50400b983b41411ebf2f68c09b1ab554517a99af270b731a84c80f4a2636a1, 2,004,526 positions. The construction manifest's legacy Logika validation (d27c5764…) must not become the runtime validation set. Validation/held-out binaries and exclusion lists are private and are not included in this repository.
The separate CPT Track run preserves real positive-step history. The first checked update 20 represented 20,971,520 new processed positions.
The separate CPT Track history preserves later processed-position samples. At the historical 10 November, 19:23 UTC snapshot, verified passage through update 10,000 corresponded to 10,485,760,000 processed positions at the unchanged batch size. Weights-only checkpoints at 5,000 and 10,000 and the then-latest full local recovery checkpoint at 10,000 had separate transfer checksums. This completed exposure is not the capacity of all six approved segments and is not a globally content-unique count.
The abandoned schedule was 95,000 updates × 1,048,576 positions = 99,614,720,000, not exactly 100B. It is not a newly approved CPT length or a count of distinct data already seen by the starting model.
The earlier random-initialization attempt used s01 v2. The references below are historical inputs, not approved fresh CPT shards. s01 contains ancestry data; s02/s03 need replacement Polish inputs and complete ancestral deduplication checks on reusable English material. No consumed language ratio or recovered checkpoint is established by their capacities.
| Historical reference, not a clean CPT shard | Input positions | Historical TRAIN SHA256 |
|---|---|---|
| s01 v2; withdrawn from new CPT inputs | 5,316,374,064 | 2ab70797649947e0c14b8067161ed8a2541585428ca52c5ac8c9a744669bbb06 |
| s02; rebuild/checks required for CPT | 5,302,913,370 | b4df1406b88f131ac14a431108e33a68450fbd783a748a4978dc5a99a5886bb3 |
| s03; rebuild/checks required for CPT | 5,284,760,292 | b1091fbc7bcb1a677c829dd07b288f18c353c6b923d99dd0a85338deb5f727be |
Their 15,904,047,726 positions are historical reference capacity, not ready fresh CPT data. Queue 6e06f0979f27c60e524852d89660bb2febef59a8b5f21f846e0618af1235715d belongs to the abandoned attempt. Its old Track run last received update 450 / 471,859,200 positions; these have not been relabelled as CPT from 1.08. The revised continuation has the distinct Track history linked above.
The reported ancestry inventory starts 1.00 from random weights, then continues through 1.06 on slayer-pl-8x3b@d50df04a, 1.07 on slayer-1.07-pretrain@6a337bbf, and 1.08 on R1 TRAIN b73a69ea (including Logika and English Technical). The conservative exclusion reference covers the union of these inputs, not a claim that every reference document was actually trained on. Known gaps include unavailable shared-history train digests for pack-04–07 and uncertain consumed tails/partial 1.08 coverage. Used-document exclusion, exact token deduplication and normalized-text fingerprints do not prove semantic uniqueness or a lower count of inherited exposure.
Used source categories and later candidates
Selected GoLLeM web, FineWeb-Edu and FinePDFs-Edu inputs occur in the approved queue above. Later allocation depends on the budget-limited endpoint and post-filter availability. Other catalog entries below remain candidates; published upstream sizes do not establish fresh retained v6 capacity or permission to use every document.
Linked public cards describe upstream material, not approval of every document:
| Source or candidate | Published unit / selection condition |
|---|---|
GoLLeM PL, 17f9d37a |
Selected web input in c01/c02/c06 after ancestral checks, including HPLT overlap with 1.06/1.07. Published 16.58B V32k / 14.62B web are not fresh retained v6 counts. CC0-1.0 labels HPLT packaging, not text rights; web TDM and curated per-record rights are reviewed separately |
FinePDFs-Edu, 9cfabe21, PL/EN |
Selected Polish input in c03–c06; retained amounts and provenance belong to each approved segment. ODC-By/Common Crawl terms are not blanket text redistribution rights |
PL Logika, e24c8b71 |
Previously used by 1.08: reference for exclusion and existing evaluations, not fresh CPT training input |
English Technical, 3b833d36; 1.07 replay, 6a337bbf |
Previously seen ancestry material: deduplication references, not eligible novel training capacity |
| Common Pile peS2o, PubMed, arXiv | Published text sizes: 182.6/147.1/19 UTF-8 GB; per-document open-license allowlist, not blanket permission |
| Common Pile StackExchange, Wikimedia | 89.7/57.4 UTF-8 GB; preserve attribution and applicable CC BY-SA terms |
| Common Pile DOAB, PressBooks, LibreTexts, OERCommons | 12/0.6/3.6/0.05 UTF-8 GB; per-document licensing; exclude LibreTexts GFDL material |
| FineMath, FineWeb-Edu | FineMath-4+ reports 9.6B Llama tokens; FineWeb-Edu samples use GPT-2; ODC-By plus Common Crawl terms |
Additional candidates include FineWeb2-HQ-Plus and other separately approved diverse sources; none supplies a verified retained amount or final PL share by its name alone. Replay from 1.07 is not a reserve of new data. peS2o's card says metadata.license, while the inventory identifies metadata.oa_license; use the actual pinned schema.
Admission, isolation and byte approval
Use a per-document allowlist: CC0/public domain, eligible CC BY/CC BY-SA, ODC-By, explicitly approved permissive terms or separately reviewed HPLT TDM-web. Unknown/missing, NC, ND, GPL, AGPL and GFDL are out. No The Stack or frontier-model-generated text. Preserve creators, licenses, URLs and modification notices. HPLT compilation terms, ODC database rights and TDM do not grant blanket redistribution rights.
For new material reserve 1% of documents per group before TRAIN/sharding, grouping source-book identities; this is not 1% of tokens. Preserve existing evaluation partitions. Apply exact/near-deduplication and 13-gram plus 8-gram-template decontamination with positive controls, including GSM8K/MATH/ARC/panels.
The old s01 training exception does not authorize replay in this revised CPT. Retrospective scans of the inherited training history against later benchmarks must report hit items and evaluation exclusions for the compared models; they cannot remove already learned exposure. New benchmarks remain descriptive until those checks are complete.
GoLLeM reports LLMzSzŁ variants for 1,662/18,820 items (8.83%); exclude affected items if evaluated. Its card conflicts about PoQuAD exclusion versus partition inclusion: do not use PoQuAD without decontamination. The ancestry's reported near-deduplication and EWoK omissions remain inherited concerns; filtering new data does not erase them from the starting weights.
Whole-document PII DROP changes coverage; document-drop fraction is not token-loss fraction. Independent residual checks, post-filter document/native-token counts, deduplication losses and final pins are required for each later tranche and the final corpus. The c01 evidence above does not approve those future inputs. Identifiers, extraction damage, bias and incorrect reasoning may survive.
Config/content approval and resulting-byte approval are separate. The historical c01–c06 digest above identifies only those approved ranges, not the later growing queue or globally unique history. The later aggregate inventory does not itself publish data bytes or approve new segments. Each addition requires approved source rights, filters, gold/evaluation exclusions, manifest and sampler identity. Source names alone do not establish actual composition. The tokenizer remains v6, SHA256 0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd. Rights, attribution and egress remain separate publication conditions.