---
language:
- pl
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- pretraining
- continued-pretraining
- training-in-progress
---

# Slayer1B 1.09

**Arkadiusz Słota, Anna Sieczka, Piotr Styła / Fabryka AI (SlayerLab)**

**Historical execution snapshot, 11 November 2026, 10:45 UTC:** continued pretraining from Slayer1B 1.08 was running on eight B200 spot GPUs after an H200 segment and checkpoint recoveries. Operators reported update 27,140 at that snapshot; this is dated progress telemetry, not a current availability claim or the endpoint of the run. The public tracking plan then showed 40,000 updates, while the trainer retained its 80,000-update scheduling argument. Neither value establishes completed exposure. The later planning decision below supersedes the requirement to complete terminal learning-rate decay. This is not a completed model or inference-ready weight release.

| Verified execution evidence | Scope |
|---|---|
| CPT start | Approximately 04:50 UTC, 10 November 2026, on eight B200 GPUs |
| Historical H200 recovery | Operators verified fresh numerical control and checkpoint restores during the earlier H200 segment |
| Compute at the historical 10:45 UTC snapshot | Eight B200 spot GPUs; training had continued beyond update 27,000; later availability requires operator confirmation |
| Checksum-verified weights-only copies on the workstation | Updates 5,000, 10,000, 15,000, 20,000 and 25,000; not public weight releases |
| Latest verified full-state workstation copy | Update 25,000; includes optimizer state, retained separately from weights-only copies; not a claim about the newest recovery checkpoint on the cloud disk |

The [CPT Track history](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a) provides changing public telemetry; consult it and the operators for later execution status and hardware availability. The planning update is a decision, not evidence that its future steps have executed. This card is not a completed 1.09 model, final quality verdict or inference-ready weight release.


## Objective and starting point

**Planning update — 11 November 2026, 12:07 UTC:** all of the existing **USD 425 campaign budget** is allocated to the main base on the currently selected data, with no funds earmarked for a future gold-data continuation. The objective is the largest useful affordable exposure, retaining constant learning rates after warmup and **no in-run terminal decay**. Existing cost guards and the operational reserve remain unchanged. The main-base run ends at the budget or allocation boundary with a full optimizer-state checkpoint, not by completing WSD decay. The historical 40,000-update tracking plan is not a funded endpoint; no later endpoint is asserted here. There is no promise of 80,000 updates, more than 100B globally unique tokens or a completion date. Model quality is evaluated separately; positions per dollar measure operating efficiency, not a quality score.

A separate future option is approximately **10B tokens of quality-selected “gold” data at context 4,096**, curated in parallel by the dataset team with licensed books selected with Arkadiusz Słota. It is deliberately isolated from the main base and evaluation material, requires its own manifest and admission/licensing gate, and would have its own run and budget decision. A finishing continuation with decay is an alternative for that separate run, not part of the current main-base objective. No such dataset has been assembled or released, no future training budget has been approved, and selection for quality is not evidence of a measured quality gain.

The starting weights are **[SlayerLab/Slayer1B-1.08](https://huggingface.co/SlayerLab/Slayer1B-1.08/tree/5f854a0cc7160594d54d1788d37a44e8ebabe16c)**, not random initialization or the abandoned attempt's checkpoint. Operators verified step-zero `INIT-FROM`, fresh optimizer state, approved c01 bytes, fresh numerical control and successful preflight before the observed updates. The selected public v5 launcher uses a separate output history for this continuation.

The unchanged architecture is specified as **1,073,912,851 parameters**, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization and value residuals. The reused v6 tokenizer has vocabulary 32,768 and pin `0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd`.

Architecture and tokenizer lineage includes [SlayerLab/Slayer1B-1.07](https://huggingface.co/SlayerLab/Slayer1B-1.07/blob/a090b54e658fb57dc5166c1fcc90cc6bf5c80106/README.md). The selected 1.08 base and the separate historical 1.08b ablation are different checkpoints; the continuation described here starts from 1.08.

The [training configuration](TRAINING_CONFIGURATION.md) distinguishes the still-configured 80,000-update scheduling argument from the budget-limited final length. Completed 1.08 and the abandoned first attempt remain separate historical evidence.

## Current CPT configuration and budget-limited endpoint

| Setting | Operator-selected recipe |
|---|---|
| Configured scheduling argument | `KROKI=80000` remains unchanged; a mechanical scheduling argument, not a funded endpoint; configured decay begins at update 64,000, beyond the current budget |
| New-source allocation | **20% PL / 80% EN target**; not a measured cumulative model-language ratio |
| Optimizer / schedule | Muon peak 0.02 plus AdamW peak 0.0006; 2,000-update warmup, then constant learning rates under the main-base plan; the configured final-20% WSD decay to 0.00006 is not to be entered in this run |
| Checkpoints | Full state every 400 updates after the approved cadence change; weights-only checkpoints at each 5,000-update boundary; workstation copies verified through update 25,000 |
| Code | Launcher v5 SHA256 `796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656`; exact command and converter pins in the configuration document |

Operators must review the affordable stopping point if prices or availability change, keeping the main-base run out of the configured decay phase. Do not shorten `KROKI` to the budget estimate: that would move decay earlier. This planning update changes no trainer or launcher code.

The reviewed inherited count is **18,863,257,682 source/file-disjoint positions**, not a globally content-deduplicated history. The measured c01 duplicate correction is conservatively **114,627,128 positions**, the sum of two exclusion-list counts rather than a measured union. If a 40,000-update endpoint is selected and completed, then 41,943,040,000 new positions would give a conditional source-position total of **60,691,670,554** after that correction. Historical content-level near-duplicates remain unresolved. This projection is not evidence of completed exposure or a globally unique corpus.

## Initial CPT execution evidence

The separate [CPT Track history](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a) preserves real training aggregates. Its 02:27 UTC registration timestamp is **not** the 04:50 UTC training start reported by operators. Fresh hardware control, initialization from 1.08, world size eight and preflight were independently checked for that start.

| First checked public sample | Value |
|---|---:|
| Update | 20 |
| Training loss | 2.486659 |
| Interval throughput | 578,293 positions/s |
| New processed positions | 20,971,520 |
| AdamW learning rate during warmup | 0.000006 |

The step-20 sample is retained as initial evidence; use Track for current progress. Interval throughput is not full-window or billed throughput. Training loss is measured on the current input segment: the change from c02 to c03 at update 10,000 changes the data distribution, so a lower training loss across that boundary does not establish a sudden quality gain. Later checkpoint-copy and recovery evidence is recorded separately above.


## Archived from-scratch attempt — not the CPT recipe

| Setting | Abandoned first-attempt configuration |
|---|---|
| Trainer / optimizer | Existing r24.2; Muon with AdamW; fresh optimizer |
| Batch / context | 256 × 4,096 = 1,048,576 positions per update |
| Layout | Eight GPU ranks; microbatch 8 × accumulation 4; activation checkpointing off |
| Budget | 95,000 updates; nineteen slots of 5,000 |
| WSD | 2,000 warmup updates; constant LR 0.0006, Muon 0.02; final 20% uses 1−sqrt decay to nonzero LR floor 0.00006 |
| Checkpoints | Every 500 updates and retained slot-end milestones |

The abandoned plan's **95,000 × 1,048,576 = 99,614,720,000 positions** was not exactly 100B and is **not the approved length of the revised continuation**. Its nineteen planned 5,000-update slots are historical configuration, not a measured unique-history count or evidence that those updates completed.

Slot-1 GPU execution was observed before that historical interruption. **Do not resume the from-scratch state as the requested CPT.** The revised continuation has its own output history and Track identity; historical points have not been relabelled as training from 1.08. Each queue extension still requires approved byte ranges, manifest digests and preserved sampler identity within its experiment.

## Archived first-attempt evidence — not CPT progress

The [public Track run](https://track.fabryka.ai/run/08cf77cf-e995-5ee1-b843-05cbb96e4f4b) is separate from 1.08. At update 30 it records training loss **10.3959**, interval throughput **576,593 positions/s** and **31,457,280 processed positions** (30 × 1,048,576). This is an early interval, not end-to-end billed throughput or a quality evaluation. The first slot targets 5,000 updates; its completion is not claimed.

The last received public sample before the pause is update **450**, training loss **4.3910**, with **471,859,200 processed positions**. Its ingestion timestamp is 22:36:50 UTC. It is not a live GPU reading, a final exposure count or proof of a checkpoint at update 500. The trainer's recoverable step must be checked from the saved state, not inferred from elapsed time or the tracker label.

Fresh numerical control and step-zero `INIT-FROM` preflight were verified for the CPT start described above. The earlier permission for the abandoned slot was not reused as that evidence. Any later physical-host restart still requires its own numerical control; successful start does not prove a checkpoint restore.

Working checkpoints are copied to a trusted workstation, not published as Hugging Face weights. Operators independently verified weights-only copies at updates 5,000, 10,000, 15,000, 20,000 and 25,000, and a separate full-state workstation copy at update 25,000. These transfer records are distinct from the newer recovery checkpoints retained on the running node and from the evaluated updates in the benchmark table. Weights-only copies are retained, while the full workstation recovery copy is replaced by the next verified transfer. This retention policy does not eliminate exposure to a cloud-disk failure between transfers.

## Data and exposure

The first CPT reference, **c01**, contains **5,347,737,900 input positions** and TRAIN SHA256 `3341080241aeabd1e03c77530a6d735cb9501ecd788fba60ea065a944cd09ee1`. One 5,000-update stage processes **5,242,880,000 positions**, less than that input capacity. The separately approved c01–c06 queue supplies six stages through update 30,000; only the earlier stages have been consumed. Available input capacity, processed positions and historical content uniqueness remain distinct. See [the data card](DATA_CARD.md) for the queue identities, source changes and c01 duplicate correction.

The PL/EN balance will follow the approved fresh-data inventory. The user explicitly allows a different ratio; source buckets still do not establish the language share of actually consumed windows. Missing ancestral data must be reported as a limitation, not silently counted as verified unique exposure.

New material requires licensing, PII filtering, contamination scans, grouped held-outs and deduplication against the base model's full data ancestry. A 1% document holdout is not necessarily 1% of tokens. Historical exposure in the starting weights cannot be undone by filtering new inputs; retrospective benchmark hits must be handled explicitly in evaluation denominators.

## Standard benchmark measurements — updates 5,000–20,000

We thank **BenchTeam / Fabryka AI** for benchmark implementations, protocol work and evaluation proposals. Its independent [full benchmark report](https://github.com/slayerlabs/benchmarks-fabryka-ai/blob/8b547007526971bd2fc35dfe56ff2c7d9b802568/research/results/full-run-2026-11/README.md) is distinct from the **SlayerLab measurements below**, published in the [CPT Track history](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a). Updates 5,000, 10,000, 15,000 and 20,000 are distinct intermediate working checkpoints of the ongoing 1.09 run, not its eventual budget-limited endpoint or inference-ready weight releases.

The same nine full splits were scored zero-shot in float32 for 1.07 update 22,327, 1.08 update 1,709 and 1.09 updates 5,000, 10,000, 15,000 and 20,000. Evaluator SHA, dataset revisions, file digests, task protocols and sample counts match across these rows. Scores are percentages with **pointwise 95% confidence intervals**; the named metric is accuracy (`acc`) or length-normalized accuracy (`acc_norm`). WinoGrande uses partial-sentence likelihood; MMLU-PL is micro-averaged. Polish HellaSwag and MMLU use the OpenGPTX task definitions.

| Task | Metric | n | 1.07 | 1.08 | 1.09, update 5,000 | 1.09, update 10,000 | 1.09, update 15,000 | 1.09, update 20,000 |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| ARC-Challenge | acc_norm | 1,172 | 27.30 [24.74, 29.86] | 30.03 [27.39, 32.68] | 29.35 [26.79, 32.00] | 30.46 [27.82, 33.11] | 32.51 [29.86, 35.24] | 33.11 [30.46, 35.84] |
| HellaSwag | acc_norm | 10,042 | 43.38 [42.43, 44.34] | 43.42 [42.43, 44.39] | 46.53 [45.55, 47.52] | 48.93 [47.97, 49.91] | 49.77 [48.80, 50.76] | 50.38 [49.39, 51.35] |
| PIQA | acc_norm | 1,838 | 66.97 [64.80, 69.15] | 66.21 [64.15, 68.34] | 68.28 [66.21, 70.40] | 68.99 [66.92, 71.11] | 68.99 [66.87, 71.06] | 70.78 [68.72, 72.85] |
| WinoGrande | acc | 1,267 | 53.59 [50.83, 56.35] | 53.67 [50.91, 56.43] | 54.93 [52.17, 57.70] | 54.93 [52.17, 57.62] | 56.75 [54.06, 59.43] | 55.01 [52.25, 57.77] |
| BoolQ | acc | 3,270 | 58.29 [56.61, 59.97] | 59.97 [58.26, 61.59] | 61.80 [60.18, 63.49] | 61.13 [59.45, 62.78] | 59.88 [58.20, 61.53] | 51.41 [49.69, 53.12] |
| OpenBookQA | acc_norm | 500 | 33.60 [29.40, 37.80] | 34.80 [30.60, 39.00] | 35.60 [31.40, 40.00] | 34.40 [30.20, 38.60] | 36.00 [31.80, 40.20] | 35.20 [31.00, 39.40] |
| Belebele, Polish | acc | 900 | 26.00 [23.11, 28.89] | 24.00 [21.22, 26.89] | 24.11 [21.33, 26.89] | 24.11 [21.33, 27.00] | 22.56 [19.89, 25.33] | 24.67 [21.89, 27.56] |
| HellaSwag-PL | acc_norm | 10,042 | 43.83 [42.84, 44.79] | 43.90 [42.90, 44.87] | 44.73 [43.76, 45.71] | 45.51 [44.51, 46.48] | 44.18 [43.21, 45.16] | 44.24 [43.26, 45.22] |
| MMLU-PL | acc | 14,042 | 24.01 [23.30, 24.71] | 23.04 [22.34, 23.74] | 24.10 [23.39, 24.82] | 23.32 [22.62, 24.03] | 23.50 [22.80, 24.21] | 23.86 [23.17, 24.58] |

Intervals use the percentile bootstrap with 10,000 replicates and seed 20261110. The updated approved aggregate does not supply the random-stream derivation or CI-generator SHA; the previous aggregate documented model-specific streams. **Items are treated as independent; subject, template and source clusters are not modeled.** These are intervals for individual results, not paired differences or training-seed variance; overlap is not a test of a difference.

The [machine-readable aggregate record](EVALUATION.json) supplies exact values, intervals, counts, dataset revisions and file digests, checkpoint and evaluator identities, and metric selection, without prompts, answers or per-item predictions. ARC-Challenge, OpenBookQA and MMLU-PL use test splits; HellaSwag, PIQA, WinoGrande, BoolQ and HellaSwag-PL use validation splits; Belebele uses the full Polish evaluation set.

The approved aggregate SHA-256 is `fc6f67a20bdc6ada687349d35de26f6811e27766dfe1aa197082000c03535796`. Its earlier 1.07, 1.08 and 1.09 update-5,000 scores, intervals and dataset pins match the preceding aggregate; the three later checkpoints extend that comparison without replacing the independent two-task protocol below.

**Descriptive results; contamination scan of the training lineage against these tasks is pending.** Belebele and MMLU-PL are near their four-choice 25% random baseline. No overall average, ranking or release threshold is inferred. Numbers from different evaluators must not be silently substituted: prompting, BOS handling, normalization and software versions can change the result.

There is no replay-only control for this continuation, and initialization, exposure and schedule differ between checkpoints. These measurements do not isolate a causal effect of the new datasets or quantify training-seed variance. Earlier internal diagnostics are separate historical records, not a criterion for naming or publishing this model.

## Additional GSM8K and SST-2 measurements — update 10,000

SlayerLab measured this two-task subset on 11 November 2026 using the [pinned BenchTeam code](https://github.com/slayerlabs/benchmarks-fabryka-ai/tree/8b547007526971bd2fc35dfe56ff2c7d9b802568), lm-eval 0.4.9.1, the `slayer` adapter, float32, batch 1 and context limit 4,096. These are full evaluated splits, not a full 50-task run or a copy of the community's independent report. **This separate two-task protocol measures 1.09 update 10,000 only; it does not supply GSM8K or SST-2 results for updates 15,000 or 20,000.** Update 10,000 is an internal working checkpoint, not a published or inference-ready 1.09 weight release.

| Task | Metric / prompting | n | 1.07 | 1.08 | 1.09, update 10,000 |
|---|---|---:|---:|---:|---:|
| GSM8K | strict exact match, 5-shot | 1,319 | 1.52 [0.91, 2.20] | 0.91 [0.45, 1.44] | 0.99 [0.45, 1.52] |
| SST-2 | accuracy, 0-shot | 872 | 51.03 [47.71, 54.36] | 50.46 [47.13, 53.67] | 58.26 [54.93, 61.35] |

Scores are percentages with reported pointwise 95% bootstrap intervals (10,000 draws, seed 20261028). These intervals describe individual results, not model differences or training-seed variance. This protocol and its intervals are recorded separately under `additional_measurements` in [EVALUATION.json](EVALUATION.json); the preceding nine-task protocol is unchanged.

**Descriptive results; training-lineage contamination scan pending.** GSM8K strict-match scores are near zero. SST-2 zero-shot accuracy is calibration-sensitive; these scores alone do not establish a general reasoning or sentiment-understanding gain. No paired-difference or internal calibration analysis is published here.

## Cost, licensing and limitations

The total campaign ceiling remains **USD 425**, including GPU allocations, preparation, interruptions, storage and evaluation tracked by the resource meter. At the operator's 11 November, 10:39 UTC snapshot, the meter reported **approximately USD 239 spent**, leaving approximately **USD 186**; this is counter attribution, not a settled invoice or a permanently current balance. Hardware selection is based on effective cost per processed position and verified availability. Under the 12:07 planning decision, budget reviews preserve the existing cost guards and operational reserve, with no earmark for future gold-data training and no requirement to fund terminal decay in the main base. No higher spending limit or guaranteed completion date is implied.

The selected 1.08 base declares Apache-2.0; source-text rights and future publication still require separate approval. Privacy filters are heuristic; generated text may be inaccurate, biased, offensive or personal. No reasoning quality, instruction-following ability or professional reliability has been demonstrated for the revised 1.09.