File size: 11,753 Bytes
0fcd392
 
 
 
 
b2c38fd
0fcd392
 
 
 
 
64ae577
3652dcd
5fe4b27
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
5fe4b27
 
 
 
 
 
 
 
 
 
 
6b7fd5d
0fcd392
 
 
 
6d876f2
0fcd392
6d876f2
 
 
a4b6366
5f92b4e
 
 
 
 
 
 
 
 
 
 
 
 
 
6d876f2
b2c38fd
6d876f2
 
 
 
 
 
 
 
 
 
b2c38fd
6d876f2
b2c38fd
6b7fd5d
a4b6366
6d876f2
9cdd7fd
6d876f2
b2c38fd
6d876f2
 
 
 
 
b2c38fd
6d876f2
6eed3c4
5fe4b27
 
 
6d876f2
 
 
5fe4b27
 
6d876f2
5fe4b27
6d876f2
 
 
5fe4b27
6eed3c4
6d876f2
b25aec4
6d876f2
a4b6366
6d876f2
0fcd392
6b7fd5d
6d876f2
6b7fd5d
 
 
 
fc34c1f
0fcd392
6b7fd5d
3910478
6b7fd5d
3910478
6d876f2
3910478
6d876f2
9cdd7fd
6d876f2
9cdd7fd
6b7fd5d
64b3931
6d876f2
64b3931
6d876f2
 
 
 
 
0fcd392
6d876f2
0fcd392
6d876f2
9cdd7fd
6d876f2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
---
language:
- pl
- en
library_name: pytorch
license: apache-2.0
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- pretraining
- continued-pretraining
- training-in-progress
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceFW/finepdfs-edu
- SlayerLab/gollem-corpus-16b-pl
base_model: SlayerLab/Slayer1B-1.08
model-index:
- name: Slayer1B-1.09 update 60000
  results:
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: allenai/ai2_arc
      name: ARC-Challenge
      revision: 210d026faf9955653af8916fad021475a3f00453
      split: test
    metrics:
    - type: acc_norm
      name: acc_norm (%)
      value: 36.01
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: Rowan/hellaswag
      name: HellaSwag
      revision: 218ec52e09a7e7462a5400043bb9a69a41d06b76
      split: validation
    metrics:
    - type: acc_norm
      name: acc_norm (%)
      value: 55.07
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: baber/piqa
      name: PIQA
      revision: 142f6d7367fd9877f0fb3b5734ea6a545f54cdd1
      split: validation
    metrics:
    - type: acc_norm
      name: acc_norm (%)
      value: 72.09
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: allenai/winogrande
      name: WinoGrande
      revision: 01e74176c63542e6b0bcb004dcdea22d94fb67b5
      split: validation
    metrics:
    - type: acc
      name: acc (%)
      value: 57.46
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: aps/super_glue
      name: BoolQ
      revision: 3de24cf8022e94f4ee4b9d55a6f539891524d646
      split: validation
    metrics:
    - type: acc
      name: acc (%)
      value: 63.18
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: allenai/openbookqa
      name: OpenBookQA
      revision: 388097ea7776314e93a529163e0fea805b8a6454
      split: test
    metrics:
    - type: acc_norm
      name: acc_norm (%)
      value: 35.0
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: facebook/belebele
      name: Belebele, Polish
      revision: 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2
      split: pol_Latn evaluation set
    metrics:
    - type: acc
      name: acc (%)
      value: 24.78
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: Eurolingua/hellaswagx
      name: HellaSwag-PL
      revision: c38911a204759866dbcde35eeb584c36caf11a8b
      split: validation
    metrics:
    - type: acc_norm
      name: acc_norm (%)
      value: 47.48
  - task:
      type: text-generation
      name: Zero-shot multiple-choice evaluation
    dataset:
      type: Eurolingua/mmlux
      name: MMLU-PL
      revision: 7fb62cf05c1ce9ea550dbfd2ea6c145204344882
      split: test; micro-average over subjects
    metrics:
    - type: acc
      name: acc (%)
      value: 24.46
---

# Slayer1B 1.09

A compact **Polish鈥揈nglish base language model** for text continuation, experimentation, and further training. Slayer1B 1.09 continues the Slayer1B 1.08 checkpoint with a documented **80% English / 20% Polish** new-input mix through update 45,000.

- **Use it to:** continue Polish or English text, study a small bilingual model, or start domain-specific continued pretraining and fine-tuning.
- **Why this checkpoint:** an openly released 1.07B-parameter model with published intermediate weights and a reproducible native PyTorch loader.
- **Know before you load:** this is a **base model**, not a chat assistant. It has not been instruction-tuned and has no chat template or demonstrated tool-use behavior.

## Train your own Slayer1B: CPT, SFT and RL notebooks

Start directly from this **Slayer1B-1.09 base model**, pinned to the released update-45,000 checkpoint. These tutorials include the native loader, tokenizer and training loops in this repository; no poetry adapter is required. Each training lesson starts independently from the frozen base and updates LoRA parameters.

| Notebook | Learn by doing |
| --- | --- |
| [00 路 Compare generation quality](tutorials/00_generation_comparison.ipynb) | Inspect the published poetry CPT case study and design a fair before/after evaluation |
| [01 路 CPT on raw text](tutorials/01_cpt.ipynb) | Predict every next token in domain text; save an adapter and compare continuations |
| [02 路 SFT on prompt/response pairs](tutorials/02_sft.ipynb) | Mask prompt labels, train response tokens and use a consistent inference template |
| [03 路 Reward-based RL](tutorials/03_rl.ipynb) | Run a small on-policy REINFORCE loop with rewards and frozen-parent KL; inspect reward hacking |
| [04 路 Average N checkpoints](tutorials/04_checkpoint_averaging.ipynb) | Build a full averaged checkpoint, check compatibility and compare it with every source |

[Setup and walkthrough](tutorials/README.md) 路 [Training-loop source](tutorials/slayer_training.py) 路 [Checks actually run](tutorials/VALIDATION.md). Download a notebook and open it in Jupyter, VS Code, or Google Colab. Training cells are opt-in. Toy SFT/RL examples teach the mechanics; they are not claims that this base model has undergone instruction tuning or RL. The verse-form reward does not measure literary quality.

## Model details

| Property | Specification |
|---|---|
| Checkpoint | Base model; continued pretraining from [Slayer1B 1.08](https://huggingface.co/SlayerLab/Slayer1B-1.08) |
| Parameters | 1.07B (1,073,912,851) |
| Languages | Polish and English |
| Context length | 4,096 tokens |
| Vocabulary | 32,768 tokens |
| Architecture | 20-layer decoder-only Transformer; SwiGLU, RMSNorm, RoPE, QK normalization, value residuals |
| Weights | Native PyTorch `.pt`, float32; approximately 4.3 GB per checkpoint |
| License | Apache-2.0 |

## Is this the right model for your task?

Choose Slayer1B 1.09 when you want a small Polish鈥揈nglish **starting checkpoint** for text completion, model research, or your own post-training. The public archive contains weights-only checkpoints every 5,000 updates from 5,000 through 60,000, so you can inspect how measured benchmark scores change during this continuation.

For direct question answering, following instructions, tool calling, or agent workflows, use an instruction-tuned model instead. Slayer1B 1.09 predicts continuations from a text prompt; it may repeat itself and does not reliably follow requests. Its 4K context also makes it a poor fit for long-document tasks without additional adaptation.

## Inference

The release uses a small native PyTorch loader rather than the Transformers `AutoModel` interface. The helper pins model-code, tokenizer, and archive revisions and checks the downloaded weights against the archive checksum. Install Python 3.11 or later and the dependencies:

```bash
pip install torch numpy tokenizers huggingface_hub
```

Load the published 45,000-update checkpoint and generate a greedy continuation:

```python
from huggingface_hub import hf_hub_download
import importlib.util

loader_path = hf_hub_download("SlayerLab/Slayer1B-1.09", "load_slayer.py")
spec = importlib.util.spec_from_file_location("slayer", loader_path)
slayer = importlib.util.module_from_spec(spec)
spec.loader.exec_module(slayer)

model, tokenizer = slayer.load(device="cpu", update=45000)
prompt = "Warszawa jest stolic膮 Polski i"
continuation = slayer.generate(model, tokenizer, prompt, max_new_tokens=48)
print(prompt + continuation)
```

`device` can be `"cpu"`, `"mps"` on Apple Silicon, or `"cuda"` on a supported NVIDIA system. The helper is intended as a minimal example, not a high-throughput server: it recomputes the prompt for each token and does not use a KV cache. See [inference notes and unedited examples](USAGE.md) and the [checkpoint archive](https://huggingface.co/SlayerLab/Slayer1B-1.09-checkpoints) for available weights and checksums.

## Performance

These are zero-shot, float32 multiple-choice evaluations. The 1.08 and 1.09 columns use the same documented Slayer evaluation setup; scores are percentages. WinoGrande uses accuracy; the other listed tasks use length-normalized accuracy.

| Benchmark | 1.08 | 1.09, update 60,000 |
|---|---:|---:|
| ARC-Challenge | 30.0 | 36.0 |
| HellaSwag | 43.4 | 55.1 |
| PIQA | 66.2 | 72.1 |
| WinoGrande | 53.7 | 57.5 |
| HellaSwag-PL | 43.9 | 47.5 |

On HellaSwag, the measured score rose from 43.4% to 55.1% across these two checkpoints. This is a descriptive checkpoint comparison, not a controlled experiment isolating the effect of the new training data. The [nine-task overview](plots/benchmark-overview.png) includes sample counts and pointwise 95% confidence intervals; [EVALUATION.json](EVALUATION.json) records exact dataset revisions, metrics, and checkpoint identities.

Polish knowledge measurements remain limited: at update 60,000, Belebele-PL is **24.78%** and MMLU-PL is **24.46%**, near the 25% uniform-choice reference. These results do not establish strong Polish question answering. Training-lineage contamination checks are incomplete, so benchmark scores are descriptive rather than proof that every evaluation example was unseen during training. GSM8K and SST-2 were measured under a separate protocol at update 10,000; details are in [EVALUATION.json](EVALUATION.json).

![Benchmark scores across Slayer1B versions and 1.09 checkpoints](plots/benchmark-overview.png)

## Training

Version 1.09 starts from the released Slayer1B 1.08 checkpoint and continues pretraining. Through the evaluated update-45,000 checkpoint, new source allocation is documented as 80% English and 20% Polish. Selected inputs include GoLLeM web, FinePDFs-Edu, and FineWeb-Edu. The 4,096-token context and training recipe are described in [TRAINING_CONFIGURATION.md](TRAINING_CONFIGURATION.md); source selection, filtering, and licensing notes are in [DATA_CARD.md](DATA_CARD.md).

At update 60,000, the documented cumulative exposure estimate is 81.7B token occurrences, including inherited training and a known duplicate correction. This is not a count of globally unique tokens. See the [training log](TRAINING_LOG.md), [mix by stage](TRAINING_MIX.md), and the [training telemetry](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a). The public telemetry stopped updating after update 50,750; this card and the training log are the current record. The benchmark table above is specifically for update 60,000.

## Limitations

- Not instruction-tuned: do not expect reliable chat, instruction following, structured output, tool calling, or agent behavior.
- Short context: maximum configured context is 4,096 tokens.
- Greedy examples can repeat, especially in English; generated content can be inaccurate, biased, offensive, or personal.
- Polish multiple-choice scores on Belebele-PL and MMLU-PL are near the uniform-choice reference.
- Evaluation contamination checks across the full training lineage are incomplete. No benchmark here demonstrates professional reliability or a causal gain from a particular data source.

## Further reading

[Exact evaluation record](EVALUATION.json) 路 [Detailed training history](TRAINING_LOG.md) 路 [Training data card](DATA_CARD.md) 路 [Native inference notes](USAGE.md) 路 [Checkpoint archive](https://huggingface.co/SlayerLab/Slayer1B-1.09-checkpoints) 路 [Research and scaling notes](LINEAGE_SCALING.md)

**Authors:** Arkadiusz S艂ota, Anna Sieczka, Piotr Sty艂a, Aleksander Ogrodzki, Pawe艂 Puzio, Marcin Ku艣mierski, Kacper Wikie艂. **Funded by Fabryka AI.**