File size: 11,753 Bytes
0fcd392 b2c38fd 0fcd392 64ae577 3652dcd 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 5fe4b27 6b7fd5d 0fcd392 6d876f2 0fcd392 6d876f2 a4b6366 5f92b4e 6d876f2 b2c38fd 6d876f2 b2c38fd 6d876f2 b2c38fd 6b7fd5d a4b6366 6d876f2 9cdd7fd 6d876f2 b2c38fd 6d876f2 b2c38fd 6d876f2 6eed3c4 5fe4b27 6d876f2 5fe4b27 6d876f2 5fe4b27 6d876f2 5fe4b27 6eed3c4 6d876f2 b25aec4 6d876f2 a4b6366 6d876f2 0fcd392 6b7fd5d 6d876f2 6b7fd5d fc34c1f 0fcd392 6b7fd5d 3910478 6b7fd5d 3910478 6d876f2 3910478 6d876f2 9cdd7fd 6d876f2 9cdd7fd 6b7fd5d 64b3931 6d876f2 64b3931 6d876f2 0fcd392 6d876f2 0fcd392 6d876f2 9cdd7fd 6d876f2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 | ---
language:
- pl
- en
library_name: pytorch
license: apache-2.0
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- pretraining
- continued-pretraining
- training-in-progress
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceFW/finepdfs-edu
- SlayerLab/gollem-corpus-16b-pl
base_model: SlayerLab/Slayer1B-1.08
model-index:
- name: Slayer1B-1.09 update 60000
results:
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: allenai/ai2_arc
name: ARC-Challenge
revision: 210d026faf9955653af8916fad021475a3f00453
split: test
metrics:
- type: acc_norm
name: acc_norm (%)
value: 36.01
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: Rowan/hellaswag
name: HellaSwag
revision: 218ec52e09a7e7462a5400043bb9a69a41d06b76
split: validation
metrics:
- type: acc_norm
name: acc_norm (%)
value: 55.07
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: baber/piqa
name: PIQA
revision: 142f6d7367fd9877f0fb3b5734ea6a545f54cdd1
split: validation
metrics:
- type: acc_norm
name: acc_norm (%)
value: 72.09
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: allenai/winogrande
name: WinoGrande
revision: 01e74176c63542e6b0bcb004dcdea22d94fb67b5
split: validation
metrics:
- type: acc
name: acc (%)
value: 57.46
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: aps/super_glue
name: BoolQ
revision: 3de24cf8022e94f4ee4b9d55a6f539891524d646
split: validation
metrics:
- type: acc
name: acc (%)
value: 63.18
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: allenai/openbookqa
name: OpenBookQA
revision: 388097ea7776314e93a529163e0fea805b8a6454
split: test
metrics:
- type: acc_norm
name: acc_norm (%)
value: 35.0
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: facebook/belebele
name: Belebele, Polish
revision: 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2
split: pol_Latn evaluation set
metrics:
- type: acc
name: acc (%)
value: 24.78
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: Eurolingua/hellaswagx
name: HellaSwag-PL
revision: c38911a204759866dbcde35eeb584c36caf11a8b
split: validation
metrics:
- type: acc_norm
name: acc_norm (%)
value: 47.48
- task:
type: text-generation
name: Zero-shot multiple-choice evaluation
dataset:
type: Eurolingua/mmlux
name: MMLU-PL
revision: 7fb62cf05c1ce9ea550dbfd2ea6c145204344882
split: test; micro-average over subjects
metrics:
- type: acc
name: acc (%)
value: 24.46
---
# Slayer1B 1.09
A compact **Polish鈥揈nglish base language model** for text continuation, experimentation, and further training. Slayer1B 1.09 continues the Slayer1B 1.08 checkpoint with a documented **80% English / 20% Polish** new-input mix through update 45,000.
- **Use it to:** continue Polish or English text, study a small bilingual model, or start domain-specific continued pretraining and fine-tuning.
- **Why this checkpoint:** an openly released 1.07B-parameter model with published intermediate weights and a reproducible native PyTorch loader.
- **Know before you load:** this is a **base model**, not a chat assistant. It has not been instruction-tuned and has no chat template or demonstrated tool-use behavior.
## Train your own Slayer1B: CPT, SFT and RL notebooks
Start directly from this **Slayer1B-1.09 base model**, pinned to the released update-45,000 checkpoint. These tutorials include the native loader, tokenizer and training loops in this repository; no poetry adapter is required. Each training lesson starts independently from the frozen base and updates LoRA parameters.
| Notebook | Learn by doing |
| --- | --- |
| [00 路 Compare generation quality](tutorials/00_generation_comparison.ipynb) | Inspect the published poetry CPT case study and design a fair before/after evaluation |
| [01 路 CPT on raw text](tutorials/01_cpt.ipynb) | Predict every next token in domain text; save an adapter and compare continuations |
| [02 路 SFT on prompt/response pairs](tutorials/02_sft.ipynb) | Mask prompt labels, train response tokens and use a consistent inference template |
| [03 路 Reward-based RL](tutorials/03_rl.ipynb) | Run a small on-policy REINFORCE loop with rewards and frozen-parent KL; inspect reward hacking |
| [04 路 Average N checkpoints](tutorials/04_checkpoint_averaging.ipynb) | Build a full averaged checkpoint, check compatibility and compare it with every source |
[Setup and walkthrough](tutorials/README.md) 路 [Training-loop source](tutorials/slayer_training.py) 路 [Checks actually run](tutorials/VALIDATION.md). Download a notebook and open it in Jupyter, VS Code, or Google Colab. Training cells are opt-in. Toy SFT/RL examples teach the mechanics; they are not claims that this base model has undergone instruction tuning or RL. The verse-form reward does not measure literary quality.
## Model details
| Property | Specification |
|---|---|
| Checkpoint | Base model; continued pretraining from [Slayer1B 1.08](https://huggingface.co/SlayerLab/Slayer1B-1.08) |
| Parameters | 1.07B (1,073,912,851) |
| Languages | Polish and English |
| Context length | 4,096 tokens |
| Vocabulary | 32,768 tokens |
| Architecture | 20-layer decoder-only Transformer; SwiGLU, RMSNorm, RoPE, QK normalization, value residuals |
| Weights | Native PyTorch `.pt`, float32; approximately 4.3 GB per checkpoint |
| License | Apache-2.0 |
## Is this the right model for your task?
Choose Slayer1B 1.09 when you want a small Polish鈥揈nglish **starting checkpoint** for text completion, model research, or your own post-training. The public archive contains weights-only checkpoints every 5,000 updates from 5,000 through 60,000, so you can inspect how measured benchmark scores change during this continuation.
For direct question answering, following instructions, tool calling, or agent workflows, use an instruction-tuned model instead. Slayer1B 1.09 predicts continuations from a text prompt; it may repeat itself and does not reliably follow requests. Its 4K context also makes it a poor fit for long-document tasks without additional adaptation.
## Inference
The release uses a small native PyTorch loader rather than the Transformers `AutoModel` interface. The helper pins model-code, tokenizer, and archive revisions and checks the downloaded weights against the archive checksum. Install Python 3.11 or later and the dependencies:
```bash
pip install torch numpy tokenizers huggingface_hub
```
Load the published 45,000-update checkpoint and generate a greedy continuation:
```python
from huggingface_hub import hf_hub_download
import importlib.util
loader_path = hf_hub_download("SlayerLab/Slayer1B-1.09", "load_slayer.py")
spec = importlib.util.spec_from_file_location("slayer", loader_path)
slayer = importlib.util.module_from_spec(spec)
spec.loader.exec_module(slayer)
model, tokenizer = slayer.load(device="cpu", update=45000)
prompt = "Warszawa jest stolic膮 Polski i"
continuation = slayer.generate(model, tokenizer, prompt, max_new_tokens=48)
print(prompt + continuation)
```
`device` can be `"cpu"`, `"mps"` on Apple Silicon, or `"cuda"` on a supported NVIDIA system. The helper is intended as a minimal example, not a high-throughput server: it recomputes the prompt for each token and does not use a KV cache. See [inference notes and unedited examples](USAGE.md) and the [checkpoint archive](https://huggingface.co/SlayerLab/Slayer1B-1.09-checkpoints) for available weights and checksums.
## Performance
These are zero-shot, float32 multiple-choice evaluations. The 1.08 and 1.09 columns use the same documented Slayer evaluation setup; scores are percentages. WinoGrande uses accuracy; the other listed tasks use length-normalized accuracy.
| Benchmark | 1.08 | 1.09, update 60,000 |
|---|---:|---:|
| ARC-Challenge | 30.0 | 36.0 |
| HellaSwag | 43.4 | 55.1 |
| PIQA | 66.2 | 72.1 |
| WinoGrande | 53.7 | 57.5 |
| HellaSwag-PL | 43.9 | 47.5 |
On HellaSwag, the measured score rose from 43.4% to 55.1% across these two checkpoints. This is a descriptive checkpoint comparison, not a controlled experiment isolating the effect of the new training data. The [nine-task overview](plots/benchmark-overview.png) includes sample counts and pointwise 95% confidence intervals; [EVALUATION.json](EVALUATION.json) records exact dataset revisions, metrics, and checkpoint identities.
Polish knowledge measurements remain limited: at update 60,000, Belebele-PL is **24.78%** and MMLU-PL is **24.46%**, near the 25% uniform-choice reference. These results do not establish strong Polish question answering. Training-lineage contamination checks are incomplete, so benchmark scores are descriptive rather than proof that every evaluation example was unseen during training. GSM8K and SST-2 were measured under a separate protocol at update 10,000; details are in [EVALUATION.json](EVALUATION.json).

## Training
Version 1.09 starts from the released Slayer1B 1.08 checkpoint and continues pretraining. Through the evaluated update-45,000 checkpoint, new source allocation is documented as 80% English and 20% Polish. Selected inputs include GoLLeM web, FinePDFs-Edu, and FineWeb-Edu. The 4,096-token context and training recipe are described in [TRAINING_CONFIGURATION.md](TRAINING_CONFIGURATION.md); source selection, filtering, and licensing notes are in [DATA_CARD.md](DATA_CARD.md).
At update 60,000, the documented cumulative exposure estimate is 81.7B token occurrences, including inherited training and a known duplicate correction. This is not a count of globally unique tokens. See the [training log](TRAINING_LOG.md), [mix by stage](TRAINING_MIX.md), and the [training telemetry](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a). The public telemetry stopped updating after update 50,750; this card and the training log are the current record. The benchmark table above is specifically for update 60,000.
## Limitations
- Not instruction-tuned: do not expect reliable chat, instruction following, structured output, tool calling, or agent behavior.
- Short context: maximum configured context is 4,096 tokens.
- Greedy examples can repeat, especially in English; generated content can be inaccurate, biased, offensive, or personal.
- Polish multiple-choice scores on Belebele-PL and MMLU-PL are near the uniform-choice reference.
- Evaluation contamination checks across the full training lineage are incomplete. No benchmark here demonstrates professional reliability or a causal gain from a particular data source.
## Further reading
[Exact evaluation record](EVALUATION.json) 路 [Detailed training history](TRAINING_LOG.md) 路 [Training data card](DATA_CARD.md) 路 [Native inference notes](USAGE.md) 路 [Checkpoint archive](https://huggingface.co/SlayerLab/Slayer1B-1.09-checkpoints) 路 [Research and scaling notes](LINEAGE_SCALING.md)
**Authors:** Arkadiusz S艂ota, Anna Sieczka, Piotr Sty艂a, Aleksander Ogrodzki, Pawe艂 Puzio, Marcin Ku艣mierski, Kacper Wikie艂. **Funded by Fabryka AI.**
|