| --- |
| license: cc-by-sa-4.0 |
| language: |
| - pl |
| library_name: pytorch |
| tags: |
| - polish |
| - language-model |
| - research |
| - tokenizer-comparison |
| pipeline_tag: text-generation |
| --- |
| |
| # TokenizerTest-32m-A (Polish research base model, tokenizer comparison — arm A) |
|
|
| A small Polish **base** language model trained as arm **A** of a controlled tokenizer experiment. It is a research |
| checkpoint for comparing tokenizers, not a model for use. It is a Fabryka AI project (formerly SlayerLab). |
|
|
| **Author:** Arkadiusz Słota (Fabryka AI), who designed and ran the experiment. Tokenizer of arm B: stubbornGuy. |
|
|
| ## Companion model (tokenizer A/B experiment) |
|
|
| This model is one half of a **controlled tokenizer comparison**. Two models were trained with the **same architecture, training recipe, seed and the same Polish text**. The **only** difference is the tokenizer: |
|
|
| | arm | tokenizer | repo | final step | |
| |---|---|---|---| |
| | **A** | Fabryka AI BPE, vocab 32 000 (`60e23148`) | SlayerLab/TokenizerTest-32m-A | 89 000 | |
| | **B** | tokenizer by stubbornGuy, vocab 32 000 (`77eeebed`) | SlayerLab/TokenizerTest-32m-B | 95 000 | |
|
|
| This model: **A**. |
|
|
| **Shared recipe:** ~38.9M parameters (tied input/output embeddings); "32m" in the repo name is the recipe name ("Glint 32M r6"), not the parameter count. Architecture: 15 layers, d_model 384, 6 heads, context 1024, Muon optimizer, batch 32, seed 1337, trainer `train_gpt_ref_r6.py` (`0223c083`). |
|
|
| Both arms saw the same ~12.4 GB of Polish text (public dataset [`SlayerLab/slayer-pl-8x3b`](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) @ `d50df04a`, files `pack-00`, `pack-01` and `shared/{core,core_sa,legal,legal_sa}`; validation = `pack-07` after 13-gram decontamination); B splits it more finely (on the training text 3.96 vs 4.25 bytes per token incl. EOS; on validation 4.14 vs 4.41), so it needed more steps for the same bytes. For the same reason B has lower bits per token (val 4.395 vs 4.668), which says nothing about model quality. |
|
|
| **How to compare:** loss per token is **not** comparable across tokenizers. We compare **bits per byte** on the same bytes (lower is better): |
|
|
| | | A | B | B − A | |
| |---|---|---|---| |
| | validation (7 232 820 B) | **1.0583** | 1.0622 | +0.0040 (+0.37 %) | |
| | held-out W (1 430 217 B) | **1.0781** | 1.0883 | +0.0101 (+0.94 %) | |
|
|
| From ~4 B bytes onward the gap is roughly constant (val +0.003…+0.004, W +0.008…+0.010) and does not close by the end of training. Curve every 10k steps (bits per byte; note: at the same **step** B has seen fewer bytes than A): |
|
|
| | step | A val | B val | A W | B W | |
| |---|---|---|---|---| |
| | 10 000 | 1.1910 | 1.2009 | 1.2327 | 1.2463 | |
| | 20 000 | 1.1448 | 1.1523 | 1.1799 | 1.1896 | |
| | 30 000 | 1.1200 | 1.1278 | 1.1499 | 1.1634 | |
| | 40 000 | 1.1032 | 1.1109 | 1.1304 | 1.1435 | |
| | 50 000 | 1.0888 | 1.0969 | 1.1141 | 1.1291 | |
| | 60 000 | 1.0773 | 1.0850 | 1.1000 | 1.1145 | |
| | 70 000 | 1.0675 | 1.0752 | 1.0885 | 1.1025 | |
| | 80 000 | 1.0610 | 1.0677 | 1.0818 | 1.0944 | |
| | 89 000 | 1.0583 | — | 1.0781 | — | |
| | 90 000 | — | 1.0633 | — | 1.0892 | |
| | 95 000 | — | 1.0622 | — | 1.0883 | |
|
|
| The same numbers, with checkpoint and data hashes, are in `bpb_curve.json`. |
|
|
| **Caveat:** one seed per arm, no noise estimate. The honest reading is *"at this scale the B tokenizer brings no gain in bits per byte"*, not *"B is worse"*. |
|
|
| ## Training data |
|
|
| Public dataset [`SlayerLab/slayer-pl-8x3b`](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b) at revision `d50df04a`; the same files for both arms: |
|
|
| | file | use | sha256 | |
| |---|---|---| |
| | `pack-00/web.parquet` | training (web) | `5cbb6dfa3a276a30db3c2034105ffffa75cc29b767456e7d75c305f9bd7ec562` | |
| | `pack-01/web.parquet` | training (web) | `802f4829f41ef83e23979d2460500907bcfd44278e300eccb5e39ac666a04c2c` | |
| | `shared/core.parquet` | training (core) | `ea0b4b901ace3780501b02a3972f014f77ce722315476e6f93681616958e0a99` | |
| | `shared/core_sa.parquet` | training (core, CC BY-SA) | `910b18eeebbc26efbbd76554d7484d016c6df692a3b6a18492b5c9ceb32549cb` | |
| | `shared/legal.parquet` | training (legal) | `6d4cc2f604d76452cd7e7ea52d1a04826f5e44f09bb035ba0853232fe373e74a` | |
| | `shared/legal_sa.parquet` | training (legal, CC BY-SA) | `a1609cd31e12e0f6fdfdb030edb538bc230fc5f5ab376d23136180338e9dbd1e` | |
| | `pack-07/web.parquet` | validation (13-gram decontaminated against training) | `3d93331061595949f2e2cab187cfe688a117785c048e94688808b399b3f53b67` | |
|
|
| ## Usage |
|
|
| The model is a custom PyTorch architecture (same code as `SlayerLab/GoLLeM-v6-250M`), not a `transformers` class. |
|
|
| | file | content | |
| |---|---| |
| | `model.safetensors` | final weights, step 89 000 (fp32, 38,866,958 parameters; output head stored as a copy of the tied embedding) | |
| | `config.json` | architecture | |
| | `tokenizer.json` | arm A tokenizer, 32 000 tokens (`tokenizers`) | |
| | `modeling_gollem_v6.py` | model code (inference) and a small `generate` helper | |
| | `step_10000/` … `step_80000/` | intermediate checkpoints every 10k steps (same three files each; weights only, no optimizer state) | |
| | `bpb_curve.json` | bits-per-byte curve (step, checkpoint sha256, val, W) | |
| | `SHA256SUMS` | sha256 of every file | |
|
|
| ```python |
| # pip install torch safetensors tokenizers huggingface_hub |
| from huggingface_hub import snapshot_download |
| import sys |
| |
| path = snapshot_download("SlayerLab/TokenizerTest-32m-A") # pin revision="<commit>" for reproducibility |
| sys.path.insert(0, path) |
| from modeling_gollem_v6 import load_gollem_v6, generate |
| |
| model, tok = load_gollem_v6(path) # final checkpoint |
| print(generate(model, tok, "Kraków jest miastem", max_new_tokens=40, temperature=0.8, top_k=50, seed=1)) |
| model_10k, _ = load_gollem_v6(f"{path}/step_10000") # an intermediate checkpoint |
| ``` |
|
|
| Loading any `model.safetensors` of this repository into `modeling_gollem_v6.py` reproduces the logits of the |
| corresponding training checkpoint exactly (checked with `torch.equal` for every step). |
|
|
| ## Limitations |
|
|
| - Small **base** model (~38.9M parameters): it continues Polish text; it does not answer questions or follow |
| instructions. Limited knowledge; may reproduce errors and biases of web text. Research use only. |
| - Single run, single seed per arm; see the caveat above. |
|
|
| ## License |
|
|
| **Weights: CC BY-SA 4.0** (the training text includes CC BY-SA parts of `SlayerLab/slayer-pl-8x3b`). Attribution: |
| TokenizerTest-32m-A, Arkadiusz Słota / Fabryka AI, link to this repository; derivative weights under the same licence. |
| Training data keep their upstream licences; see the dataset card of `SlayerLab/slayer-pl-8x3b`. |
|
|
| *TokenizerTest — Fabryka AI. **Author: Arkadiusz Słota.*** |
|
|
