Serblo-125M

This is the first open Bosnian-Croatian-Serbian model where the entire stack is from scratch and public: corpus pipeline, tokenizer, architecture, training loop, SFT and preference datasets. All 70.8B training tokens are BCS. Open Serbian models do exist (jerteh/gpt2-orao, YugoGPT) but the claim here is the fully open stack.

I built this over one summer. The final training run cost ~$230 of Spot H100 time in cloud credits. Out of pocket the whole project was about ~$110.

The model is here to demonstrate the released data pipeline and to be a native-BCS base for finetunes. It writes fluent, native Serbian (Latin/gajica) across registers: news, forums, recipes, casual chat, literary prose. But it's (obviously) not a knowledge base or a competitive assistant.

Specs

Parameters	125,064,960
Architecture	Llama-style decoder, from scratch in PyTorch: RoPE, GQA (12 query / 4 KV heads), RMSNorm, SwiGLU, tied embeddings
Layers / d_model / ctx	14 / 768 / 2048
Tokenizer	32k byte-level BPE, trained on the corpus (1.63 tokens/word on BCS, 43% leaner than GPT-2's, 9% leaner than XLM-R's on Serbian)
Training	135,000 steps × 524,288 tokens = 70.8B tokens seen ≈ 3 epochs over 23.3B unique
Recipe	AdamW (0.9/0.95, wd 0.1), peak LR 4e-4, cosine → 4e-5, warmup 2k, bf16, torch.compile
Final validation	loss 2.7478, perplexity 15.61, ≈0.96 bits/byte (own tokenizer, not comparable across tokenizers/languages)
Hardware	1× Spot H100 (Vertex AI), ~57 GPU-hours at ~28% MFU (small models can't feed big tensor cores)

Corpus (~23.3B unique tokens)

The corpus is a 96% FineWeb-2 backbone plus deliberate register injection. The pipeline (released alongside) does LID → Cyrillic→Latin transliteration (diacritics preserved; č ć š ž đ are load-bearing) → decluttering → dedup → tokenization.

slice	tokens	notes
FineWeb-2 BCS (bos/hrv/srp ×2 scripts)	22.5B	28.2M documents, cleaned + transliterated
srWaC	0.64B
Synthetic conversational pillar	0.14B (×3 upsampled)	multi-model generated casual register the web lacks; diversity-gated (self-BLEU vs real)
OpenSubtitles (repaired)	0.02B (×3)	double-mojibake repair (CP1250-as-CP1251 and CP1250-as-Latin1)

Use

import torch, json
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import GPT, GPTConfig   # model.py ships in this repo

cfg = GPTConfig(**json.load(open("config.json"))["config"])
sd = load_file("model.safetensors")
sd["lm_head.weight"] = sd["tok_emb.weight"]     # embeddings are tied, stored once
model = GPT(cfg); model.load_state_dict(sd); model.eval()
tok = Tokenizer.from_file("tokenizer.json")
# sample with temperature 0.7, top-k 200, repetition penalty 1.3 (see sample.py)

This is a base model: it continues text. For chat, use Serblo-125M-Instruct or fine-tune with the released SFT stack.

Limitations

• It makes up specifics. Names, numbers, institutions etc. are fluent inventions ("Beograd, grad sa preko 400 miliona stanovnika").
• It can't faithfully rewrite long passages. Giving it source text to transform; it riffs on the theme instead. Found this by accident while building the eval set and documented it there.
• One rare prompt context (em-dash dialogue openers) makes ~1 in 5 generations start with a stray wrong-script byte before recovering. A repetition penalty of 1.3 and the instruct-tune both do suppress it, but no promises.
• English does suprisingly show up, but often times as authentic BCS code-switching ("Ok so anyway, šta ti se dešava..."), which is corpus-faithful. Standalone English prompts produce word salad.
• Trained on web text with the usual biases, no alignment stage beyond data curation.

Training story

Pretraining ran on Spot H100, checkpointing to GCS every 1,000 steps with an auto-resubmitting supervisor. The run survived three incidents: a plateau false-alarm (actually LR starvation; cutting the cosine schedule 200k→135k gave an annealed finish and val dropped 2.85→2.75), a multi-hour Cloud Logging blackout (checkpoint timestamps were used to verify it was alive) and the cloud project getting billing-suspended hours after the final checkpoint saved. The model was rescued at step 135,000. The details are in the pipeline repo.

Lineage & credits

The synthetic conversational pillar was written by:

model	docs	share of pillar
Gemini 2.5 Flash-Lite	499,804	98.93%
Llama 3.3 70B (Groq + SambaNova)	3,504	0.69%
MiMo (mimo-auto + V2.5)	1,041	0.21%
DeepSeek (V4 Flash + V3.2 + V4 Pro)	593	0.12%
GLM 5.2	175	0.03%
Claude Opus (4.6 + 4.8)	86	0.02%
GPT-5.5, Kimi K2.6	7	trace

Shares are by document count (505,210 total). Gemini carried the volume, the rest is deliberate diversity seasoning: pooling lineages cut phrase repetition 41% below single-model output in balanced tests (self-BLEU 0.331→0.195; the measurement tools ship in the pipeline).

Built by Stefan Selakov (@sterlixlol)