gpt-alpha-bg-91m

A 91.26M-parameter decoder-only GPT trained from scratch on Bulgarian text, with no pretrained initialization, in a nanoGPT-style codebase. Its measured strength is judging Bulgarian fluency by perplexity: on error/correction pairs it is statistically on par with Gemini, offline, on a CPU.

What it is for

A fluency gate or reranker over text produced somewhere else: ranking OCR, ASR or MT output, flagging garbled or bot-written Bulgarian. It ranks candidates. It does not write them.

Two boundaries, both measured rather than assumed. It is a base LM with no instruction tuning, so it neither corrects text nor answers questions. And it is not a script or gibberish detector: it was trained on web data, so romanized Bulgarian can outscore natural Cyrillic. It compares Bulgarian candidates against each other, nothing more.

Usage

The checkpoint is a plain PyTorch state dict carrying its own config, so loading it needs the class definitions rather than a guess. Those are one file: bg-eval, a torch-only loader verified bit-identical to the training implementation (max |Δlogit| 0.000e+00 on all seven published checkpoints), shipped with the evaluations that produced the numbers below.

git clone https://github.com/iksnerd/bg-eval && cd bg-eval && uv sync
uv run bpc.py   --checkpoints ckpt_gpt2bg.pt --tokenizer tokenizer_gpt2.json   # 1.3041
uv run judge.py --checkpoint  ckpt_gpt2bg.pt --tokenizer tokenizer_gpt2.json   # 0.9305

Every figure has its protocol and its provenance in expected/*.json, so the numbers here are something you check rather than take on trust. model.py's load() hands back a plain torch.nn.Module if you want to score or sample against it yourself.

The weights are also here as safetensors, which needs neither a loader nor pickle:

from safetensors.torch import load_file

weights = load_file("model.safetensors")
weights["token_embedding_table.weight"].shape   # torch.Size([8192, 768])

config.json beside it carries the resolved architecture, every toggle spelled out rather than left to a defaults table. The LM head is tied to the token embedding and is therefore stored once, under the embedding's name. Not transformers-compatible either way.

Results

Fluency judging, against two independent LLM judges

Sentence pairs from thebogko/bulgarian-grammar-mistakes and thebogko/bulgarian-spelling-mistakes, each an erroneous sentence and its gold correction across 11 error types (article and pronoun misuse, verb-suffix and noun/adjective agreement, vowel change, double consonant, and others). A fluency judge should score the correction above the error. Measured as paired preference accuracy (chance 50%) and AUC, under two selections of the same pool:

judge	26 curated pairs	200 random pairs
this model (91.26M), perplexity	92.3% (81–100), AUC 0.77	100.0% (98.2–100*), AUC 0.85
Gemini (gemini-3.5-flash), independent	88.5% (77–100), AUC 0.85	98.0% (96.3–99.5), AUC 0.94
qwen2.5:3b, independent	65.4% (56–75), AUC 0.63	69.2% (65.5–73.0), AUC 0.65

* 0 losses out of 200 makes the percentile bootstrap degenerate at [100, 100], so the Clopper-Pearson bound is reported instead. Other intervals are 95% bootstrap.

Cite the curated column. Those 26 cases are hand-selected for known error types, which makes them the harder, more informative slice and the one where "ties Gemini" is a claim about judge quality. The model prefers the correct sentence in 24 of 26 (p < 0.0001); the intervals overlap, the model leads on paired accuracy and Gemini on pooled AUC. The 200-pair column is a different protocol over an easier slice (random draw, ≤ 120 characters, error_preference.py) where both judges saturate, and it is reported so that the curated set is not the only sample on the card.

qwen2.5:3b is far weaker on either selection, and rates both sentences of a pair identically in 16 of the 26 curated cases. It also shows the expected self-grading bias: asked to rate its own corrections rather than these gold pairs it averages 74/100 against ~49–52 from the independent judges, and on one case where its correction came out as non-Bulgarian gibberish it rated itself 85/100 while this model scored it 0 and Gemini 1.

So a free, offline 91M CPU model can stand in for a frontier API call as a Bulgarian fluency gate.

Against BgGPT, head to head

Two generations of INSAIT's BgGPT, Gemma-2-2.6B-IT (28× this model) and Gemma-3-4B-IT (47×), with identical code on all three sides: same held-out text, same 256-token windows, same scoring function, same pairs, one device.

	this model (91.26M)	BgGPT-Gemma-2-2.6B-IT	BgGPT-Gemma-3-4B-IT
parameters	91.26M	2.61B	4.30B
bits per character, ~49.5k characters	1.3041	1.1600	1.1683
fluency preference, 2000 pairs	0.9305	0.9175	0.846

On compression it loses, by 0.144 bpc to the better of the two. Models with far more Bulgarian pretraining are better at modelling Bulgarian, which is what should happen.

On fluency judging it wins, and the paired test is what says so rather than the two accuracy figures. Exact McNemar over 2,000 identical pairs:

opponent	only this model	only theirs	discordant	p
Gemma-2-2.6B	79	53	132	0.0292
Gemma-3-4B	225	56	281	≈0

Against Gemma-2-2.6B that clears the conventional 0.05 threshold and only just; two comparisons were run, and under a Bonferroni correction for two (0.025) it does not clear. So: ahead of Gemma-2-2.6B, significant uncorrected and marginal under correction; clearly ahead of Gemma-3-4B.

What counts as a pair. Some rows in the source datasets have the erroneous and correct sentences character-for-character identical. There is nothing to discriminate in those, every model scores them the same, and counting them would put a silent ceiling on the benchmark. They are skipped and reading continues, so 2000 means 2000 usable pairs. All three models report zero ties, which is how you can check none was scored.

One caveat belongs with this, and it cuts both ways. Both opponents are instruct models and both metrics read raw-text likelihood, which instruction tuning is known to decalibrate. That is visible in INSAIT's own line: Gemma-3-4B is worse than Gemma-2-2.6B on both measures despite being newer and larger. This model is a base model on a base-model task, and INSAIT publishes no Gemma-3 base, so a cleaner comparison does not exist. The claim to take away is narrow and still worth making: a 91M model trained from scratch beats Bulgarian-specialised instruct models at fluency discrimination, while being clearly worse at general modelling. Neither opponent is INSAIT's largest either. They publish up to Gemma-3-27B, which was not tested, and 256-token windows are this model's native context, which mildly disadvantages a long-context opponent.

Language modelling

Validation loss is 2.6689, comparable only to models sharing this tokenizer, so cross-model figures here are bits per character on one shared external held-out text (~200k characters, a larger sample than the ~49.5k slice used against BgGPT above):

model	parameters	corpus	bpc
this model	91.26M	mixed, ~1.83B tokens	1.2813
mixed-corpus sibling	29.37M	mixed, 672M tokens	1.9339
earlier books-era model	29.36M	fiction	2.1304

It also shifts register with the prompt: given a factual sentence the books-only model continues in fiction (invented pseudo-history, fantasy vocabulary), while this model continues in a factual, encyclopedic register. Facts stay unreliable at this scale; only the voice matches.

Zero-shot, against an untrained control

On EXAMS (Bulgarian school-exam multiple choice, 1472 items, 25.0% chance) it scores 27.11%, clearing chance by 2.4σ and clearing a randomly initialized network of its own shape by 5.0 points (z = 3.1). The smaller models in the family clear neither.

The control is the point. On Belebele (bul_Cyrl, 900 items) the untrained network matches or beats every trained model, which means that benchmark measures nothing about models at this scale and any above-chance score on it is an artifact of the scoring procedure. A random network is not a blank slate: it has systematic preferences over answer length and token frequency, and those alone move the score off 25%.

Limits

• A stylistic language model, not an assistant. Fluent, locally coherent Bulgarian prose in the register it is prompted toward, with no reliable factual knowledge, limited long-range coherence, and no instruction following. Expected at this scale. For factual QA or instruction following, fine-tune a pretrained multilingual model instead.
• 256-token context, which is also the window every number above was measured in.
• Not a script detector. Romanized Bulgarian can outscore natural Cyrillic, as above.
• n = 26 on the curated judge set. One case is 3.8 points, so only large deltas mean anything there; the 2000-pair BgGPT comparison is the one with a paired test behind it.

How it was built

• 768 embedding dim, 12 heads, 12 layers (GPT-2-small depth), block size 256, vocab 8192
• RoPE, RMSNorm, SwiGLU, weight tying (Llama-style stack)
• Highway-gated residuals, validated against the plain residual stack in a controlled multi-seed ablation and reconfirmed at this scale with a 5000-step A/B test
• Byte-level BPE tokenizer, vocab 8192
• bf16 mixed precision, warmup + cosine LR decay

Trained on 1.83B tokens, Chinchilla-optimal (20 tokens/param) for this size: Bulgarian book prose (chitanka literary corpus), news, forum and Wikipedia text, plus a slice of hasankursun/bulgarian-corpus-33b (FineWeb-2 web crawl + FineWiki BG) added to reach the compute-optimal token count. A Modal A10G GPU for 13 hours ($15).

Files

file	what it is
ckpt_gpt2bg.pt	365 MB PyTorch state dict carrying its own config; optimizer state stripped, loads under weights_only=True
model.safetensors	the same weights, no pickle and no class definition needed to read them
config.json	the resolved architecture: sizes plus every toggle (rmsnorm, rope, swiglu, softmax attention, highway-scalar residuals, weight tying)
tokenizer_gpt2.json	byte-level BPE, vocab 8192 (HuggingFace tokenizers format)

Part of a family

Six models from one project on small models doing narrow tasks, these among them:

• gpt-alpha-bg-91m - you are here
• gpt-alpha-bg-restorer - shlyokavitsa to Cyrillic, character by character, running in your browser

The other four appear under huggingface.co/glassbox as each ships.

Findings, the interpretability lab and the full model roster: restorer-interp-lab.iksnerd.workers.dev.

License

MIT.