SabiBERTa-135M

A 135M-parameter encoder for 69 African languages, pretrained from scratch with masked language modelling. Sabi is Nigerian Pidgin for "to know".

Its point is coverage. AfroXLMR covers 17 African languages and AfriBERTa 11; SabiBERTa covers 69, including languages with no prior encoder at all — Dagbani, Mossi, Kamba, Nuer, Sango, Umbundu, Kimbundu, Tiv, Urhobo, Efik.

Architecture

ModernBERT (Warner et al., 2024), the current best-practice encoder recipe:

layers / dim / heads	12 / 768 / 12
context	4096
vocabulary	65,536 (+1 <mask>)
parameters	135.9M
positions	RoPE
FFN	GeGLU, pre-norm, no bias
attention	alternating local (128) / global, every 3rd layer global

The small vocabulary is why this is 135M where XLM-R base is 277M at the same depth and width: a 65,536 SabiYarn vocabulary trained on African text costs 50M embedding parameters against XLM-R's 192M for 250k multilingual tokens.

Training data

36.0B tokens drawn from a 100B-token deduplicated corpus (15.7% cross-document duplication removed), allocated by UniMax (Chung et al., ICLR 2023) with a 2-epoch repeat cap, plus a fixed 3B English anchor for cross-lingual transfer and code-switching.

Documents were selected by prior-based filtering (Seo et al., 2025): each document is scored by mean log unigram probability against its own language's prior, and by the standard deviation over equal chunks; those furthest from the language median are dropped, which removes gibberish and boilerplate without a trained classifier.

The corpus is 100% NFC-normalised (verified over ~418k documents).

Training

• Dynamic masking at 30% — a fresh mask every time a window is seen, per RoBERTa's ablation, at ModernBERT's rate rather than BERT's 15%.
• Cross-document attention masking — windows pack multiple documents (mean ~246 tokens), so without this a masked token would be predicted partly from an unrelated neighbouring article.
• Muon (Jordan et al. 2024, Moonlight scaling) on the hidden matrices, AdamW on embeddings, head and 1-D parameters.
• WSD schedule — constant LR after warmup, decayed over the final 10%, so the token budget stays a decision rather than a commitment.

Best checkpoint: step 153218, loss EMA 1.4867, 40.17B tokens seen.

Intended use and limits

Designed as a base encoder for fine-tuning: classification, retrieval, token labelling, and as a backbone for data-quality filtering across African languages.

It will be weak on the smallest languages. The repeat cap means languages with very little text contribute a tiny share — ijo has ~4k tokens available, kln ~250, gaa ~44k. Those are present so the model does not treat them as noise, not because it has learned them. Do not rely on it for quality judgements in those languages; there is not enough text in existence to support one.

Untoned Yoruba and Igbo are both well represented (18% of Yoruba documents and 52% of Igbo carry no tone marks), which is deliberate — that is what users type — but it means the model does not assume diacritics are present.

Citation

Uses UniMax (Chung et al., ICLR 2023), ModernBERT (Warner et al., 2024), prior-based filtering (Seo et al., 2025), Muon (Jordan et al., 2024), RoBERTa's dynamic masking (Liu et al., 2019).