xlmr-sepedi — Sepedi masked-language model

XLM-RoBERTa-base, further pre-trained on Sepedi (Sesotho sa Leboa) text by Sediba AI.

This is a masked-language model — it fills in blanks. It is not a chat or instruction model. For Sepedi text generation see Sedibaai/SedibaLM.

What is actually in this repo

Read from config.json in this repository:

Architecture	XLMRobertaForMaskedLM
Model type	xlm-roberta
Hidden size	768
Layers	12
Vocab size	250,002 (stock XLM-R multilingual)
Tokenizer	XLMRobertaTokenizer
Weights	1.11 GB (model.safetensors, fp32)

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="Sediba-AI/xlmr-sepedi")
fill("Batho ba <mask> Sepedi.")

Or for embeddings / as an NLU encoder:

from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("Sediba-AI/xlmr-sepedi")
model = AutoModel.from_pretrained("Sediba-AI/xlmr-sepedi")

Intended use

• Sepedi fill-mask and masked-LM research
• An encoder backbone for downstream Sepedi classification tasks
• Embeddings for Sepedi retrieval and similarity

Limitations — read before using

• No published evaluation. This model has no benchmark numbers in this card. Perplexity, fill-mask accuracy and downstream transfer were not recorded at publication time. Treat it as a research artefact and evaluate it on your own task before relying on it.
• Stock multilingual tokenizer. The 250,002-token XLM-R vocabulary is not Sepedi-optimised, so Sepedi fragments into more sub-word pieces than necessary.
• Not generative. A masked-LM cannot hold a conversation or follow instructions.
• Training corpus not documented in this card. Provenance is held in the Sediba vault, not restated here; no formal data-licence audit has been completed.
• No safety tuning.

Sediba Sepedi model family

Model	Type	Use	Status
Sedibaai/SedibaLM	Generative (Qwen2.5-1.5B + QLoRA)	Sepedi text generation, chat	Shipped — validated vs base
xlmr-sepedi (this repo)	Masked LM (XLM-R base)	Fill-mask, NLU encoder	Shipped — unevaluated
xlmr-sepedi-sentiment	Binary classifier	Sepedi sentiment	Shipped — unevaluated
sediba-XLM-R	Masked LM	Sepedi NLU anchor	Shipped

Licence

Apache 2.0 for these weights, inheriting from XLM-RoBERTa-base upstream.

Training data is not released under this licence — Sediba corpus material is governed separately (NOODL, in legal review). Open weights, restricted corpus.

About

Built by Sediba AI — sovereign AI for South African languages, starting with Sepedi (~4.7 million speakers) and extending to the other official languages of South Africa.