LHRS-UM-FERI/SIGMA-BERT-base

SIGMA-BERT

Table of Contents

1. Model Summary
2. Model Details
3. Usage
4. Training
5. Training Data
6. Tokenizer
7. Limitations
8. Citation

Model Summary

SIGMA-BERT is a Slovenian encoder language model based on the ModernBERT architecture. It was pretrained from scratch using the masked language modeling (MLM) objective on a diverse collection of Slovenian text, including web, news, parliamentary proceedings, scientific publications, and user-generated content. The model supports sequence lengths of up to 8192 tokens and is intended for downstream language understanding tasks such as classification, retrieval, semantic similarity, and token-level prediction.

As a pretrained encoder, SIGMA-BERT is intended to be fine-tuned for downstream Slovenian language understanding tasks such as:

• Text classification (e.g. topic, sentiment, genre)
• Named entity recognition and other token classification tasks
• Extractive question answering
• Natural language inference / semantic textual similarity
• Dense retrieval and semantic search, including over long documents
• As a backbone for rerankers in retrieval-augmented generation (RAG) pipelines

It is available in the following variants:

• SIGMA-BERT-short - 22 layers, 149 million parameters, 1,024 token context
• SIGMA-BERT-base - 22 layers, 149 million parameters, 8,192 token context
• SIGMA-BERT-large - 28 layers, 395 million parameters, 8,192 token context

Model Details

SigmaBERT-base is built on the ModernBERT architecture and pretrained on Slovenian text with a Masked Language Modeling (MLM) objective.

Property	Value
Language	Slovenian
Training objective	Masked Language Modeling (MLM)
Parameters	149M
Layers	22
Hidden size	768
Intermediate size (GeGLU)	1,152
Attention heads	12
Vocab size	50,000
Max sequence length	8,192 (extended from 1,024)
Attention pattern	Global attention every 3rd layer; local sliding-window attention (128 tokens) elsewhere
Activation	GeGLU
Normalization / linear layers	Pre-LayerNorm, no bias terms

Training

SIGMA-BERT was pretrained in two stages.

Stage 1

The initial pretraining stage focused on learning general Slovenian language representations.

• Maximum sequence length: 1,024
• MLM masking probability: 30%
• Learning rate: 5e-4
• Optimizer: StableAdamW
• Weight decay: 0.1

Stage 2

The second stage extended the context window and continued pretraining on long-form, information-dense text.

• Maximum sequence length: 8,192
• MLM masking probability: 15%
• Learning rate: 5e-5
• Optimizer: StableAdamW
• Weight decay: 0.1
• Increased RoPE base frequency to support the longer context window

Both stages used a warmup–plateau–cooldown learning rate schedule.

Training Data

The training corpus contains approximately 13.88 billion tokens of Slovenian text.

Stage 1 used a mixture of publicly available Slovenian corpora:

• FineWeb2 — Slovenian subset (slv_Latn)
• Janes 1.0 (Wiki, Blog, Forum, News) — Slovenian text subcorpora
• OpenScience — Slovenian academic and scientific texts
• siParl 4.0 — Slovenian parliamentary proceedings corpus
• MaCoCu-sl 2.0 — Slovenian web corpus

Stage 2 continued pretraining using:

• OpenScience
• siParl 4.0

Tokenizer

SIGMA-BERT uses a custom SentencePiece BPE tokenizer trained from scratch with a vocabulary of 50,000 tokens. The tokenizer employs Metaspace pre-tokenization together with newline isolation and digit splitting to better preserve document structure and numeric information.

Usage

Load the model like any other transformers model:

from transformers import AutoTokenizer, AutoModel
 
tokenizer = AutoTokenizer.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-base")
model = AutoModel.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-base")

Masked language modeling

For MLM, load the model with AutoModelForMaskedLM and use the [MASK] token:

from transformers import AutoTokenizer, AutoModelForMaskedLM
 
model_id = "LHRS-UM-FERI/SIGMA-BERT-base"  
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
 
text = "Ljubljana je[MASK] mesto Slovenije."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
 
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
print("Predicted token:", tokenizer.decode(predicted_token_id))

⚠️ Note on [MASK] spacing: SIGMA-BERT's tokenizer uses a Metaspace pre-tokenizer, which encodes the space before a word as part of that word's token (e.g. " mesto" → ▁mesto). Because of this, if you write [MASK] in a text string, it should be placed directly against the preceding word, with no space ("je[MASK] mesto", not "je [MASK] mesto"). Adding a literal space before [MASK] produces an extra, out-of-distribution ▁ token that the model never saw during training:

"je [MASK]"  → ['▁je', '▁', '[MASK]', ...]   ✗ stray '▁' token
"je[MASK]"   → ['▁je', '[MASK]', ...]        ✓ matches training

Limitations

• SIGMA-BERT is an encoder-only model trained with masked language modeling and cannot generate text.
• The model is intended primarily for Slovenian.
• Like other pretrained language models, it may reflect biases present in the training data.

Citation

If you use SIGMA-BERT in your work, please cite:

@misc{Borovic2026,
  title = {SIGMA-BERT-base},
  author = {Mladen Borovič and Marija Jovanova},
  year = {2026},
  url = {https://huggingface.co/LHRS-UM-FERI/SIGMA-BERT-base},
  doi = { 10.57967/hf/9821 },
  publisher = { Hugging Face }
}