SIGMA-BERT-large

Table of Contents

1. Model Summary
2. Model Details
3. Usage
4. Training
5. Training Data
6. Tokenizer
7. Limitations
8. Citation

Model Summary

SIGMA-BERT-large is a Slovenian encoder language model based on the ModernBERT architecture. It was pretrained from scratch using the masked language modeling (MLM) objective on a diverse collection of Slovenian text, including web, news, parliamentary proceedings, scientific publications, and user-generated content. The model supports sequence lengths of up to 8192 tokens and is intended for downstream language understanding tasks such as classification, retrieval, semantic similarity, and token-level prediction.

Compared to SIGMA-BERT-base, SIGMA-BERT-large increases the model capacity from 149 million to 395 million parameters while retaining support for sequences of up to 8192 tokens, providing greater representational power for downstream Slovenian NLP tasks.

As a pretrained encoder, SIGMA-BERT-large is intended to be fine-tuned for downstream Slovenian language understanding tasks such as:

• Text classification (e.g. topic, sentiment, genre)
• Named entity recognition and other token classification tasks
• Extractive question answering
• Natural language inference / semantic textual similarity
• Dense retrieval and semantic search, including over long documents
• As a backbone for rerankers in retrieval-augmented generation (RAG) pipelines

SIGMA-BERT models are available in the following variants:

• SIGMA-BERT-short - 22 layers, 149 million parameters, 1,024 token context
• SIGMA-BERT-base - 22 layers, 149 million parameters, 8,192 token context
• SIGMA-BERT-large - 28 layers, 395 million parameters, 8,192 token context

Model Details

SIGMA-BERT-large is built on the ModernBERT architecture and pretrained on Slovenian text with a Masked Language Modeling (MLM) objective.

Property	Value
Language	Slovenian
Training objective	Masked Language Modeling (MLM)
Parameters	395M
Layers	28
Hidden size	1,024
Intermediate size (GeGLU)	2,624
Attention heads	16
Vocab size	50,000
Max sequence length	8,192 (extended from 1,024)
Attention pattern	Global attention every 3rd layer; local sliding-window attention (128 tokens) elsewhere
Activation	GeGLU
Normalization / linear layers	Pre-LayerNorm, no bias terms

Training

SIGMA-BERT-large was pretrained in two stages.

Stage 1

The initial pretraining stage focused on learning general Slovenian language representations.

• Maximum sequence length: 1,024
• MLM masking probability: 30%
• Learning rate: 3e-4
• Optimizer: StableAdamW
• Weight decay: 0.1

Stage 2

The second stage extended the context window and continued pretraining on long-form, information-dense text.

• Maximum sequence length: 8,192
• MLM masking probability: 15%
• Learning rate: 3e-5
• Optimizer: StableAdamW
• Weight decay: 0.1
• Increased RoPE base frequency to support the longer context window

Both stages used a linear warmup followed by linear learning-rate decay to a minimum learning-rate ratio.

Training Data

The training corpus contains approximately 13.88 billion tokens of Slovenian text.

Stage 1 used a mixture of publicly available Slovenian corpora:

• FineWeb2 — Slovenian subset (slv_Latn)
• Janes 1.0 (Wiki, Blog, Forum, News) — Slovenian text subcorpora
• OpenScience — Slovenian academic and scientific texts
• siParl 4.0 — Slovenian parliamentary proceedings corpus
• MaCoCu-sl 2.0 — Slovenian web corpus

Stage 2 continued pretraining using:

• OpenScience
• siParl 4.0

Tokenizer

SIGMA-BERT-large uses a custom SentencePiece BPE tokenizer trained from scratch with a vocabulary of 50,000 tokens. The tokenizer employs Metaspace pre-tokenization together with newline isolation and digit splitting to better preserve document structure and numeric information.

Usage

Load the model like any other transformers model:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-large")
model = AutoModel.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-large")

Masked language modeling

For MLM, load the model with AutoModelForMaskedLM and use the [MASK] token:

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "LHRS-UM-FERI/SIGMA-BERT-large"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

text = "Ljubljana je[MASK] mesto Slovenije."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
print("Predicted token:", tokenizer.decode(predicted_token_id))

⚠️ Note on [MASK] spacing: SIGMA-BERT's tokenizer uses a Metaspace pre-tokenizer, which encodes the space before a word as part of that word's token (e.g. " mesto" → ▁mesto). Because of this, if you write [MASK] in a text string, it should be placed directly against the preceding word, with no space ("je[MASK] mesto", not "je [MASK] mesto"). Adding a literal space before [MASK] produces an extra, out-of-distribution ▁ token that the model never saw during training:

"je [MASK]"  → ['▁je', '▁', '[MASK]', ...]   ✗ stray '▁' token
"je[MASK]"   → ['▁je', '[MASK]', ...]        ✓ matches training

Limitations

• SIGMA-BERT-large is an encoder-only model trained with masked language modeling and cannot generate text.
• The model is intended primarily for Slovenian.
• Like other pretrained language models, it may reflect biases present in the training data.

Citation

If you use SIGMA-BERT-large in your work, please cite:

@misc{Borovic2026,
  title = {SIGMA-BERT-large},
  author = { Mladen Borovič and Marija Jovanova},
  year = {2026},
  url = {https://huggingface.co/LHRS-UM-FERI/SIGMA-BERT-large},
  doi = { 10.57967/hf/9822 },
  publisher = { Hugging Face }
}