e5-sk-small

e5-sk-small is a compact Slovak text embedding model (45M parameters, 384-dimensional embeddings) built by applying vocabulary trimming and fine-tuning to multilingual-e5-small. Despite being 62% smaller than the original model, it achieves competitive performance on SkMTEB — the first comprehensive Slovak text embedding benchmark — while remaining fully locally deployable.

Released as part of the SkMTEB project (paper · GitHub · collection).

For a larger, higher-accuracy variant, see e5-sk-large (365M parameters).

---

Model Details

Base model	intfloat/multilingual-e5-small (via mrshu/e5-small-trim-sk)
Parameters	45M (vs. 118M original — 62% reduction)
Embedding dimension	384
Max sequence length	256 tokens
Pooling	Mean pooling
Languages	Slovak (primary); Slovak–English and Slovak–Czech cross-lingual tasks preserved

---

Usage

This model follows the standard E5 prefix convention: prepend query: to queries and passage: to documents during retrieval. For symmetric tasks (STS, clustering, classification), no prefix is needed.

With sentence-transformers

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("slovak-nlp/e5-sk-small")
 
# Retrieval
query_embedding = model.encode("query: Čo je hlavné mesto Slovenska?")
passage_embedding = model.encode("passage: Bratislava je hlavné a najväčšie mesto Slovenska.")
similarity = model.similarity(query_embedding, passage_embedding)
print(similarity)  # tensor([[0.92...]])
 
# Batch encoding
sentences = [
    "query: Aké je počasie v Bratislave?",
    "passage: V Bratislave je dnes slnečno a teplo.",
    "passage: Bratislava leží na brehu Dunaja.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 384)

With transformers directly

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel
 
def average_pool(last_hidden_states, attention_mask):
    last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0)
    return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
 
tokenizer = AutoTokenizer.from_pretrained("slovak-nlp/e5-sk-small")
model = AutoModel.from_pretrained("slovak-nlp/e5-sk-small")
 
texts = [
    "query: Čo je hlavné mesto Slovenska?",
    "passage: Bratislava je hlavné a najväčšie mesto Slovenska.",
]
 
batch_dict = tokenizer(texts, max_length=512, padding=True, truncation=True, return_tensors="pt")
with torch.no_grad():
    outputs = model(**batch_dict)
 
embeddings = average_pool(outputs.last_hidden_state, batch_dict["attention_mask"])
embeddings = F.normalize(embeddings, p=2, dim=1)
print((embeddings[0] @ embeddings[1]).item())

Prefix guide

Task	Input prefix
Search / retrieval query	query:
Document / passage to index	passage:
STS, clustering, classification	(no prefix)

---

Training

How it was built

Step 1 — Vocabulary Trimming. Before fine-tuning, Vocabulary Trimming (Ushio et al., 2023) was applied to multilingual-e5-small to remove tokens irrelevant to Slovak. Token frequencies were computed on FineWeb2-Slovak, a quality-filtered Slovak web corpus, and the top 60K tokens (out of 250K) were retained. This reduced the model from 118M to 45M parameters (62% reduction) without meaningful performance loss.

Step 2 — Fine-tuning. The trimmed model was fine-tuned on curated Slovak datasets from the skLEP benchmark:

Dataset	Task	Pairs
SK-SQuAD	Question–context retrieval	~72K
Slovak NLI (from XNLI)	Entailment	~393K
Slovak STS (from GLUE STSb)	Similarity scoring	~6K
Slovak RTE (from GLUE)	Textual entailment	~2.5K

Training configuration

Hyperparameter	Value
Pooling	Mean pooling
Max sequence length	256 tokens
Batch size	32
Learning rate	2 × 10⁻⁵
LR scheduler	Linear with 10% warmup
Epochs	3
Loss (STS)	Cosine Similarity Loss
Loss (other)	Multiple Negatives Ranking Loss
Hardware	1× NVIDIA H100 (~30 min)
Seed	42

---

Evaluation: SkMTEB Results

Evaluated on SkMTEB — 31 datasets across 7 task types. Scores are percentages (higher is better).

Model	Params	All	Bitext	Classif.	Clustering	Pair Clf.	Reranking	Retrieval	STS
e5-sk-small	45M	70.56	91.34	60.84	40.95	66.05	84.94	78.64	81.32
multilingual-e5-small	118M	70.32	91.29	59.81	41.09	64.66	85.13	79.21	81.25
text-embedding-3-small (API)	—	70.48	91.82	62.13	43.64	63.24	82.37	76.96	79.56
e5-sk-large (ours)	365M	74.70	96.39	66.34	41.43	67.32	87.81	85.60	86.25

e5-sk-small matches text-embedding-3-small (TOST equivalence test: 90% CI within ±2 points), while being open-weight, locally deployable, and free to run. It also outperforms the original multilingual-e5-small (118M) at 62% of the size. Cross-lingual Slovak–English and Slovak–Czech bitext mining performance is preserved within 1 F1 point compared to the original model.

---

Intended Uses

• Semantic search and retrieval-augmented generation (RAG) over Slovak text
• Semantic textual similarity (STS)
• Text clustering and classification via embedding features
• Cross-lingual retrieval (Slovak–English, Slovak–Czech)
• Edge deployment, high-throughput inference, or any setting where model size matters

Limitations

• Optimised for Slovak; cross-lingual transfer to non-Slavic languages is not evaluated.
• Vocabulary trimming removes non-Slovak tokens; performance on heavily code-mixed text may be reduced.
• Training data skews toward news, parliamentary, and encyclopedic domains.
• Max sequence length during fine-tuning is 256 tokens (underlying architecture supports up to 512).

---

Citation

@inproceedings{suppa2025skmteb,
  title     = {{SkMTEB}: {Slovak} Massive Text Embedding Benchmark and Model Adaptation},
  author    = {{\v{S}}uppa, Marek and Ridzik, Andrej and Hl{\'a}dek, Daniel and
               Kna{\v{z}}ekov{\'a}, Nat{\'a}lia and Ondrejov{\'a}, Vikt{\'o}ria},
  year      = {2025},
  eprint    = {2606.13647},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2606.13647}
}

@inproceedings{reimers-2019-sentence-bert,
  title     = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
  author    = {Reimers, Nils and Gurevych, Iryna},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
  year      = {2019},
  publisher = {Association for Computational Linguistics},
  url       = {https://arxiv.org/abs/1908.10084}
}

---

License

MIT