---
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- dense
- multilingual
- code search
- arxiv:2607.27178
pipeline_tag: sentence-similarity
library_name: sentence-transformers
license: apache-2.0
language:
- en
- fr
- de
- it
- es
- pt
- sv
- 'no'
- ar
- code
base_model:
- jhu-clsp/mmBERT-base
---
[](https://lighton.ai)
[](https://www.linkedin.com/company/lighton/)
[](https://x.com/LightOnIO)
📚 [Collection](https://huggingface.co/collections/lightonai/mdenseon-and-mlateon) | 📝 [Multilingual Blog](https://huggingface.co/blog/lightonai/mdenseon-mlateon) | 📝 [English Blog](https://huggingface.co/blog/lightonai/denseon-lateon) | 📝 [Paper](https://arxiv.org/abs/2607.27178)
mDenseOn-unsupervised
State-of-the-Art Multilingual Dense Retrieval Model by LightOn
mDenseOn |
mLateOn |
DenseOn |
LateOn |
PyLate |
FastPlaid
> 🎯 **TL;DR**: The intermediate multilingual dense checkpoint produced by **Stage 1 only** (unsupervised contrastive pre-training) of the [mDenseOn](https://huggingface.co/lightonai/mDenseOn) pipeline, trained on a multilingual dataset with **2.8B query–document pairs across nine languages** (including 25% cross-lingual pairs). Released as a **strong starting point for your own supervised fine-tuning, knowledge distillation, or downstream adaptation**.
## About the mDenseOn / mLateOn Family
With [DenseOn](https://huggingface.co/lightonai/DenseOn) and [LateOn](https://huggingface.co/lightonai/LateOn), we demonstrated that an open, carefully curated data recipe can match closed-data retrieval models on English. mDenseOn and mLateOn extend this recipe to multilingual, long-context, and code retrieval.
Rather than independently collecting multilingual corpora from scratch (which would be expensive, uneven across languages, and hard to curate at the same quality), we applied the **translate-train** approach: machine-translating our validated English data into eight target languages (French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, and Arabic) and adding cross-lingual pairs for cross-lingual alignment.
For more information, please read our [multilingual blog post](https://huggingface.co/blog/lightonai/mdenseon-mlateon) and our [English blog post](https://huggingface.co/blog/lightonai/denseon-lateon).
## mDenseOn-unsupervised
mDenseOn-unsupervised is the output of the **first stage** of the [mDenseOn](https://huggingface.co/lightonai/mDenseOn) training pipeline. It has been pre-trained on a large, filtered multilingual corpus of approximately **2.8B query–document pairs** across nine languages (including **25% cross-lingual pairs**) using in-batch contrastive learning, but has **not** yet been fine-tuned with mined hard negatives.
For most production use cases, you should use the **fully-trained [mDenseOn](https://huggingface.co/lightonai/mDenseOn)** instead. This unsupervised checkpoint is intended for:
- Researchers studying what each pipeline stage contributes in a multilingual setting
- Practitioners who want to fine-tune on their own domain-specific or language-specific data
- Distillation experiments where you want to start from a strong but un-aligned multilingual base
- Anyone running their own ablations on hard-negative mining strategies or translate-train recipes
If your use case permits multi-vector retrieval, also consider [mLateOn-unsupervised](https://huggingface.co/lightonai/mLateOn-unsupervised), the late-interaction counterpart of this checkpoint. The fully-trained mLateOn achieves substantially stronger results, especially on multilingual and long-context tasks, and generalizes to languages outside of the training set.
For more information, please read our [multilingual models blog post](https://huggingface.co/blog/lightonai/mdenseon-mlateon), our [English models blog post](https://huggingface.co/blog/lightonai/denseon-lateon) and our [paper](https://arxiv.org/abs/2607.27178).
## Related Checkpoints
| Model | Description | Link |
|:---|:---|:---|
| **mDenseOn-unsupervised** *(this card)* | Multilingual dense, pre-training only | [`lightonai/mDenseOn-unsupervised`](https://huggingface.co/lightonai/mDenseOn-unsupervised) |
| mDenseOn | Multilingual dense retriever *(recommended)* | [`lightonai/mDenseOn`](https://huggingface.co/lightonai/mDenseOn) |
| mLateOn-unsupervised | Multilingual late-interaction, pre-training only | [`lightonai/mLateOn-unsupervised`](https://huggingface.co/lightonai/mLateOn-unsupervised) |
| mLateOn | Multilingual late-interaction retriever (strongest multilingual) | [`lightonai/mLateOn`](https://huggingface.co/lightonai/mLateOn) |
| DenseOn-unsupervised | English-only dense, pre-training only | [`lightonai/DenseOn-unsupervised`](https://huggingface.co/lightonai/DenseOn-unsupervised) |
| DenseOn | English-only dense retriever | [`lightonai/DenseOn`](https://huggingface.co/lightonai/DenseOn) |
| LateOn-unsupervised | English-only late-interaction, pre-training only | [`lightonai/LateOn-unsupervised`](https://huggingface.co/lightonai/LateOn-unsupervised) |
| LateOn | English-only late-interaction retriever | [`lightonai/LateOn`](https://huggingface.co/lightonai/LateOn) |
## Model Details
### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base)
- **Maximum Sequence Length:** 8,192 tokens
- **Output Dimensionality:** 768 dimensions
- **Similarity Function:** Cosine Similarity
- **Pooling:** [CLS] token
- **Prompts:** `query:` for queries, `document:` for documents
- **Languages:** English, French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, Arabic
- **License:** Apache 2.0
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
### Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer
# Download from the Hub
model = SentenceTransformer("lightonai/mDenseOn-unsupervised")
# Run inference with multilingual queries and documents
queries = [
"Quelle planète est connue comme la planète rouge ?",
"Which planet is known as the Red Planet?",
]
documents = [
"Venus wird oft als Zwilling der Erde bezeichnet wegen ihrer ähnlichen Größe.",
"Mars, connu pour son apparence rougeâtre, est souvent appelé la planète rouge.",
"Marte, conocido por su apariencia rojiza, es a menudo llamado el Planeta Rojo.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
]
query_embeddings = model.encode(queries, prompt_name="query")
document_embeddings = model.encode(documents, prompt_name="document")
print(query_embeddings.shape, document_embeddings.shape)
# [2, 768] [4, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
```
### Downstream Usage (Sentence Transformers)
You can fine-tune this model on your own dataset. This checkpoint is specifically designed as a strong starting point for supervised fine-tuning with mined hard negatives or knowledge distillation, following the recipe described in our [multilingual blog post](https://huggingface.co/blog/lightonai/mdenseon-mlateon).
### Framework Versions
- Python: 3.11.10
- Sentence Transformers: 5.1.1
- Transformers: 4.57.5
- PyTorch: 2.9.0+cu128
- Accelerate: 1.12.0
- Datasets: 3.6.0
- Tokenizers: 0.22.1
## Citation
### BibTeX
#### mDenseOn and mLateOn
```bibtex
@misc{sourty2026denseonlateonfullyopen,
title = {DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search},
author = {Raphaël Sourty and Antoine Chaffin and Paulo Roberto Moura Junior and Amélie Chatelain},
year = {2026},
eprint = {2607.27178},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2607.27178},
}
```
#### DenseOn and LateOn
```bibtex
@misc{sourty2026denseonlateon,
title={DenseOn with the LateOn: Open State-of-the-Art Single and Multi-Vector Models},
author={Sourty, Raphael and Chaffin, Antoine and Weller, Orion and Moura Junior, Paulo Roberto and Chatelain, Amelie},
year={2026},
howpublished={\url{https://huggingface.co/blog/lightonai/denseon-lateon}},
}
```
#### PyLate
```bibtex
@inproceedings{DBLP:conf/cikm/ChaffinS25,
author = {Antoine Chaffin and
Rapha{\"{e}}l Sourty},
editor = {Meeyoung Cha and
Chanyoung Park and
Noseong Park and
Carl Yang and
Senjuti Basu Roy and
Jessie Li and
Jaap Kamps and
Kijung Shin and
Bryan Hooi and
Lifang He},
title = {PyLate: Flexible Training and Retrieval for Late Interaction Models},
booktitle = {Proceedings of the 34th {ACM} International Conference on Information
and Knowledge Management, {CIKM} 2025, Seoul, Republic of Korea, November
10-14, 2025},
pages = {6334--6339},
publisher = {{ACM}},
year = {2025},
url = {https://github.com/lightonai/pylate},
doi = {10.1145/3746252.3761608},
}
```
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084"
}
```
## Acknowledgements
We thank Eugene Yang for his feedback on adapting our English study to multilinguality through translate-train. We again thank Xin Zhang, Zach Nussbaum, Tom Aarsen, Bo Wang, Eugene Yang, Benjamin Clavié, Nandan Thakur, Oskar Hallström and Iacopo Poli for their valuable contributions and feedback on the original English study. We thank Orion Weller for building the FineWeb-derived Common Crawl split as well as for his feedback and help. We are grateful to the teams behind Sentence Transformers, BEIR, and MIRACL, and to the open-source retrieval community, in particular the authors of Nomic Embed.
This work was granted access to the HPC resources of IDRIS under GENCI allocations AS011016449, A0181016214, and A0171015706 (Jean Zay supercomputer). We also acknowledge the Barcelona Supercomputing Center (BSC-CNS) for providing access to MareNostrum 5 under EuroHPC AI Factory Fast Lane project EHPC-AIF-2025FL01-445.
This project is also supported by the OpenEuroLLM project, co-funded by the Digital Europe Programme under GA no. 101195233.