Sparseup-fr: A French replication of Linkup's sparse embedding model

Introduction

Three weeks ago, Linkup released its first open-source model, the linkup-sparseup-embed-v1 [1], accompanied by a blog post [2] explaining how they built it.
This is an embedding sparse of 150 M parameters based on LightOn's [3] work published earlier this year. The details are in his blog post and in the .py files available on Hugging Face but, overall, his approach is to start from the LateOn-unsupervised, to graft the MLM head of the ModernBERT [4], then finetune the whole thing on the data provided by LightOn.
The resulting sparse model gaining 2 points on BEIR-13 compared to the previous SOTA of equivalent size for this model family.
To familiarize ourselves with the approach, we wanted to replicate the Linkup's model, but in French rather than English.

Model Design

In order to be able to apply Linkup's methodology to French, we made two architectural changes.
The first is to change the backbone using the mLateOn-unsupervised [5], which is the multilingual version of LateOn-unsupervised. It handles 10 languages, including French, which we are interested in.
The second is to use the MLM head mmBERT-base [6] rather than the ModernBERT. However, we did not use it as it is because we have trimmed [7] to keep only 50,240 tokens mined on French rather than the original 256,000.
The idea here is to get a French model of equivalent size to the English version (the latter uses 50,370 tokens in its vocabulary, but we didn't want to match this number exactly in order to keep a multiple of 64, which is beneficial to GPUs).

Training

We trained our model on a single RTX 6000 Pro for about 55 hours, which is the time it takes to complete a full epoch on the Frecnh training data provided by LightOn.
Regarding hyperparameters, we invite readers to consult the model card. These are the same ones used by Linkup. The only exception is that with the memory available on our GPU, we allowed ourselves to double the size of the request and the document (from 128 to 256 and from 512 to 1024 respectively).

Results

To evaluate the models, we use a cleaned version of LightOn's NanoBEIR-fr: we made sure that there were no data leaks between the training data from LightOn and these benchmarks, but also removed suspicious lines (see for example this GitHub issue listing problems encountered for NanoBEIR). However, we did not look at possible annotation problems (pointed out for example by [8] or [9]).
We put the version of this benchmark available on our Hugging Face profile (the metadata folder lists all the problematic lines).

As far as models are concerned, we are interested in two comparisons.
First, we evaluate the finetuned model against the mLateOn and mDenseOn, as the Linkup teams have done. Indeed, as they use LightOn's backbones and data, it allows them to fully compare the different major architectural approaches in Information Retrieval. For English, they observe that LateOn (late interaction) is much more efficient than DenseOn, which itself is slightly above their sparse model. We then want to determine whether or not we observe the same trends in French.
Second, we compare the trained model to other sparse models in French available on the Hugging Face Hub, in order to get an idea of its position among the existing open source solutions. More specifically, it is the splade-camembert-base-v2 [10] by Arthur Bresnu, who is monolingual, and the opensearch-neural-sparse-encoding-multilingual-v1 [11] by Amazon/Linux, which is multilingual and includes French.

Note that we ran the evaluations on two different configurations: one where the documents are sliced to the maximum size of what each of the models can take in context, and the other where a sliding window is applied to the entire document, with a window size equal to the context size, plus an overlap of 64 tokens. The latter case, which moreover looks more like a real RAG use case, should allow us to evaluate more accurately the different sparse models, which have a smaller request and document size than the LightOn models.

Finally, we specify that in the following tables, the blue cells indicate the performance of the best model across all families, while the green cells indicate the performance of the best sparse model on the benchmark.

Configuration with truncation

Sliding window configuration

Observations

We can see, like the Linkup team, that the late interaction model performs better than the dense, and itself better than the different sparse models. The use of a sliding window does not improve the LightOn models, which already benefit from a query and document context size of 8,192 tokens. It appears to be slightly beneficial to the French sparse models, but strangely degrades those of Sparseup. We do not report the results here because we focus on the French one, but to verify this behavior we also evaluated the models on the cleaned NanoBEIR-en and observed that the sliding window does benefit Sparseup (also allowing to surpass the mDenseOn).

Regarding the French models, the trained model outperforms the other two available so far on the Hub by more than 3 points for the splade-camembert-base-v2 (our model being nevertheless 35% larger) and by 2 points for the opensearch-neural-sparse-encoding-multilingual-v1 (this time being 11% smaller). It is first on 8 NanoBEIR-fr datasets, where opensearch is first on 3 and splade-camembert-base-v2 on 2.
The Sparseup is lower but it's expected because it's monolingual English (still better than a bm25).

Note that for a comparison with an equivalent number of parameters, we could consider retraining two models (one by keeping only 32,768 tokens to reach the size of the splade-camembert-base-v2 and another by keeping 73,556 tokens to reach that of the opensearch-neural-sparse-encoding-multilingual-v1), but this would again require several dozen GPU hours of training that we did not have for this first experiment consisting simply of verifying the methodology applied by Linkup.

Experience feedback

Initially, we wanted to be able to present a Linkup-style model in French, resulting from a merge of several saved weights. At the end of the training, we realized that many saves had not been made.
It turns out that the push_to_hub refuses to push the weights when the README is larger than 1 MB. The sentence_transformers library writes the intermediate results in the YAML metadata of the README at each evaluation, so this problem can appear if you make too many backups. We invite you to pay attention to this point by writing a small patch in your training code so as not to save the results in the YAML.

Conclusion and continuation

This simple reproduction of Sparseup's methodology in another language is conclusive since we obtain a new SOTA in French among the sparse models available in open access.
On the strength of this initial feedback, we plan to push the experiments further, in particular with more data, a merge of weights, or a larger version of the model in French. Another possibility would be to offer a multilingual version.

Actually, we make available this v0, in this Hugging Face collection, the trained model as well as the cleaned benchmark.

Hope this is helpful to you:)

References

1. linkup-sparseup-embed-v1 by Thibault FORMAL, Pierre ERBACHER, Cosimo RULLI (2026). link
2. Introducing SPARSEUP by Linkup: a move towards more competitive open sparse models by Thibault FORMAL, Pierre ERBACHER, Cosimo RULLI (2026). link
3. DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search by Raphaël SOURTY, Antoine CHAFFIN, Paulo Roberto MOURA JUNIOR, Amélie CHATELAIN (2026). link
4. Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference by Benjamin WARNER, Antoine CHAFFIN, Benjamin CLAVIÉ, et al. (2024). link
5. mDenseOn and mLateOn: more signal, less noise for multilingual agentic search by Raphaël SOURTY, Antoine CHAFFIN, Paulo Roberto MOURA JUNIOR, Amélie CHATELAIN (2026). link
6. mmBERT: A Modern Multilingual Encoder with Annealed Language Learning by Marc MARONE, Orion WELLER, William FLESHMAN, Eugene YANG, Dawn LAWRIE, Benjamin VAN DURME (2025). link
7. Introduction to Trimming ✂ by Loïck BOURDOIS, Tom AARSEN, Bram VANROY, Woojun JUNG, Manuel ROMERO, Prithiv SAKHI (2026). link
8. Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR by Nandan THAKUR, Luiz BONIFACIO, Maik FRÖBE, et al. (2024). link
9. Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs by Nandan THAKUR, Crystina ZHANG, Xueguang MA, Jimmy LIN (2025). link
10. splade-camembert-base-v2 by Arthur BRESNU (2025). link
11. Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers by Zhichao GENG, Yiwen WANG, Dongyu RU, Yang YANG (2025). link

Citation

@misc {hf_blogpost_french_sparseup,   
    author       = { {BOURDOIS, Loïck} },  
    organization  = { {AlphaEdge AI} },  
    title        = { Sparseup-fr: A French replication of Linkup's sparse embedding model},  
    year         = 2026,  
    url          = { https://huggingface.co/blog/alphaedge-ai/french-sparseup },  
    publisher    = { Hugging Face }
}