DoctoModernBERT-fr-large

🤗 Blog | 📄 Paper | 💻 Code | 🌐 FineMed | 🩺 DoctoBERT

📚 Introduction

DoctoModernBERT-fr-large is the large-size French medical encoder in the DoctoBERT family, for biomedical and clinical NLP. It uses the ModernBERT architecture (395M parameters, up to 8192-token context) and is pretrained from scratch on FineMed-fr and FineMed-rephrased-fr.

Its training data is curated from heterogeneous open web corpora (FineWeb-2, FinePDFs, FineWiki), which bring scale, source and stylistic diversity, and inherited quality control. Each document is annotated along three axes: subdomain, educational quality, and medical-term density. DoctoModernBERT then trains on a mix of two parts: the filtered high-quality, entity-rich documents, and LLM-rephrased variants that raise medical-term density and vary the contexts around medical terms across many genres, audiences, and registers. This breadth and density build robustness to real-world clinical text, which is often noisy and inconsistent in style.

For the base-size version, see DoctoModernBERT-fr-base. For a classic RoBERTa encoder, see DoctoBERT-fr-base.

🚀 How to Use

Requires a recent transformers with ModernBERT support.

Fill-mask

Using AutoModelForMaskedLM:

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_id = "doctolib-lab/doctomodernbert-fr-large"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)

text = f"Le patient souffre d'une {tokenizer.mask_token} aiguë."
inputs = tokenizer(text, return_tensors="pt")
logits = model(**inputs).logits

masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
print(tokenizer.decode(logits[0, masked_index].argmax(-1)))

Using a pipeline:

from transformers import pipeline

fill = pipeline("fill-mask", model="doctolib-lab/doctomodernbert-fr-large")
print(fill(f"Le patient souffre d'une {fill.tokenizer.mask_token} aiguë."))

For long inputs, load with attn_implementation="flash_attention_2" for faster, more memory-efficient attention.

Fine-tuning

DoctoModernBERT fine-tunes like any BERT/ModernBERT encoder, with the appropriate task head or framework:

• For sequence classification, load it with AutoModelForSequenceClassification (see the text-classification guide).
• For token classification (NER), use AutoModelForTokenClassification (see the token-classification guide).
• For embeddings / retrieval, use Sentence Transformers or PyLate.

📐 Model Overview

Property	Value
Architecture	ModernBERT
Parameters	395M total (343M backbone, 52M embeddings)
Layers	28
Hidden size	1024
Attention heads	16
MLP	GeGLU
Intermediate size	2624
Context window	8192 tokens
Vocabulary size	50,368
Language	French

🔧 Training

The tokenizer is a SentencePiece BPE model of 50,368 tokens, trained on the entity-rich FineMed-filtered subset (educational quality >= 4 and medical-term density >= 0.1) for efficient tokenization of dense medical vocabulary. It splits digits into single tokens and uses byte fallback for unseen characters.

DoctoModernBERT-fr-large is pretrained from scratch on a mix of FineMed-filtered (FineMed-fr under the same educational-quality and medical-term-density filter) and FineMed-rephrased-fr, over three phases totaling 140B tokens:

1. Pretraining (100B tokens). Masked-language-modeling at 1024-token context on the full mix (long documents are hierarchically chunked to fit the context window rather than truncated), building broad medical-language representations.
2. Context extension (20B tokens). Extends the context window from 1024 to 8192 tokens, training on a subset upsampled toward long documents.
3. Annealing (20B tokens). Continued training on the biomedical & clinical subdomains of the mix, focusing the final updates on content closest to downstream medical use.

⚠️ Intended Use & Limitations

DoctoModernBERT is an encoder for French biomedical and clinical NLP (NER, classification, retrieval, long-document tasks), used by fine-tuning on a downstream task. It is not generative and not a medical device; its outputs must not drive clinical decisions. It was pretrained on public web, PDF, and encyclopedic medical text and reflects the biases and gaps of those sources.

⚖️ License

Released under Apache-2.0. DoctoModernBERT was trained on FineMed-fr and FineMed-rephrased-fr, which derive from FineWeb-2 / FinePDFs (ODC-BY 1.0) and FineWiki (CC BY-SA 4.0); please attribute those upstream sources.

🏛️ Acknowledgments

This work was granted access to the HPC resources of IDRIS (Jean Zay) under the allocations 2025-AD011016291 and 2026-A0200617487 made by GENCI.