---
language:
- en
- zh
- es
- ja
- ru
tags:
- fill-mask
- clinical-nlp
- multilingual
- bert
license: mit
---

# MultiClinicalBERT: A Multilingual Transformer Pretrained on Real-World Clinical Notes

MultiClinicalBERT is a multilingual transformer model pretrained on real-world clinical notes across multiple languages. It is designed to provide strong and consistent performance for clinical NLP tasks in both high-resource and low-resource settings.

To the best of our knowledge, this is the first open-source BERT model pretrained specifically on multilingual real-world clinical notes.


## Model Overview

MultiClinicalBERT is initialized from `bert-base-multilingual-cased` and further pretrained using a two-stage domain-adaptive strategy on a large-scale multilingual clinical corpus (BRIDGE), combined with biomedical and general-domain data.

The model captures:
- Clinical terminology and documentation patterns
- Cross-lingual representations for medical text
- Robust performance across diverse healthcare datasets


## Pretraining Data

The model is trained on a mixture of three data sources:

### 1. Clinical Data (BRIDGE Corpus)
- 87 multilingual clinical datasets  
- ~1.42M documents  
- ~995M tokens  
- Languages: English, Chinese, Spanish, Japanese, Russian  

This dataset reflects real-world clinical practice and is the core contribution of this work.

### 2. Biomedical Literature (PubMed)
- ~1.25M documents  
- ~194M tokens  

Provides domain knowledge and medical terminology.

### 3. General-Domain Text (Wikipedia)
- ~5.8K documents  
- ~43M tokens  
- Languages: Spanish, Japanese, Russian  

Improves general linguistic coverage.

### Total
- ~2.7M documents  >1.2B tokens  


## Pretraining Strategy

We adopt a **two-stage domain-adaptive pretraining approach**:

### Stage 1: Mixed-domain pretraining
- Data: BRIDGE + PubMed + Wikipedia  
- Goal: Inject biomedical and multilingual knowledge  

### Stage 2: Clinical-specific adaptation
- Data: BRIDGE only  
- Goal: Learn fine-grained clinical language patterns  

### Objective
- Masked Language Modeling (MLM)  
- 15% token masking  


## Evaluation

We evaluate MultiClinicalBERT on **11 clinical NLP tasks across 5 languages**:

- English: MIMIC-III Mortality, MedNLI, MIMIC-IV CDM  
- Chinese: CEMR, IMCS-V2 NER  
- Japanese: IFMIR NER, IFMIR Incident Type  
- Russian: RuMedNLI, RuCCoNNER  
- Spanish: De-identification, PPTS  

### Key Results
- Consistently outperforms multilingual BERT (mBERT)
- Matches or exceeds strong language-specific models
- Largest gains observed in low-resource settings
- Statistically significant improvements (Welch’s t-test, p < 0.05)

Example:
- MedNLI: **83.90% accuracy**
- CEMR: **86.38% accuracy**
- IFMIR NER: **85.53 F1**
- RuMedNLI: **78.31% accuracy**


## Key Contributions

- First BERT model pretrained on **multilingual real-world clinical notes**
- Large-scale clinical corpus (BRIDGE) with diverse languages
- Effective **two-stage domain adaptation strategy**
- Strong performance across **multiple languages and tasks**
- Suitable for:
  - Clinical NLP
  - Multilingual medical text understanding
  - Retrieval-augmented generation (RAG)
  - Clinical decision support systems


## Usage

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("YLab-Open/MultiClinicalBERT")
model = AutoModel.from_pretrained("YLab-Open/MultiClinicalBERT")