Add model card for EuroBERT-210M related to MLM vs CLM paper

README.md

@@ -0,0 +1,68 @@
+---
+pipeline_tag: feature-extraction
+library_name: transformers
+license: apache-2.0
+---
+
+# EuroBERT-210M: Investigating Pretraining Strategies for Encoders
+
+This repository contains the `EuroBERT-210M` model, an encoder model trained in the context of the research presented in the paper **"[Should We Still Pretrain Encoders with Masked Language Modeling?](https://huggingface.co/papers/2507.00994)"**.
+
+The paper addresses a fundamental question in NLP: whether Masked Language Modeling (MLM) or Causal Language Modeling (CLM) is the more effective pretraining objective for learning high-quality text representations in encoders. Through a series of large-scale, carefully controlled pretraining ablations, the research demonstrates that while MLM generally yields better performance across text representation tasks, CLM-trained models exhibit superior data efficiency and improved fine-tuning stability. Building on these insights, the paper introduces a biphasic training strategy that sequentially applies CLM followed by MLM, achieving optimal performance under a fixed computational budget. This strategy is particularly appealing when initializing from readily available pretrained CLM models from the existing LLM ecosystem, reducing the computational burden required to train best-in-class encoder models.
+
+This model is part of the broader `EuroBERT` project, which utilizes the `Optimus` training library for efficient language model development.
+
+- **Paper**: [Should We Still Pretrain Encoders with Masked Language Modeling?](https://huggingface.co/papers/2507.00994)
+- **Project Page**: [https://hf.co/MLMvsCLM](https://hf.co/MLMvsCLM)
+- **Code**: [https://github.com/Nicolas-BZRD/EuroBERT](https://github.com/Nicolas-BZRD/EuroBERT)
+
+## Usage
+
+You can load and use this model for feature extraction with the `transformers` library. Note that `trust_remote_code=True` is required due to custom architectural components (`SLModel`).
+
+```python
+from transformers import AutoModel, AutoTokenizer
+import torch
+
+# Replace with the actual model ID if different (e.g., 'org/repo-name')
+model_id = "EuroBERT/EuroBERT-210m" # Assuming this is the model ID for this repository
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModel.from_pretrained(model_id, trust_remote_code=True)
+
+text = "This is an example sentence to obtain embeddings for."
+inputs = tokenizer(text, return_tensors="pt")
+
+# Move to GPU if available
+# if torch.cuda.is_available():
+#     model = model.to("cuda")
+#     inputs = {k: v.to("cuda") for k, v in inputs.items()}
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# The last hidden state contains the token embeddings
+# For EuroBERT-210M (hidden_size=1152), the shape for a single input might be e.g., (1, num_tokens, 1152)
+last_hidden_states = outputs.last_hidden_state
+print(f"Shape of last hidden states: {last_hidden_states.shape}")
+
+# For sentence-level embeddings, a common approach is mean pooling:
+sentence_embedding = last_hidden_states.mean(dim=1)
+print(f"Shape of sentence embedding: {sentence_embedding.shape}")
+```
+
+## Citation
+
+If you use this model or the associated research in your work, please cite the original paper:
+
+```bibtex
+@misc{boizard2025shouldwe,
+      title={Should We Still Pretrain Encoders with Masked Language Modeling?},
+      author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves and Andr{\\'e} Martins and Ayoub Hammal and Caio Corro and C{\\'e}line Hudelot and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan and Gabriel Hautreux and Jo{\\~a}o Alves and Kevin El-Haddad and Manuel Faysse and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes and Ricardo Rei and Pierre Colombo},
+      year={2025},
+      eprint={2507.00994},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2507.00994},
+}
+```