Fill-Mask
Transformers
PyTorch
Safetensors
Portuguese
modernbert
portuguese
encoder
masked-lm
long-context
moberto
Eval Results (legacy)
Instructions to use Tropic-AI/moBERTo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Tropic-AI/moBERTo with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="Tropic-AI/moBERTo")# Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("Tropic-AI/moBERTo") model = AutoModelForMaskedLM.from_pretrained("Tropic-AI/moBERTo") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -215,7 +215,6 @@ Cross-encoder reranking, fine-tuned on mMARCO-PT triples.
|
|
| 215 |
|
| 216 |
> **Note on GLUE:** As expected from continued pretraining on Portuguese, English
|
| 217 |
> performance degrades. ModernBERT-base remains the strongest on GLUE (0.8301);
|
| 218 |
-
> this trade-off reflects the finite capacity of a base-sized model.
|
| 219 |
|
| 220 |
---
|
| 221 |
|
|
@@ -285,11 +284,19 @@ Same hyperparameters as Phase 1, except:
|
|
| 285 |
|
| 286 |
| Hugging Face Repo | Paper Name | Tokenizer | Long-ctx post-tr. |
|
| 287 |
|------------------------------------------------|-----------------------------|-----------|-------------------|
|
| 288 |
-
| `Tropic-AI/moBERTo-orig-tokenizer-1k` | moBERTo (orig. tok.) | Original | No |
|
| 289 |
| `Tropic-AI/moBERTo-orig-tokenizer` | moBERTo-8k (orig. tok.) | Original | Yes |
|
| 290 |
-
| `Tropic-AI/moBERTo-1k` | moBERTo-SWM (PT tok.) | PT (SWM) | No |
|
| 291 |
| **`Tropic-AI/moBERTo` *(this)* * | **moBERTo-SWM-8k (PT tok.)**| PT (SWM) | **Yes** |
|
| 292 |
|
| 293 |
---
|
| 294 |
|
| 295 |
## Citation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 215 |
|
| 216 |
> **Note on GLUE:** As expected from continued pretraining on Portuguese, English
|
| 217 |
> performance degrades. ModernBERT-base remains the strongest on GLUE (0.8301);
|
|
|
|
| 218 |
|
| 219 |
---
|
| 220 |
|
|
|
|
| 284 |
|
| 285 |
| Hugging Face Repo | Paper Name | Tokenizer | Long-ctx post-tr. |
|
| 286 |
|------------------------------------------------|-----------------------------|-----------|-------------------|
|
|
|
|
| 287 |
| `Tropic-AI/moBERTo-orig-tokenizer` | moBERTo-8k (orig. tok.) | Original | Yes |
|
|
|
|
| 288 |
| **`Tropic-AI/moBERTo` *(this)* * | **moBERTo-SWM-8k (PT tok.)**| PT (SWM) | **Yes** |
|
| 289 |
|
| 290 |
---
|
| 291 |
|
| 292 |
## Citation
|
| 293 |
+
|
| 294 |
+
@misc{laitz2026mobertomodernencoderportuguese,
|
| 295 |
+
title={moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT},
|
| 296 |
+
author={Thiago Laitz and Thales Sales Almeida and João Guilherme Alves Santos and Giovana Kerche Bonás},
|
| 297 |
+
year={2026},
|
| 298 |
+
eprint={2606.22722},
|
| 299 |
+
archivePrefix={arXiv},
|
| 300 |
+
primaryClass={cs.CL},
|
| 301 |
+
url={https://arxiv.org/abs/2606.22722},
|
| 302 |
+
}
|