Fill-Mask
Transformers
PyTorch
Safetensors
Portuguese
modernbert
portuguese
encoder
masked-lm
long-context
moberto
Eval Results (legacy)
Instructions to use Tropic-AI/moBERTo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Tropic-AI/moBERTo with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="Tropic-AI/moBERTo")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("Tropic-AI/moBERTo") model = AutoModelForMaskedLM.from_pretrained("Tropic-AI/moBERTo", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,295 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- pt
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
library_name: transformers
|
| 6 |
+
tags:
|
| 7 |
+
- modernbert
|
| 8 |
+
- portuguese
|
| 9 |
+
- encoder
|
| 10 |
+
- masked-lm
|
| 11 |
+
- long-context
|
| 12 |
+
- moberto
|
| 13 |
+
pipeline_tag: fill-mask
|
| 14 |
+
base_model: answerdotai/ModernBERT-base
|
| 15 |
+
datasets:
|
| 16 |
+
- HuggingFaceFW/fineweb-2
|
| 17 |
+
metrics:
|
| 18 |
+
- nDCG@10
|
| 19 |
+
- F1
|
| 20 |
+
model-index:
|
| 21 |
+
- name: moBERTo
|
| 22 |
+
results:
|
| 23 |
+
- task:
|
| 24 |
+
type: text-retrieval
|
| 25 |
+
name: Reranking
|
| 26 |
+
dataset:
|
| 27 |
+
name: QUATI
|
| 28 |
+
type: quati
|
| 29 |
+
metrics:
|
| 30 |
+
- type: nDCG@10
|
| 31 |
+
value: 0.5609
|
| 32 |
+
- task:
|
| 33 |
+
type: text-retrieval
|
| 34 |
+
name: Reranking
|
| 35 |
+
dataset:
|
| 36 |
+
name: mMARCO-PT
|
| 37 |
+
type: mmarco-pt
|
| 38 |
+
metrics:
|
| 39 |
+
- type: nDCG@10
|
| 40 |
+
value: 0.5147
|
| 41 |
+
- task:
|
| 42 |
+
type: text-retrieval
|
| 43 |
+
name: Reranking
|
| 44 |
+
dataset:
|
| 45 |
+
name: Robust04-PT
|
| 46 |
+
type: robust04-pt
|
| 47 |
+
metrics:
|
| 48 |
+
- type: nDCG@10
|
| 49 |
+
value: 0.5010
|
| 50 |
+
- task:
|
| 51 |
+
type: text-retrieval
|
| 52 |
+
name: Long-Context Reranking
|
| 53 |
+
dataset:
|
| 54 |
+
name: MLDR (PT)
|
| 55 |
+
type: mldr
|
| 56 |
+
metrics:
|
| 57 |
+
- type: nDCG@10
|
| 58 |
+
value: 0.5777
|
| 59 |
+
name: nDCG@10 at 8192 tokens
|
| 60 |
+
- task:
|
| 61 |
+
type: token-classification
|
| 62 |
+
name: Named Entity Recognition
|
| 63 |
+
dataset:
|
| 64 |
+
name: LeNER-Br
|
| 65 |
+
type: lener-br
|
| 66 |
+
metrics:
|
| 67 |
+
- type: F1
|
| 68 |
+
value: 0.8726
|
| 69 |
+
- task:
|
| 70 |
+
type: text-classification
|
| 71 |
+
name: Natural Language Understanding
|
| 72 |
+
dataset:
|
| 73 |
+
name: PLUE-PT
|
| 74 |
+
type: plue-pt
|
| 75 |
+
metrics:
|
| 76 |
+
- type: F1
|
| 77 |
+
value: 0.6980
|
| 78 |
+
---
|
| 79 |
+
|
| 80 |
+
# moBERTo
|
| 81 |
+
|
| 82 |
+
> **Paper name:** This model is referred to as **`moBERTo-SWM-8k (PT tok.)`** in the
|
| 83 |
+
> moBERTo paper. It is the **best-performing variant** of the moBERTo family, achieving
|
| 84 |
+
> the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks
|
| 85 |
+
> and the best PLUE-PT score.
|
| 86 |
+
|
| 87 |
+
`moBERTo` is a Portuguese adaptation of [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base),
|
| 88 |
+
obtained through continued pretraining on a curated 12-billion-token Portuguese corpus
|
| 89 |
+
(60B training tokens, 5 epochs) followed by a long-context post-training phase at
|
| 90 |
+
8,192-token context.
|
| 91 |
+
|
| 92 |
+
It combines four adaptation strategies:
|
| 93 |
+
|
| 94 |
+
1. **Continued pretraining** from the original ModernBERT-base checkpoint, preserving
|
| 95 |
+
the long-context capabilities learned during the original 2T-token English pretraining.
|
| 96 |
+
2. **Portuguese tokenizer** with vocabulary optimized for Portuguese text.
|
| 97 |
+
3. **Subword Matching (SWM) embedding transfer**, which initializes each new Portuguese
|
| 98 |
+
token's embedding as a combination of the original ModernBERT subword embeddings,
|
| 99 |
+
keeping the model close to its pretrained representation space.
|
| 100 |
+
4. **Long-context post-training** at 8,192 tokens for an additional 10B tokens.
|
| 101 |
+
|
| 102 |
+
The model preserves all architectural advances of ModernBERT: rotary positional
|
| 103 |
+
embeddings (RoPE), alternating local–global attention, FlashAttention, and unpadding,
|
| 104 |
+
with a native context window of **8,192 tokens**.
|
| 105 |
+
|
| 106 |
+
---
|
| 107 |
+
|
| 108 |
+
## Model Details
|
| 109 |
+
|
| 110 |
+
| Attribute | Value |
|
| 111 |
+
|------------------------|------------------------------------------------------|
|
| 112 |
+
| Architecture | ModernBERT (encoder-only) |
|
| 113 |
+
| Base checkpoint | `answerdotai/ModernBERT-base` |
|
| 114 |
+
| Parameters | ~150M |
|
| 115 |
+
| Max context length | 8,192 tokens |
|
| 116 |
+
| Tokenizer | Portuguese (custom vocabulary) |
|
| 117 |
+
| Embedding init | Subword Matching (SWM) transfer |
|
| 118 |
+
| Pretraining tokens | 60B (5 epochs over 12B-token corpus) |
|
| 119 |
+
| Long-context post-tr. | 10B tokens at 8,192-token context |
|
| 120 |
+
| Training corpus | FineWeb-2 (PT subset) filtered with ClassiCC-PT |
|
| 121 |
+
| Framework | Composer |
|
| 122 |
+
| Precision | bfloat16 |
|
| 123 |
+
| License | Apache 2.0 |
|
| 124 |
+
|
| 125 |
+
---
|
| 126 |
+
|
| 127 |
+
## Quick Start
|
| 128 |
+
|
| 129 |
+
```python
|
| 130 |
+
long_text = "..." # documento longo em português
|
| 131 |
+
inputs = tokenizer(long_text, max_length=8192, truncation=True, return_tensors="pt")
|
| 132 |
+
with torch.no_grad():
|
| 133 |
+
outputs = model(**inputs)
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
### Recommended for downstream tasks
|
| 137 |
+
|
| 138 |
+
This model is best used as a backbone for fine-tuning on:
|
| 139 |
+
|
| 140 |
+
- **Cross-encoder reranking** (information retrieval)
|
| 141 |
+
- **Document classification**
|
| 142 |
+
- **Named entity recognition**
|
| 143 |
+
- **Natural language inference / semantic textual similarity**
|
| 144 |
+
- **Long-document retrieval** (up to 8,192 tokens)
|
| 145 |
+
|
| 146 |
+
---
|
| 147 |
+
|
| 148 |
+
## Evaluation Results
|
| 149 |
+
|
| 150 |
+
All metrics are reported on Portuguese benchmarks. Best results are in **bold**;
|
| 151 |
+
second-best are <ins>underlined</ins>.
|
| 152 |
+
|
| 153 |
+
### Information Retrieval (Reranking, nDCG@10)
|
| 154 |
+
|
| 155 |
+
Cross-encoder reranking, fine-tuned on mMARCO-PT triples.
|
| 156 |
+
|
| 157 |
+
| Model | QUATI | mMARCO | Robust04 | **Avg.** |
|
| 158 |
+
|-----------------------------|------------------|------------------|------------------|------------------|
|
| 159 |
+
| BERT-base | 0.2846 | 0.4050 | 0.2389 | 0.3095 |
|
| 160 |
+
| BERTimbau-base | 0.4870 | 0.5005 | 0.4138 | 0.4671 |
|
| 161 |
+
| ModernBERT-base | 0.3779 | 0.4799 | 0.2988 | 0.3855 |
|
| 162 |
+
| NeoBERT-base | 0.4000 | 0.4698 | 0.3117 | 0.3938 |
|
| 163 |
+
| Qwen3-0.6B-base | 0.4248 | 0.5065 | 0.2994 | 0.4102 |
|
| 164 |
+
| moBERTo-orig-tokenizer-1k | 0.5383 | 0.5109 | 0.4510 | 0.5001 |
|
| 165 |
+
| moBERTo-orig-tokenizer | 0.5231 | 0.5089 | 0.4516 | 0.4945 |
|
| 166 |
+
| moBERTo-1k | <ins>0.5410</ins>| **0.5169** | <ins>0.4782</ins>| <ins>0.5120</ins>|
|
| 167 |
+
| **moBERTo (this model)** | **0.5609** | <ins>0.5147</ins>| **0.5010** | **0.5255** |
|
| 168 |
+
|
| 169 |
+
### Long-Context Retrieval (MLDR, nDCG@10)
|
| 170 |
+
|
| 171 |
+
| Model | 512 | 2,048 | 4,096 | 8,192 |
|
| 172 |
+
|-----------------------------|------------------|------------------|------------------|------------------|
|
| 173 |
+
| ModernBERT-base | 0.4054 | 0.4206 | 0.3015 | 0.2867 |
|
| 174 |
+
| NeoBERT-base | 0.4746 | 0.5149 | 0.4676 | -- |
|
| 175 |
+
| Qwen3-0.6B-base | 0.3560 | 0.4023 | 0.4241 | 0.5351 |
|
| 176 |
+
| moBERTo-orig-tokenizer-1k | **0.5834** | <ins>0.5909</ins>| **0.6286** | **0.6166** |
|
| 177 |
+
| moBERTo-orig-tokenizer | 0.5674 | **0.6025** | 0.5876 | <ins>0.6140</ins>|
|
| 178 |
+
| moBERTo-1k | 0.5466 | 0.4791 | 0.5714 | 0.5857 |
|
| 179 |
+
| **moBERTo (this model)** | <ins>0.5827</ins>| 0.5606 | <ins>0.5905</ins>| 0.5777 |
|
| 180 |
+
|
| 181 |
+
### Classification (F1)
|
| 182 |
+
|
| 183 |
+
- **Docs**: document type classification (news, legal, academic, etc.)
|
| 184 |
+
- **Educ.**: educational content detection
|
| 185 |
+
|
| 186 |
+
| Model | Docs | Educ. | **Avg.** |
|
| 187 |
+
|-----------------------------|------------------|------------------|------------------|
|
| 188 |
+
| BERT-base | 0.8700 | 0.5690 | 0.7195 |
|
| 189 |
+
| BERTimbau-base | 0.8978 | 0.6382 | 0.7680 |
|
| 190 |
+
| ModernBERT-base | 0.8416 | 0.5730 | 0.7073 |
|
| 191 |
+
| NeoBERT-base | 0.8970 | 0.6266 | 0.7618 |
|
| 192 |
+
| Qwen3-0.6B-base | **0.9120** | 0.6289 | 0.7705 |
|
| 193 |
+
| moBERTo-orig-tokenizer-1k | 0.8942 | 0.6070 | 0.7506 |
|
| 194 |
+
| moBERTo-orig-tokenizer | 0.8962 | 0.6035 | 0.7499 |
|
| 195 |
+
| moBERTo-1k | 0.9024 | 0.6281 | 0.7653 |
|
| 196 |
+
| **moBERTo (this model)** | 0.9039 | <ins>0.6394</ins>| <ins>0.7717</ins>|
|
| 197 |
+
| NeoBERT-PT | 0.9030 | **0.6428** | **0.7729** |
|
| 198 |
+
| Qwen3-0.6B-PT | <ins>0.9070</ins>| 0.6311 | 0.7691 |
|
| 199 |
+
|
| 200 |
+
### NLU and NER (F1)
|
| 201 |
+
|
| 202 |
+
| Model | PLUE-PT | LeNER-Br | GLUE (English) |
|
| 203 |
+
|-----------------------------|------------------|------------------|------------------|
|
| 204 |
+
| BERT-base | 0.6423 | 0.8500 | <ins>0.7815</ins>|
|
| 205 |
+
| BERTimbau-base | 0.6800 | **0.9040** | 0.6772 |
|
| 206 |
+
| ModernBERT-base | 0.6420 | 0.8240 | **0.8301** |
|
| 207 |
+
| NeoBERT-base | 0.6654 | 0.8590 | 0.7430 |
|
| 208 |
+
| Qwen3-0.6B-base | 0.6343 | 0.7020 | 0.7260 |
|
| 209 |
+
| moBERTo-orig-tokenizer-1k | 0.6849 | 0.8371 | 0.7705 |
|
| 210 |
+
| moBERTo-orig-tokenizer | 0.6910 | 0.8587 | 0.7724 |
|
| 211 |
+
| moBERTo-1k | <ins>0.6959</ins>| 0.8710 | 0.7128 |
|
| 212 |
+
| **moBERTo (this model)** | **0.6980** | 0.8726 | 0.7354 |
|
| 213 |
+
| NeoBERT-PT | 0.6842 | <ins>0.8840</ins>| 0.6620 |
|
| 214 |
+
| Qwen3-0.6B-PT | 0.6632 | 0.7100 | 0.7050 |
|
| 215 |
+
|
| 216 |
+
> **Note on GLUE:** As expected from continued pretraining on Portuguese, English
|
| 217 |
+
> performance degrades. ModernBERT-base remains the strongest on GLUE (0.8301);
|
| 218 |
+
> this trade-off reflects the finite capacity of a base-sized model.
|
| 219 |
+
|
| 220 |
+
---
|
| 221 |
+
|
| 222 |
+
## Key Findings (from the paper's ablations)
|
| 223 |
+
|
| 224 |
+
1. **Continued pretraining > training from scratch.** Especially for long-context:
|
| 225 |
+
`moBERTo` achieves 0.5777 on MLDR@8192 vs. 0.1405 for a from-scratch baseline
|
| 226 |
+
trained on the same Portuguese budget. The original 2T-token ModernBERT pretraining
|
| 227 |
+
provides representations that transfer effectively even when continued pretraining
|
| 228 |
+
itself uses only 1,024-token sequences.
|
| 229 |
+
2. **Tokenizer adaptation helps token-level tasks but disrupts long context.** Moving
|
| 230 |
+
to a Portuguese tokenizer improves PLUE-PT and LeNER-Br but hurts MLDR@8192 (drops
|
| 231 |
+
by ~11 points without embedding transfer).
|
| 232 |
+
3. **SWM embedding transfer mitigates the long-context degradation.** By initializing
|
| 233 |
+
new Portuguese embeddings as combinations of the original subword embeddings, SWM
|
| 234 |
+
recovers most of the long-context performance lost by tokenizer adaptation alone.
|
| 235 |
+
4. **Long-context post-training yields the strongest reranker.** `moBERTo`
|
| 236 |
+
(this model) achieves the highest average reranking nDCG@10 (0.5255) and the best
|
| 237 |
+
PLUE-PT score (0.6980).
|
| 238 |
+
|
| 239 |
+
---
|
| 240 |
+
|
| 241 |
+
## Training Data
|
| 242 |
+
|
| 243 |
+
The pretraining corpus was curated from the Portuguese subset of
|
| 244 |
+
[FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) and further
|
| 245 |
+
filtered using the educational and STEM classifiers from ClassiCC-PT. The final
|
| 246 |
+
corpus comprises **~12 billion tokens**, roughly six times larger than BrWaC,
|
| 247 |
+
covering a broad range of domains and topics in Portuguese.
|
| 248 |
+
|
| 249 |
+
The training data has been publicly released alongside the model.
|
| 250 |
+
|
| 251 |
+
---
|
| 252 |
+
|
| 253 |
+
## Training Procedure
|
| 254 |
+
|
| 255 |
+
### Phase 1 — Continued pretraining (60B tokens at 1,024 context)
|
| 256 |
+
|
| 257 |
+
| Parameter | Value |
|
| 258 |
+
|--------------------------|-----------------------------|
|
| 259 |
+
| Training tokens | 60B (5 epochs over 12B) |
|
| 260 |
+
| Max sequence length | 1,024 |
|
| 261 |
+
| Batch size | 4,608 |
|
| 262 |
+
| Masking rate | 30% |
|
| 263 |
+
| Optimizer | StableAdamW |
|
| 264 |
+
| Learning rate | 5e-4 |
|
| 265 |
+
| Weight decay | 1e-5 |
|
| 266 |
+
| Dropout (attn output) | 0.1 |
|
| 267 |
+
| Dropout (other) | 0.0 |
|
| 268 |
+
| Precision | bfloat16 |
|
| 269 |
+
| RoPE base (global attn) | 160,000 |
|
| 270 |
+
| RoPE base (local attn) | 10,000 |
|
| 271 |
+
|
| 272 |
+
### Phase 2 — Long-context post-training (10B tokens at 8,192 context)
|
| 273 |
+
|
| 274 |
+
Same hyperparameters as Phase 1, except:
|
| 275 |
+
|
| 276 |
+
| Parameter | Value |
|
| 277 |
+
|--------------------------|-----------------------------|
|
| 278 |
+
| Training tokens | 10B |
|
| 279 |
+
| Max sequence length | 8,192 |
|
| 280 |
+
| Batch size | 576 |
|
| 281 |
+
|
| 282 |
+
---
|
| 283 |
+
|
| 284 |
+
## Related Models in the moBERTo Family
|
| 285 |
+
|
| 286 |
+
| Hugging Face Repo | Paper Name | Tokenizer | Long-ctx post-tr. |
|
| 287 |
+
|------------------------------------------------|-----------------------------|-----------|-------------------|
|
| 288 |
+
| `Tropic-AI/moBERTo-orig-tokenizer-1k` | moBERTo (orig. tok.) | Original | No |
|
| 289 |
+
| `Tropic-AI/moBERTo-orig-tokenizer` | moBERTo-8k (orig. tok.) | Original | Yes |
|
| 290 |
+
| `Tropic-AI/moBERTo-1k` | moBERTo-SWM (PT tok.) | PT (SWM) | No |
|
| 291 |
+
| **`Tropic-AI/moBERTo` *(this)* * | **moBERTo-SWM-8k (PT tok.)**| PT (SWM) | **Yes** |
|
| 292 |
+
|
| 293 |
+
---
|
| 294 |
+
|
| 295 |
+
## Citation
|