thiagolaitz commited on
Commit
6f800c6
·
verified ·
1 Parent(s): 9d6cf11

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +295 -0
README.md ADDED
@@ -0,0 +1,295 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - pt
4
+ license: apache-2.0
5
+ library_name: transformers
6
+ tags:
7
+ - modernbert
8
+ - portuguese
9
+ - encoder
10
+ - masked-lm
11
+ - long-context
12
+ - moberto
13
+ pipeline_tag: fill-mask
14
+ base_model: answerdotai/ModernBERT-base
15
+ datasets:
16
+ - HuggingFaceFW/fineweb-2
17
+ metrics:
18
+ - nDCG@10
19
+ - F1
20
+ model-index:
21
+ - name: moBERTo
22
+ results:
23
+ - task:
24
+ type: text-retrieval
25
+ name: Reranking
26
+ dataset:
27
+ name: QUATI
28
+ type: quati
29
+ metrics:
30
+ - type: nDCG@10
31
+ value: 0.5609
32
+ - task:
33
+ type: text-retrieval
34
+ name: Reranking
35
+ dataset:
36
+ name: mMARCO-PT
37
+ type: mmarco-pt
38
+ metrics:
39
+ - type: nDCG@10
40
+ value: 0.5147
41
+ - task:
42
+ type: text-retrieval
43
+ name: Reranking
44
+ dataset:
45
+ name: Robust04-PT
46
+ type: robust04-pt
47
+ metrics:
48
+ - type: nDCG@10
49
+ value: 0.5010
50
+ - task:
51
+ type: text-retrieval
52
+ name: Long-Context Reranking
53
+ dataset:
54
+ name: MLDR (PT)
55
+ type: mldr
56
+ metrics:
57
+ - type: nDCG@10
58
+ value: 0.5777
59
+ name: nDCG@10 at 8192 tokens
60
+ - task:
61
+ type: token-classification
62
+ name: Named Entity Recognition
63
+ dataset:
64
+ name: LeNER-Br
65
+ type: lener-br
66
+ metrics:
67
+ - type: F1
68
+ value: 0.8726
69
+ - task:
70
+ type: text-classification
71
+ name: Natural Language Understanding
72
+ dataset:
73
+ name: PLUE-PT
74
+ type: plue-pt
75
+ metrics:
76
+ - type: F1
77
+ value: 0.6980
78
+ ---
79
+
80
+ # moBERTo
81
+
82
+ > **Paper name:** This model is referred to as **`moBERTo-SWM-8k (PT tok.)`** in the
83
+ > moBERTo paper. It is the **best-performing variant** of the moBERTo family, achieving
84
+ > the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks
85
+ > and the best PLUE-PT score.
86
+
87
+ `moBERTo` is a Portuguese adaptation of [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base),
88
+ obtained through continued pretraining on a curated 12-billion-token Portuguese corpus
89
+ (60B training tokens, 5 epochs) followed by a long-context post-training phase at
90
+ 8,192-token context.
91
+
92
+ It combines four adaptation strategies:
93
+
94
+ 1. **Continued pretraining** from the original ModernBERT-base checkpoint, preserving
95
+ the long-context capabilities learned during the original 2T-token English pretraining.
96
+ 2. **Portuguese tokenizer** with vocabulary optimized for Portuguese text.
97
+ 3. **Subword Matching (SWM) embedding transfer**, which initializes each new Portuguese
98
+ token's embedding as a combination of the original ModernBERT subword embeddings,
99
+ keeping the model close to its pretrained representation space.
100
+ 4. **Long-context post-training** at 8,192 tokens for an additional 10B tokens.
101
+
102
+ The model preserves all architectural advances of ModernBERT: rotary positional
103
+ embeddings (RoPE), alternating local–global attention, FlashAttention, and unpadding,
104
+ with a native context window of **8,192 tokens**.
105
+
106
+ ---
107
+
108
+ ## Model Details
109
+
110
+ | Attribute | Value |
111
+ |------------------------|------------------------------------------------------|
112
+ | Architecture | ModernBERT (encoder-only) |
113
+ | Base checkpoint | `answerdotai/ModernBERT-base` |
114
+ | Parameters | ~150M |
115
+ | Max context length | 8,192 tokens |
116
+ | Tokenizer | Portuguese (custom vocabulary) |
117
+ | Embedding init | Subword Matching (SWM) transfer |
118
+ | Pretraining tokens | 60B (5 epochs over 12B-token corpus) |
119
+ | Long-context post-tr. | 10B tokens at 8,192-token context |
120
+ | Training corpus | FineWeb-2 (PT subset) filtered with ClassiCC-PT |
121
+ | Framework | Composer |
122
+ | Precision | bfloat16 |
123
+ | License | Apache 2.0 |
124
+
125
+ ---
126
+
127
+ ## Quick Start
128
+
129
+ ```python
130
+ long_text = "..." # documento longo em português
131
+ inputs = tokenizer(long_text, max_length=8192, truncation=True, return_tensors="pt")
132
+ with torch.no_grad():
133
+ outputs = model(**inputs)
134
+ ```
135
+
136
+ ### Recommended for downstream tasks
137
+
138
+ This model is best used as a backbone for fine-tuning on:
139
+
140
+ - **Cross-encoder reranking** (information retrieval)
141
+ - **Document classification**
142
+ - **Named entity recognition**
143
+ - **Natural language inference / semantic textual similarity**
144
+ - **Long-document retrieval** (up to 8,192 tokens)
145
+
146
+ ---
147
+
148
+ ## Evaluation Results
149
+
150
+ All metrics are reported on Portuguese benchmarks. Best results are in **bold**;
151
+ second-best are <ins>underlined</ins>.
152
+
153
+ ### Information Retrieval (Reranking, nDCG@10)
154
+
155
+ Cross-encoder reranking, fine-tuned on mMARCO-PT triples.
156
+
157
+ | Model | QUATI | mMARCO | Robust04 | **Avg.** |
158
+ |-----------------------------|------------------|------------------|------------------|------------------|
159
+ | BERT-base | 0.2846 | 0.4050 | 0.2389 | 0.3095 |
160
+ | BERTimbau-base | 0.4870 | 0.5005 | 0.4138 | 0.4671 |
161
+ | ModernBERT-base | 0.3779 | 0.4799 | 0.2988 | 0.3855 |
162
+ | NeoBERT-base | 0.4000 | 0.4698 | 0.3117 | 0.3938 |
163
+ | Qwen3-0.6B-base | 0.4248 | 0.5065 | 0.2994 | 0.4102 |
164
+ | moBERTo-orig-tokenizer-1k | 0.5383 | 0.5109 | 0.4510 | 0.5001 |
165
+ | moBERTo-orig-tokenizer | 0.5231 | 0.5089 | 0.4516 | 0.4945 |
166
+ | moBERTo-1k | <ins>0.5410</ins>| **0.5169** | <ins>0.4782</ins>| <ins>0.5120</ins>|
167
+ | **moBERTo (this model)** | **0.5609** | <ins>0.5147</ins>| **0.5010** | **0.5255** |
168
+
169
+ ### Long-Context Retrieval (MLDR, nDCG@10)
170
+
171
+ | Model | 512 | 2,048 | 4,096 | 8,192 |
172
+ |-----------------------------|------------------|------------------|------------------|------------------|
173
+ | ModernBERT-base | 0.4054 | 0.4206 | 0.3015 | 0.2867 |
174
+ | NeoBERT-base | 0.4746 | 0.5149 | 0.4676 | -- |
175
+ | Qwen3-0.6B-base | 0.3560 | 0.4023 | 0.4241 | 0.5351 |
176
+ | moBERTo-orig-tokenizer-1k | **0.5834** | <ins>0.5909</ins>| **0.6286** | **0.6166** |
177
+ | moBERTo-orig-tokenizer | 0.5674 | **0.6025** | 0.5876 | <ins>0.6140</ins>|
178
+ | moBERTo-1k | 0.5466 | 0.4791 | 0.5714 | 0.5857 |
179
+ | **moBERTo (this model)** | <ins>0.5827</ins>| 0.5606 | <ins>0.5905</ins>| 0.5777 |
180
+
181
+ ### Classification (F1)
182
+
183
+ - **Docs**: document type classification (news, legal, academic, etc.)
184
+ - **Educ.**: educational content detection
185
+
186
+ | Model | Docs | Educ. | **Avg.** |
187
+ |-----------------------------|------------------|------------------|------------------|
188
+ | BERT-base | 0.8700 | 0.5690 | 0.7195 |
189
+ | BERTimbau-base | 0.8978 | 0.6382 | 0.7680 |
190
+ | ModernBERT-base | 0.8416 | 0.5730 | 0.7073 |
191
+ | NeoBERT-base | 0.8970 | 0.6266 | 0.7618 |
192
+ | Qwen3-0.6B-base | **0.9120** | 0.6289 | 0.7705 |
193
+ | moBERTo-orig-tokenizer-1k | 0.8942 | 0.6070 | 0.7506 |
194
+ | moBERTo-orig-tokenizer | 0.8962 | 0.6035 | 0.7499 |
195
+ | moBERTo-1k | 0.9024 | 0.6281 | 0.7653 |
196
+ | **moBERTo (this model)** | 0.9039 | <ins>0.6394</ins>| <ins>0.7717</ins>|
197
+ | NeoBERT-PT | 0.9030 | **0.6428** | **0.7729** |
198
+ | Qwen3-0.6B-PT | <ins>0.9070</ins>| 0.6311 | 0.7691 |
199
+
200
+ ### NLU and NER (F1)
201
+
202
+ | Model | PLUE-PT | LeNER-Br | GLUE (English) |
203
+ |-----------------------------|------------------|------------------|------------------|
204
+ | BERT-base | 0.6423 | 0.8500 | <ins>0.7815</ins>|
205
+ | BERTimbau-base | 0.6800 | **0.9040** | 0.6772 |
206
+ | ModernBERT-base | 0.6420 | 0.8240 | **0.8301** |
207
+ | NeoBERT-base | 0.6654 | 0.8590 | 0.7430 |
208
+ | Qwen3-0.6B-base | 0.6343 | 0.7020 | 0.7260 |
209
+ | moBERTo-orig-tokenizer-1k | 0.6849 | 0.8371 | 0.7705 |
210
+ | moBERTo-orig-tokenizer | 0.6910 | 0.8587 | 0.7724 |
211
+ | moBERTo-1k | <ins>0.6959</ins>| 0.8710 | 0.7128 |
212
+ | **moBERTo (this model)** | **0.6980** | 0.8726 | 0.7354 |
213
+ | NeoBERT-PT | 0.6842 | <ins>0.8840</ins>| 0.6620 |
214
+ | Qwen3-0.6B-PT | 0.6632 | 0.7100 | 0.7050 |
215
+
216
+ > **Note on GLUE:** As expected from continued pretraining on Portuguese, English
217
+ > performance degrades. ModernBERT-base remains the strongest on GLUE (0.8301);
218
+ > this trade-off reflects the finite capacity of a base-sized model.
219
+
220
+ ---
221
+
222
+ ## Key Findings (from the paper's ablations)
223
+
224
+ 1. **Continued pretraining > training from scratch.** Especially for long-context:
225
+ `moBERTo` achieves 0.5777 on MLDR@8192 vs. 0.1405 for a from-scratch baseline
226
+ trained on the same Portuguese budget. The original 2T-token ModernBERT pretraining
227
+ provides representations that transfer effectively even when continued pretraining
228
+ itself uses only 1,024-token sequences.
229
+ 2. **Tokenizer adaptation helps token-level tasks but disrupts long context.** Moving
230
+ to a Portuguese tokenizer improves PLUE-PT and LeNER-Br but hurts MLDR@8192 (drops
231
+ by ~11 points without embedding transfer).
232
+ 3. **SWM embedding transfer mitigates the long-context degradation.** By initializing
233
+ new Portuguese embeddings as combinations of the original subword embeddings, SWM
234
+ recovers most of the long-context performance lost by tokenizer adaptation alone.
235
+ 4. **Long-context post-training yields the strongest reranker.** `moBERTo`
236
+ (this model) achieves the highest average reranking nDCG@10 (0.5255) and the best
237
+ PLUE-PT score (0.6980).
238
+
239
+ ---
240
+
241
+ ## Training Data
242
+
243
+ The pretraining corpus was curated from the Portuguese subset of
244
+ [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) and further
245
+ filtered using the educational and STEM classifiers from ClassiCC-PT. The final
246
+ corpus comprises **~12 billion tokens**, roughly six times larger than BrWaC,
247
+ covering a broad range of domains and topics in Portuguese.
248
+
249
+ The training data has been publicly released alongside the model.
250
+
251
+ ---
252
+
253
+ ## Training Procedure
254
+
255
+ ### Phase 1 — Continued pretraining (60B tokens at 1,024 context)
256
+
257
+ | Parameter | Value |
258
+ |--------------------------|-----------------------------|
259
+ | Training tokens | 60B (5 epochs over 12B) |
260
+ | Max sequence length | 1,024 |
261
+ | Batch size | 4,608 |
262
+ | Masking rate | 30% |
263
+ | Optimizer | StableAdamW |
264
+ | Learning rate | 5e-4 |
265
+ | Weight decay | 1e-5 |
266
+ | Dropout (attn output) | 0.1 |
267
+ | Dropout (other) | 0.0 |
268
+ | Precision | bfloat16 |
269
+ | RoPE base (global attn) | 160,000 |
270
+ | RoPE base (local attn) | 10,000 |
271
+
272
+ ### Phase 2 — Long-context post-training (10B tokens at 8,192 context)
273
+
274
+ Same hyperparameters as Phase 1, except:
275
+
276
+ | Parameter | Value |
277
+ |--------------------------|-----------------------------|
278
+ | Training tokens | 10B |
279
+ | Max sequence length | 8,192 |
280
+ | Batch size | 576 |
281
+
282
+ ---
283
+
284
+ ## Related Models in the moBERTo Family
285
+
286
+ | Hugging Face Repo | Paper Name | Tokenizer | Long-ctx post-tr. |
287
+ |------------------------------------------------|-----------------------------|-----------|-------------------|
288
+ | `Tropic-AI/moBERTo-orig-tokenizer-1k` | moBERTo (orig. tok.) | Original | No |
289
+ | `Tropic-AI/moBERTo-orig-tokenizer` | moBERTo-8k (orig. tok.) | Original | Yes |
290
+ | `Tropic-AI/moBERTo-1k` | moBERTo-SWM (PT tok.) | PT (SWM) | No |
291
+ | **`Tropic-AI/moBERTo` *(this)* * | **moBERTo-SWM-8k (PT tok.)**| PT (SWM) | **Yes** |
292
+
293
+ ---
294
+
295
+ ## Citation