| --- |
| language: |
| - pt |
| license: apache-2.0 |
| library_name: transformers |
| tags: |
| - modernbert |
| - portuguese |
| - encoder |
| - masked-lm |
| - long-context |
| - moberto |
| pipeline_tag: fill-mask |
| base_model: answerdotai/ModernBERT-base |
| datasets: |
| - HuggingFaceFW/fineweb-2 |
| metrics: |
| - nDCG@10 |
| - F1 |
| model-index: |
| - name: moBERTo-orig-tokenizer |
| results: |
| - task: |
| type: text-retrieval |
| name: Reranking |
| dataset: |
| name: QUATI |
| type: quati |
| metrics: |
| - type: nDCG@10 |
| value: 0.5231 |
| - task: |
| type: text-retrieval |
| name: Reranking |
| dataset: |
| name: mMARCO-PT |
| type: mmarco-pt |
| metrics: |
| - type: nDCG@10 |
| value: 0.5089 |
| - task: |
| type: text-retrieval |
| name: Reranking |
| dataset: |
| name: Robust04-PT |
| type: robust04-pt |
| metrics: |
| - type: nDCG@10 |
| value: 0.4516 |
| - task: |
| type: text-retrieval |
| name: Long-Context Reranking |
| dataset: |
| name: MLDR (PT) |
| type: mldr |
| metrics: |
| - type: nDCG@10 |
| value: 0.6140 |
| name: nDCG@10 at 8192 tokens |
| - task: |
| type: token-classification |
| name: Named Entity Recognition |
| dataset: |
| name: LeNER-Br |
| type: lener-br |
| metrics: |
| - type: F1 |
| value: 0.8587 |
| - task: |
| type: text-classification |
| name: Natural Language Understanding |
| dataset: |
| name: PLUE-PT |
| type: plue-pt |
| metrics: |
| - type: F1 |
| value: 0.6910 |
| --- |
| |
| # moBERTo-orig-tokenizer |
|
|
| > **Paper name:** This model is referred to as **`moBERTo-8k (orig. tok.)`** in the |
| > moBERTo paper. It is the variant that retains the **original ModernBERT tokenizer**, |
| > followed by long-context post-training. |
|
|
| `moBERTo-orig-tokenizer` is a Portuguese adaptation of |
| [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base), obtained through |
| continued pretraining on a curated 12-billion-token Portuguese corpus (60B training |
| tokens, 5 epochs) followed by a long-context post-training phase at 8,192-token context. |
|
|
| Unlike the flagship [`moBERTo`](https://huggingface.co/Tropic-AI/moBERTo) variant, |
| this model **keeps the original ModernBERT tokenizer** rather than adopting a |
| Portuguese-optimized one. This makes it particularly strong for long-context |
| retrieval tasks (where it achieves the **best results in the moBERTo family** at |
| 8,192 tokens on MLDR), at the cost of slightly weaker token-level performance |
| (NER and NLU). |
|
|
| The model preserves all architectural advances of ModernBERT: rotary positional |
| embeddings (RoPE), alternating local–global attention, FlashAttention, and unpadding, |
| with a native context window of **8,192 tokens**. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | Attribute | Value | |
| |------------------------|------------------------------------------------------| |
| | Architecture | ModernBERT (encoder-only) | |
| | Base checkpoint | `answerdotai/ModernBERT-base` | |
| | Parameters | ~150M | |
| | Max context length | 8,192 tokens | |
| | Tokenizer | **Original ModernBERT tokenizer (unchanged)** | |
| | Embedding init | Inherited from ModernBERT-base | |
| | Pretraining tokens | 60B (5 epochs over 12B-token corpus) | |
| | Long-context post-tr. | 10B tokens at 8,192-token context | |
| | Training corpus | FineWeb-2 (PT subset) filtered with ClassiCC-PT | |
| | Framework | Composer | |
| | Precision | bfloat16 | |
| | License | Apache 2.0 | |
|
|
| --- |
|
|
| ## Quick Start |
|
|
| ```python |
| long_text = "..." # documento longo em português |
| inputs = tokenizer(long_text, max_length=8192, truncation=True, return_tensors="pt") |
| with torch.no_grad(): |
| outputs = model(**inputs) |
| ``` |
|
|
| ### Recommended for downstream tasks |
|
|
| This model is best used as a backbone for fine-tuning on: |
|
|
| - **Long-document retrieval** (its strongest use case) |
| - **Cross-encoder reranking** (information retrieval) |
| - **Document classification** |
| - **Named entity recognition** |
| - **Natural language inference / semantic textual similarity** |
|
|
| --- |
|
|
| ## Evaluation Results |
|
|
| All metrics are reported on Portuguese benchmarks. Best results are in **bold**; |
| second-best are <ins>underlined</ins>. |
|
|
| ### Information Retrieval (Reranking, nDCG@10) |
|
|
| Cross-encoder reranking, fine-tuned on mMARCO-PT triples. |
|
|
| | Model | QUATI | mMARCO | Robust04 | **Avg.** | |
| |----------------------------------------|------------------|------------------|------------------|------------------| |
| | BERT-base | 0.2846 | 0.4050 | 0.2389 | 0.3095 | |
| | BERTimbau-base | 0.4870 | 0.5005 | 0.4138 | 0.4671 | |
| | ModernBERT-base | 0.3779 | 0.4799 | 0.2988 | 0.3855 | |
| | NeoBERT-base | 0.4000 | 0.4698 | 0.3117 | 0.3938 | |
| | Qwen3-0.6B-base | 0.4248 | 0.5065 | 0.2994 | 0.4102 | |
| | moBERTo-orig-tokenizer-1k | 0.5383 | 0.5109 | 0.4510 | 0.5001 | |
| | **moBERTo-orig-tokenizer (this model)**| 0.5231 | 0.5089 | 0.4516 | 0.4945 | |
| | moBERTo-1k | <ins>0.5410</ins>| **0.5169** | <ins>0.4782</ins>| <ins>0.5120</ins>| |
| | moBERTo | **0.5609** | <ins>0.5147</ins>| **0.5010** | **0.5255** | |
|
|
| ### Long-Context Retrieval (MLDR, nDCG@10) |
|
|
| | Model | 512 | 2,048 | 4,096 | 8,192 | |
| |----------------------------------------|------------------|------------------|------------------|------------------| |
| | ModernBERT-base | 0.4054 | 0.4206 | 0.3015 | 0.2867 | |
| | NeoBERT-base | 0.4746 | 0.5149 | 0.4676 | -- | |
| | Qwen3-0.6B-base | 0.3560 | 0.4023 | 0.4241 | 0.5351 | |
| | moBERTo-orig-tokenizer-1k | **0.5834** | <ins>0.5909</ins>| **0.6286** | **0.6166** | |
| | **moBERTo-orig-tokenizer (this model)**| 0.5674 | **0.6025** | 0.5876 | <ins>0.6140</ins>| |
| | moBERTo-1k | 0.5466 | 0.4791 | 0.5714 | 0.5857 | |
| | moBERTo | <ins>0.5827</ins>| 0.5606 | <ins>0.5905</ins>| 0.5777 | |
|
|
| ### Classification (F1) |
|
|
| - **Docs**: document type classification (news, legal, academic, etc.) |
| - **Educ.**: educational content detection |
|
|
| | Model | Docs | Educ. | **Avg.** | |
| |----------------------------------------|------------------|------------------|------------------| |
| | BERT-base | 0.8700 | 0.5690 | 0.7195 | |
| | BERTimbau-base | 0.8978 | 0.6382 | 0.7680 | |
| | ModernBERT-base | 0.8416 | 0.5730 | 0.7073 | |
| | NeoBERT-base | 0.8970 | 0.6266 | 0.7618 | |
| | Qwen3-0.6B-base | **0.9120** | 0.6289 | 0.7705 | |
| | moBERTo-orig-tokenizer-1k | 0.8942 | 0.6070 | 0.7506 | |
| | **moBERTo-orig-tokenizer (this model)**| 0.8962 | 0.6035 | 0.7499 | |
| | moBERTo-1k | 0.9024 | 0.6281 | 0.7653 | |
| | moBERTo | 0.9039 | <ins>0.6394</ins>| <ins>0.7717</ins>| |
| | NeoBERT-PT | 0.9030 | **0.6428** | **0.7729** | |
| | Qwen3-0.6B-PT | <ins>0.9070</ins>| 0.6311 | 0.7691 | |
|
|
| ### NLU and NER (F1) |
|
|
| | Model | PLUE-PT | LeNER-Br | GLUE (English) | |
| |----------------------------------------|------------------|------------------|------------------| |
| | BERT-base | 0.6423 | 0.8500 | <ins>0.7815</ins>| |
| | BERTimbau-base | 0.6800 | **0.9040** | 0.6772 | |
| | ModernBERT-base | 0.6420 | 0.8240 | **0.8301** | |
| | NeoBERT-base | 0.6654 | 0.8590 | 0.7430 | |
| | Qwen3-0.6B-base | 0.6343 | 0.7020 | 0.7260 | |
| | moBERTo-orig-tokenizer-1k | 0.6849 | 0.8371 | 0.7705 | |
| | **moBERTo-orig-tokenizer (this model)**| 0.6910 | 0.8587 | 0.7724 | |
| | moBERTo-1k | <ins>0.6959</ins>| 0.8710 | 0.7128 | |
| | moBERTo | **0.6980** | 0.8726 | 0.7354 | |
| | NeoBERT-PT | 0.6842 | <ins>0.8840</ins>| 0.6620 | |
| | Qwen3-0.6B-PT | 0.6632 | 0.7100 | 0.7050 | |
|
|
|
|
| ## Training Data |
|
|
| The pretraining corpus was curated from the Portuguese subset of |
| [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) and further |
| filtered using the educational and STEM classifiers from ClassiCC-PT. The final |
| corpus comprises **~12 billion tokens**, roughly six times larger than BrWaC, |
| covering a broad range of domains and topics in Portuguese. |
|
|
| The training data has been publicly released alongside the model. |
|
|
| --- |
|
|
| ## Training Procedure |
|
|
| ### Phase 1 — Continued pretraining (60B tokens at 1,024 context) |
|
|
| | Parameter | Value | |
| |--------------------------|-----------------------------| |
| | Training tokens | 60B (5 epochs over 12B) | |
| | Max sequence length | 1,024 | |
| | Batch size | 4,608 | |
| | Masking rate | 30% | |
| | Optimizer | StableAdamW | |
| | Learning rate | 5e-4 | |
| | Weight decay | 1e-5 | |
| | Dropout (attn output) | 0.1 | |
| | Dropout (other) | 0.0 | |
| | Precision | bfloat16 | |
| | RoPE base (global attn) | 160,000 | |
| | RoPE base (local attn) | 10,000 | |
|
|
| ### Phase 2 — Long-context post-training (10B tokens at 8,192 context) |
|
|
| Same hyperparameters as Phase 1, except: |
|
|
| | Parameter | Value | |
| |--------------------------|-----------------------------| |
| | Training tokens | 10B | |
| | Max sequence length | 8,192 | |
| | Batch size | 576 | |
|
|
|
|
|
|
| ## Related Models in the moBERTo Family |
|
|
| | Hugging Face Repo | Paper Name | Tokenizer | Long-ctx post-tr. | |
| |--------------------------------------------------|-----------------------------|-----------|-------------------| |
| | **`Tropic-AI/moBERTo-orig-tokenizer` *(this)*** | **moBERTo-8k (orig. tok.)** | Original | **Yes** | |
| | `Tropic-AI/moBERTo` | moBERTo-SWM-8k (PT tok.) | PT (SWM) | Yes | |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{laitz2026mobertomodernencoderportuguese, |
| title={moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT}, |
| author={Thiago Laitz and Thales Sales Almeida and João Guilherme Alves Santos and Giovana Kerche Bonás}, |
| year={2026}, |
| eprint={2606.22722}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2606.22722}, |
| } |
| ``` |
