cvm-bertimbau-sentence-transformer

Fine-tuned BERTimbau sentence transformer for dense retrieval over Brazilian public company filings (CVM ITR/DFP). Part of the CVM Filing Intelligence System.

Training

Parameter	Value
Base model	neuralmind/bert-base-portuguese-cased
Loss	MultipleNegativesRankingLoss
Training pairs	14,500 (adjacent same-section chunk pairs from 686 CVM filings)
Epochs	10
Batch size	16 (effective 64 with gradient accumulation ×4)
Mixed precision	fp16
Max sequence length	256 tokens
Hardware	NVIDIA RTX A1000 (6 GB VRAM), ~2 hours
Initial loss	2.201 (step 50)
Final loss	0.115 (step 2,270)

Training data: 97,138 management commentary chunks from 49 B3 large-cap companies (Petrobras, Vale, Itaú, Bradesco, Ambev, etc.), 2022–2025. Pairs are adjacent paragraphs within the same section of the same filing.

Retrieval Results

Evaluated on 94 synthetic queries over the 97,138-chunk corpus (dense-only configuration):

Metric	Value
Recall@5	0.057
Recall@10	0.071
MRR	0.100
NDCG@10	0.063

Note: The model underperforms BM25 on query–document retrieval because it was fine-tuned with doc–doc contrastive pairs. Query–doc performance improves significantly with GPL (Generative Pseudo Labeling) fine-tuning using synthetic query–chunk pairs.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("conderafael/cvm-bertimbau-sentence-transformer")

# Encode a single passage
embeddings = model.encode(["Receita líquida cresceu 12% no trimestre"])

# Encode a batch
texts = [
    "O EBITDA ajustado atingiu R$ 4,2 bilhões no 3T24.",
    "A Companhia mantém posição conservadora de hedge cambial.",
]
embeddings = model.encode(texts, normalize_embeddings=True)
print(embeddings.shape)  # (2, 768)

Limitations

• Trained on Portuguese-language financial filings only; degrades on other domains.
• Max sequence length 256 tokens; longer passages are truncated.
• Query-time performance is below doc-time performance due to training objective mismatch (doc–doc pairs vs. query–doc retrieval).