---
license: apache-2.0
language:
- en
base_model: answerdotai/ModernBERT-large
pipeline_tag: fill-mask
library_name: transformers
tags:
- modernbert
- finance
- financial-nlp
- domain-adaptation
- dapt
- masked-lm
---

# FinModernBERT-large-DAPT

**ModernBERT-large domain-adapted to finance with 5.65B tokens of masked-language-model
pretraining.** This is the Stage-1 foundation of a pipeline whose final embedding model,
[FinModernBERT-embed-large-v1](https://huggingface.co/BatuhanECB/FinModernBERT-embed-large-v1),
**beats the 7B Fin-E5 on FinMTEB Summarization (+0.109) and STS (+0.010) at 1/18th the
size** — and this checkpoint is also the WiSE-FT interpolation anchor that made that
release possible. That final model is now on the **official
[FinMTEB leaderboard](https://huggingface.co/spaces/FinanceMTEB/FinMTEB)**: STS #2 of 18
(top embedding model on the board) and Summarization #2, see §2.

This card documents the full project journey — goal, every stage, every measured number —
with this checkpoint's role in it.

## 1. The project goal

Build the strongest *defensible* finance embedder on a single ModernBERT-large (395M),
evaluated on the full English [FinMTEB](https://github.com/yixuantt/FinMTEB) benchmark
(35 tasks / 7 types) against the published SOTA, **Fin-E5** (finance-adapted
e5-mistral-7B; [paper](https://arxiv.org/abs/2502.10990)). Target: win STS and
Summarization, make Retrieval competitive. "Defensible": every stage trains behind a hard
decontamination gate against all FinMTEB-EN eval sets.

## 2. Where the pipeline ended up (context for this checkpoint)

Final model results, full FinMTEB-EN 35 tasks (see the
[embed card](https://huggingface.co/BatuhanECB/FinModernBERT-embed-large-v1) for per-task detail):

| Task type | Final model (395M) | Fin-E5 (7B) | Δ |
|---|--:|--:|--:|
| **Summarization** | **0.588** | 0.480 | **+0.109** 👑 |
| **STS** | **0.444** | 0.434 | **+0.010** 👑 |
| Reranking | 0.961 | 0.990 | −0.028 |
| Clustering | 0.513 | 0.565 | −0.052 |
| Classification | 0.623 | 0.757 | −0.133 |
| PairClassification | 0.617 | 0.801 | −0.184 |
| Retrieval | 0.502 | 0.711 | −0.208 |
| **Overall (7-type mean)** | 0.607 | 0.677 | −0.070 |

**Official [FinMTEB leaderboard](https://huggingface.co/spaces/FinanceMTEB/FinMTEB)
placement** (EN board, 18 models, as of 2026-07-19): the final model ranks **STS #2**
(the **top embedding model** — only a lexical bag-of-words baseline scores higher; ahead
of Fin-E5, voyage-3-large and NV-Embed v2), **Summarization #2** (behind only
voyage-3-large, with the board's single best FINDsum score, 0.745), and #11 overall.
Only four models on the board hold two or more top-2 task-type finishes — voyage-3-large,
text-embedding-3-large, Fin-E5, and this pipeline's 395M open-weights model.

And the journey milestone table:

| Milestone | STS | Summarization | Retrieval |
|---|--:|--:|--:|
| Stage 0: untrained ModernBERT-large | 0.450 | 0.118 | 0.051 |
| **Stage 1: + 5.65B-token DAPT (this model)** | 0.446 | 0.120 | 0.056 |
| + chunk+mean-pool doc embedding (no training) | 0.446 | 0.305 | 0.056 |
| Stage 3: + 270k-pair contrastive (pure) | 0.371 | 0.625 | 0.569 |
| + WiSE-FT α=0.65 (released embed model) | 0.444 | 0.588 | 0.502 |
| Fin-E5 7B (the bar) | 0.434 | 0.480 | 0.711 |

## 3. This training run (from the actual logs)

| | |
|---|---|
| Base | `answerdotai/ModernBERT-large` (395M) |
| Objective | Masked LM, mask rate **0.30** (ModernBERT's own rate) |
| Corpus | **5,520,194 packed windows × 1,024 tokens ≈ 5.65B tokens** finance text (SEC filings, financial news, finance web), packed from a 185 GB raw pool after cleaning + decontamination |
| Epochs / steps | 1 epoch, **43,127 steps** (128 windows ≈ 131k tokens per step) |
| LR schedule | peak **5e-5**, ~2.3k-step warmup, cosine decay to ~0 |
| Precision / memory | bf16, gradient checkpointing, full finetune (no LoRA) |
| Hardware | 2× RTX 3090 (24 GB), DDP |
| Loss trajectory | 1.134 (init) → 1.042 @1k → 0.951 @3k → 0.907 @7k → **0.888 final** (1k-step window means); grad-norm stable ≈0.5–0.9 |

### Decontamination (before any training)

Every FinMTEB-English eval text was indexed with a word-shingle overlap index; corpus
records overlapping any eval set, eval-source datasets, and test/dev splits were dropped;
a hard `assert_clean` gate refuses to emit a corpus otherwise. Known-contaminated
lineages (FiQA/ConvFinQA etc.) excluded outright. The corpus is **not released** (source
licensing does not permit redistribution).

## 4. What DAPT does — and doesn't do (measured honestly)

Scored on FinMTEB-EN (15-task subset: 2 STS + 3 Summ + 10 Retrieval), mean pooling:

| task type | stock ModernBERT-large | **this model** | Δ | Fin-Retriever-base | Fin-E5 |
|---|--:|--:|--:|--:|--:|
| STS | 0.450 | 0.446 | −0.005 | 0.294 | 0.434 |
| Summarization | 0.118 | 0.120 | +0.002 | 0.281 | 0.480 |
| Retrieval | 0.051 | 0.056 | +0.005 | 0.402 | 0.711 |

**DAPT alone barely moves zero-shot embedding scores** — an MLM objective doesn't reshape
sentence geometry. Its measured value in this project came from two places:

1. **Foundation for contrastive training.** Stage 3 (270,238 synthetic + open pairs,
   positive-aware hard negatives mined with e5-large-v2, CachedMNRL scale 20 + CoSENT,
   2 epochs on 2×3090) was trained *from this checkpoint* and reached
   Retrieval 0.569 / Summarization 0.625 (pure).
2. **WiSE-FT interpolation anchor.** Pure contrastive training collapsed STS
   0.450 → 0.371, failing the pre-registered release gate (≥0.434 = Fin-E5 parity).
   The fix: `w = α·contrastive + (1−α)·this_checkpoint`:

| α (contrastive share) | STS | Summarization | Retrieval | gate ≥0.434 |
|--:|--:|--:|--:|---|
| 1.00 (pure contrastive) | 0.371 | 0.625 | 0.569 | ✗ |
| 0.80 | 0.402 | — | — | ✗ |
| **0.65 (released)** | **0.444** | **0.588** | **0.502** | **✓** |
| 0.50 | 0.465 | 0.457 | 0.327 | ✓ (retrieval collapses) |
| 0.30 | 0.446 | — | — | ✓ |
| 0.00 (this model) | 0.446 | 0.305* | 0.056 | ✓ |

\* with chunk+mean-pool doc-side embedding. Note α=0.5 beats *both endpoints* on STS —
the classic WiSE-FT effect. A finance-DAPT anchor made the interpolation *safe*: general
knowledge returns without leaving the finance domain.

## 5. The rest of the pipeline (what was built on top)

- **Stage 2 — synthetic pairs:** an open-weights 35B MoE teacher (AWQ/vLLM) over SEC
  chunks → 115,238 filtered finance pairs incl. **16,696 doc↔summary positives + 16,700
  mismatched score-0 summary negatives** (the lever Fin-E5 never trained — its own
  adaptation *regressed* Summarization 0.528→0.480) + 1,452 term→definitions; blended
  with open general data to **270,238 pairs (59% finance)**.
- **Long-doc discovery:** chunk+mean-pool doc embedding = Summarization 0.120→0.305 with
  zero training (FNS docs average ~70k+ tokens; head-truncation was destroying signal).
- **Stage 3 — contrastive:** 2 epochs / 1,976 steps, effective 256 anchors/step,
  255 in-batch negatives/device, lr 2e-5 cosine, bf16, ~8h on the same 2×3090 pair.

## 6. Usage

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tok = AutoTokenizer.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")
model = AutoModelForMaskedLM.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")

text = "The company's [MASK] margin expanded 150 basis points year-over-year."
inputs = tok(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
mask_idx = (inputs.input_ids == tok.mask_token_id).nonzero()[0, 1]
print(tok.decode(logits[0, mask_idx].topk(5).indices))
```

As a fine-tuning base (recommended use):

```python
from transformers import AutoModel
encoder = AutoModel.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")  # drops MLM head
```

Native context 8,192 tokens (ModernBERT); DAPT windows were 1,024 tokens. English,
finance-domain, Apache-2.0.

## 7. Limitations

- Not an embedding model out of the box — zero-shot sentence geometry ≈ the untrained
  base (see §4). Use the [embed model](https://huggingface.co/BatuhanECB/FinModernBERT-embed-large-v1)
  for embeddings, or fine-tune this.
- English-only, finance-domain MLM; general-domain MLM quality may have drifted.
- Corpus not released (licensing); decontamination applies to FinMTEB-EN specifically.

## Related

- **[FinModernBERT-embed-large-v1](https://huggingface.co/BatuhanECB/FinModernBERT-embed-large-v1)** —
  the final contrastive embedding model built from this checkpoint.
- [FinModernBERT-pairs-sec-synthetic-v1](https://huggingface.co/datasets/BatuhanECB/FinModernBERT-pairs-sec-synthetic-v1) —
  the released 115k SEC-synthetic contrastive pair set used in Stage 3.
- [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) — base model.
- [FinMTEB](https://github.com/yixuantt/FinMTEB) — benchmark ([paper](https://arxiv.org/abs/2502.10990)).