dmbert-230m-fineweb2

A Dutch ModernBERT encoder pretrained from scratch with a masked-language-modelling objective on Dutch web text, using JAX/Flax on TPU. It has 22 layers, hidden size 1024 and ~230M parameters, and uses a Dutch tokenizer with a 32k vocabulary.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="yhavinga/dmbert-230m-fineweb2")
# Write <mask> directly after the preceding word, without a space.
for r in fill("Amsterdam is de<mask> van Nederland.", top_k=5):
    print(r["token_str"], round(r["score"], 3))

This is a pretrained base model: fine-tune it for classification, NER, retrieval etc.

Model details

Architecture	ModernBERT (alternating local/global attention, RoPE, GeGLU)
Layers / hidden / heads / intermediate	22 / 1024 / 16 / 1536
Parameters	230,110,592 (~230M). The Hub's parameter counter shows ~263M because the tied decoder weight is stored a second time in the file.
Attention	local window 128, global attention every 3rd layer
Max positions	8192
Tokenizer	yhavinga/dutch-llama-tokenizer (32,000 tokens + <mask>, padded to 32,128)

Training

• Data: Dutch portion (nld_Latn) of HuggingFaceFW/fineweb-2
• Steps: 2,000,000 (full schedule)
• Batch size: 8 per device, TPU v4-32
• Max sequence length: 1024
• Learning rate: 3e-5, 20,000 warmup steps, cosine decay to 1e-6
• Weight decay: 0.01
• Sequence packing and fp32 parameters (bfloat16 compute)
• Code: JAX/Flax, scripts/train_minimal.py in mbert-jax

Pretraining sequence format

This model was trained on FineWeb-2 (nld_Latn) with sequence packing and block-diagonal attention: documents were tokenized without special tokens, a </s> was appended after each document, and no <s> was used. </s> is therefore a familiar document separator, while <s> was not seen during pretraining.

Evaluation

The same tables appear on the card of every model in this family; the row of this model is in bold.

Tuned comparison (DUMB tasks, 5 seeds)

Test accuracy (%), mean ± standard deviation over 5 seeds, final-step model (no early stopping), on two tasks of the DUMB benchmark: DBRD (book-review sentiment, 2,224 test reviews) and SICK-NL (natural language inference, 4,906 test pairs).

Model	Params	Pretraining data	DBRD	SICK-NL
dmbert-144m-mc4	144M	mC4-nl-cleaned	95.69 ± 0.14	84.17 ± 0.30
dmbert-230m-mc4	230M	mC4-nl-cleaned	95.75 ± 0.25	84.25 ± 0.30
dmbert-230m-culturax	230M	CulturaX (nl), packed	95.98 ± 0.25	84.78 ± 0.24 (pairs as <s> A </s> B </s>: 84.83 ± 0.44)
dmbert-230m-fineweb2 (this model)	230M	FineWeb-2 (nl), packed	95.80 ± 0.42	84.51 ± 0.30 (pairs as <s> A </s> B </s>: 85.07 ± 0.25)
BERTje (GroNLP/bert-base-dutch-cased), our pipeline, same protocol, full DUMB grid	110M		92.92 ± 0.34	84.85 ± 0.20
BERTje, DUMB paper	110M		93.3	85.2
RobBERT-v2, DUMB paper	117M		93.2	84.6
RobBERT-2022, DUMB paper	119M		93.5	85.6

Protocol. For each model and task a small grid of epochs and learning rates was run with one seed (639808) and the best configuration on validation accuracy was repeated with the five DUMB seeds (639808, 107584, 251499, 370845, 890307). Batch size 32, AdamW with weight decay 0, linear learning-rate decay, gradient clipping 1.0, bf16 autocast, up to 512 tokens. Selected configurations:

Model	DBRD	SICK-NL
dmbert-144m-mc4	3 ep, LR 5e-5	3 ep, LR 5e-5
dmbert-230m-mc4	3 ep, LR 5e-5, dropout 0.1	10 ep, 30% warmup, LR 5e-5
dmbert-230m-culturax	2 ep, LR 3e-5	3 ep, LR 5e-5
dmbert-230m-fineweb2	2 ep, LR 3e-5	3 ep, LR 3e-5

Caveats. This is not the full DUMB protocol: our grid has 4 configurations per task instead of 36 (DBRD) or 64 (SICK-NL), the DBRD validation split (500 training reviews, seed 42) differs from DUMB's, and with 495 validation examples for SICK-NL the choice of configuration is noisy. The published rows are therefore only roughly comparable. The BERTje row was run through the same pipeline with the full DUMB grid and lands close to the published numbers. The dmbert-230m-mc4 SICK-NL result is from a configuration with warmup; a 5e-5 learning rate without warmup was unstable for that model.

Simple fixed-protocol comparison (no tuning)

Every model fine-tuned the same way: learning rate 2e-5, 3 epochs unless noted, batch 32, 10% warmup, weight decay 0.01, bf16, 3 seeds, test accuracy at the epoch with the best validation accuracy.

Model	DBRD	SICK-NL
dmbert-144m-mc4	94.53 ± 0.26	81.06 (1 seed)
dmbert-230m-mc4	96.00 ± 0.12	84.33 ± 0.51
dmbert-230m-culturax	95.67 ± 0.31	84.08 ± 0.40
dmbert-230m-fineweb2 (this model)	96.22 ± 0.09	83.91 ± 0.28
RobBERT-2022	not run	83.78 ± 0.17 (3 ep) / 84.37 ± 1.29 (10 ep)
RobBERT-2023-base	not run	85.28 ± 0.44 (3 ep) / 86.69 ± 0.29 (10 ep)

Sentence pairs. Tokenizer pairs come out as <s> A <s> B. For this model, which was pretrained with </s> as document separator and without <s>, building pairs as <s> A </s> B </s> improved SICK-NL by about 0.6 points (84.51 → 85.07, roughly two standard deviations). For single-sentence tasks no special handling was needed.

a_ids = tok(premise, add_special_tokens=False).input_ids
b_ids = tok(hypothesis, add_special_tokens=False).input_ids
input_ids = [tok.bos_token_id] + a_ids + [tok.eos_token_id] + b_ids + [tok.eos_token_id]

How to read this. On DBRD the dmbert models are 2.4 to 3.1 points above BERTje. On SICK-NL the best dmbert variants are on par with BERTje (within one standard deviation), while RobBERT-2023-base is clearly stronger in the untuned comparison; it is the better choice if NLI-style tasks are your target. Differences of a few tenths of a point between the dmbert models are within seed noise.

Related models

Same architecture family, different data: dmbert-144m-mc4, dmbert-230m-mc4, dmbert-230m-culturax, dmbert-230m-fineweb2.

Acknowledgements

Cloud TPUs were provided by Google's TPU Research Cloud (TRC).

Citation

@misc{dmbert_230m_fineweb2,
  title={dmbert-230m-fineweb2: a Dutch ModernBERT},
  author={Yeb Havinga},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/yhavinga/dmbert-230m-fineweb2}
}