dmbert-144m-mc4

A Dutch ModernBERT encoder pretrained from scratch with a masked-language-modelling objective on Dutch web text, using JAX/Flax on TPU. It has 22 layers, hidden size 512 and 143,645,056 (~144M) parameters, and uses a Dutch tokenizer with a 32k vocabulary.

Usage

from transformers import pipeline

fill = pipeline("fill-mask", model="yhavinga/dmbert-144m-mc4")
# Write <mask> directly after the preceding word, without a space.
for r in fill("Amsterdam is de<mask> van Nederland.", top_k=5):
    print(r["token_str"], round(r["score"], 3))

This is a pretrained base model: fine-tune it for classification, NER, retrieval etc.

Model details

Architecture	ModernBERT (alternating local/global attention, RoPE, GeGLU)
Layers / hidden / heads / intermediate	22 / 512 / 8 / 3072
Parameters	143,645,056 (~144M). The Hub's parameter counter shows ~160M because the tied decoder weight is stored a second time in the file.
Attention	local window 128, global attention every 3rd layer
Max positions	8192
Tokenizer	yhavinga/dutch-llama-tokenizer (32,000 tokens + <mask>, padded to 32,128)

Training

• Data: yhavinga/mc4_nl_cleaned (config full)
• Steps: 1,350,000 (checkpoint of a run scheduled for 2,000,000 steps; the run crashed at step 1,350,040)
• Batch size: 8 per device, TPU v4-32
• Learning rate: 1.41e-4, 20,000 warmup steps, cosine decay to 1e-6
• Weight decay: 0.01
• Code: JAX/Flax, scripts/train_minimal.py in mbert-jax

Pretraining sequence format

This model was trained on mC4-nl-cleaned without sequence packing: one document per sequence, truncated or padded to 1024 tokens. According to the training code of that period (not verified in the run logs), every sequence starts with <s> and contains no </s>. The standard tokenizer pair format <s> A <s> B is therefore close to what the model saw.

Evaluation

The same tables appear on the card of every model in this family; the row of this model is in bold.

Tuned comparison (DUMB tasks, 5 seeds)

Test accuracy (%), mean ± standard deviation over 5 seeds, final-step model (no early stopping), on two tasks of the DUMB benchmark: DBRD (book-review sentiment, 2,224 test reviews) and SICK-NL (natural language inference, 4,906 test pairs).

Model	Params	Pretraining data	DBRD	SICK-NL
dmbert-144m-mc4 (this model)	144M	mC4-nl-cleaned	95.69 ± 0.14	84.17 ± 0.30
dmbert-230m-mc4	230M	mC4-nl-cleaned	95.75 ± 0.25	84.25 ± 0.30
dmbert-230m-culturax	230M	CulturaX (nl), packed	95.98 ± 0.25	84.78 ± 0.24 (pairs as <s> A </s> B </s>: 84.83 ± 0.44)
dmbert-230m-fineweb2	230M	FineWeb-2 (nl), packed	95.80 ± 0.42	84.51 ± 0.30 (pairs as <s> A </s> B </s>: 85.07 ± 0.25)
BERTje (GroNLP/bert-base-dutch-cased), our pipeline, same protocol, full DUMB grid	110M		92.92 ± 0.34	84.85 ± 0.20
BERTje, DUMB paper	110M		93.3	85.2
RobBERT-v2, DUMB paper	117M		93.2	84.6
RobBERT-2022, DUMB paper	119M		93.5	85.6

Protocol. For each model and task a small grid of epochs and learning rates was run with one seed (639808) and the best configuration on validation accuracy was repeated with the five DUMB seeds (639808, 107584, 251499, 370845, 890307). Batch size 32, AdamW with weight decay 0, linear learning-rate decay, gradient clipping 1.0, bf16 autocast, up to 512 tokens. Selected configurations:

Model	DBRD	SICK-NL
dmbert-144m-mc4	3 ep, LR 5e-5	3 ep, LR 5e-5
dmbert-230m-mc4	3 ep, LR 5e-5, dropout 0.1	10 ep, 30% warmup, LR 5e-5
dmbert-230m-culturax	2 ep, LR 3e-5	3 ep, LR 5e-5
dmbert-230m-fineweb2	2 ep, LR 3e-5	3 ep, LR 3e-5

Caveats. This is not the full DUMB protocol: our grid has 4 configurations per task instead of 36 (DBRD) or 64 (SICK-NL), the DBRD validation split (500 training reviews, seed 42) differs from DUMB's, and with 495 validation examples for SICK-NL the choice of configuration is noisy. The published rows are therefore only roughly comparable. The BERTje row was run through the same pipeline with the full DUMB grid and lands close to the published numbers. The dmbert-230m-mc4 SICK-NL result is from a configuration with warmup; a 5e-5 learning rate without warmup was unstable for that model.

Simple fixed-protocol comparison (no tuning)

Every model fine-tuned the same way: learning rate 2e-5, 3 epochs unless noted, batch 32, 10% warmup, weight decay 0.01, bf16, 3 seeds, test accuracy at the epoch with the best validation accuracy.

Model	DBRD	SICK-NL
dmbert-144m-mc4 (this model)	94.53 ± 0.26	81.06 (1 seed)
dmbert-230m-mc4	96.00 ± 0.12	84.33 ± 0.51
dmbert-230m-culturax	95.67 ± 0.31	84.08 ± 0.40
dmbert-230m-fineweb2	96.22 ± 0.09	83.91 ± 0.28
RobBERT-2022	not run	83.78 ± 0.17 (3 ep) / 84.37 ± 1.29 (10 ep)
RobBERT-2023-base	not run	85.28 ± 0.44 (3 ep) / 86.69 ± 0.29 (10 ep)

How to read this. On DBRD the dmbert models are 2.4 to 3.1 points above BERTje. On SICK-NL the best dmbert variants are on par with BERTje (within one standard deviation), while RobBERT-2023-base is clearly stronger in the untuned comparison; it is the better choice if NLI-style tasks are your target. Differences of a few tenths of a point between the dmbert models are within seed noise.

Related models

Same architecture family, different data: dmbert-144m-mc4, dmbert-230m-mc4, dmbert-230m-culturax, dmbert-230m-fineweb2.

Acknowledgements

Cloud TPUs were provided by Google's TPU Research Cloud (TRC).

Citation

@misc{dmbert_144m_mc4,
  title={dmbert-144m-mc4: a Dutch ModernBERT},
  author={Yeb Havinga},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/yhavinga/dmbert-144m-mc4}
}