---
language: en
license: mit
library_name: transformers
tags:
  - bert
  - masked-language-modeling
  - mlm
datasets:
  - lucadiliello/bookcorpusopen
  - wikimedia/wikipedia
---

# BERT-MLM

BERT-base (110M params) trained from scratch with the **classic masked language modeling (MLM)** objective from [Devlin et al., 2018](https://arxiv.org/abs/1810.04805).

This model is part of a paired experiment comparing classic BERT MLM training against modern diffusion language model (DLM) training. See [AntonXue/BERT-DLM](https://huggingface.co/AntonXue/BERT-DLM) for the counterpart.

## Training Objective

Standard BERT MLM: 15% of tokens selected as targets, with 80/10/10 corruption (80% replaced with [MASK], 10% random token, 10% unchanged). Cross-entropy loss on target positions only.

## Dataset

- **BookCorpusOpen** () — ~17K books
- **English Wikipedia** (, 20231101.en) — ~6.4M articles
- **Split:** 95/5 train/eval on raw documents, then tokenized and packed into 512-token sequences (no padding)
- **Train sequences:** 10,784,085
- **Total train tokens:** 5.52B

## Training Configuration

| Parameter | Value |
|---|---|
| Architecture |  (fresh random init) |
| Parameters | 109.5M |
| Sequence length | 512 |
| Global batch size | 256 (128 per GPU x 2 GPUs) |
| Training steps | 100,000 |
| Tokens seen | ~13.1B |
| Optimizer | AdamW |
| Learning rate | 1e-4 |
| LR schedule | Constant with warmup |
| Warmup steps | 500 |
| Adam betas | (0.9, 0.999) |
| Weight decay | 0.01 |
| Max grad norm | 1.0 |
| Precision | bf16 |
| Hardware | 2x NVIDIA H100 NVL |

## Usage



## Code

Training code: [github.com/AntonXue/dBERT](https://github.com/AntonXue/dBERT)