c3-mdlm-110m-v4

110M-parameter masked-diffusion language model (MDLM objective, bidirectional transformer: 12 layers, d_model 768, 12 heads, d_ff 3072, vocab 50258 incl. mask id 50257, max length 256) trained for the C3 project (calibration of diffusion language models). This is the "v4" training run, run_name: train110m-v4.

• Training: 169,055 steps, ~9.995B tokens, seq_len 256, batch 16 x grad_accum 4, lr 3e-4 (cosine to 10%), warmup 2000, EMA 0.9999, seed 20260917, config hash 0a0a0a4b02d99a8a. Full config is in train_summary.json; loss curves in history.jsonl.
• Training data: FineWeb-Edu (10BT sample), ODC-By 1.0 licence, https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu . Cite: Penedo et al., "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale", 2024. The data is not redistributed here.
• Files: final.pt (final weights, PyTorch pickle, loaded by the C3 project code; not converted to safetensors), history.jsonl, train_summary.json, checkpoint.json. The full training checkpoint with optimizer state (checkpoint.pt, 2.1 GB) is intentionally not uploaded.
• Related: fine-tuned probe model ANLP-project/c3-ft110m-v4; traces in dataset ANLP-project/c3-traces.