---
license: apache-2.0
language: [en]
tags: [masked-diffusion, mdlm, text-diffusion, quentin-dlm]
---

# mdlm-owt-diff1 — summary-conditioned MDLM (DIFF_1), 100k steps

**DIFF_1** from the `quentin-dlm` cascade: a masked-diffusion LM finetuned from
[`kuleshov-group/mdlm-owt`](https://huggingface.co/kuleshov-group/mdlm-owt) to
generate OpenWebText documents conditioned on a coarse summary prefix.

- Layout `[summary 256 | text 768]` @ L1024; prefix always revealed (never
  masked); masked-CE NELBO on the text region only. `time_conditioning=False`.
- 169.6M vendored Duo DiT backbone, GPT-2 tokenizer, vocab 50258
  (`[MASK]`=50257, pad=eos=50256).
- Data: [`EER6/openwebtext-coarse`](https://huggingface.co/datasets/EER6/openwebtext-coarse)
  (doc_idx >= 2048; first 2048 held out).
- Recipe: 100k steps, global batch 384 (8x GH200 DDP), lr 3e-4 cosine
  (warmup 500), AdamW(0.9, 0.95), wd 0, bf16, EMA 0.99.
- **These are the EMA weights of checkpoint-100000** (DiT backbone state_dict,
  same layout as mdlm-owt: `model.safetensors` at repo root).

Results / caveats: held-out val NELBO 2.996 (ppl 20.0) vs trash-prefix control
3.293 (26.9) — strong conditioning (samples reproduce ~44% of summary content
words, 5.5x the shuffled baseline). NOTE: the hot 100k finetune degraded
*sampling* fluency (gen-PPL ~207 @512 steps vs ~59 for the base model); see
RESULTS_MDLM_100K.md in the project repo for the full diagnosis (earlier
checkpoints sample better; remasking samplers recommended).

Load (project code): `duo_core.load_model("EER6/mdlm-owt-diff1", 1024, 50258, device)`
or as `--init_ckpt EER6/mdlm-owt-diff1` in `train/train_big_mdlm.py`.

Companion control: [`EER6/mdlm-owt-trash`](https://huggingface.co/EER6/mdlm-owt-trash).