| --- |
| license: apache-2.0 |
| language: [en] |
| tags: [masked-diffusion, mdlm, text-diffusion, quentin-dlm] |
| --- |
| |
| # mdlm-owt-diff1 — summary-conditioned MDLM (DIFF_1), 100k steps |
| |
| **DIFF_1** from the `quentin-dlm` cascade: a masked-diffusion LM finetuned from |
| [`kuleshov-group/mdlm-owt`](https://huggingface.co/kuleshov-group/mdlm-owt) to |
| generate OpenWebText documents conditioned on a coarse summary prefix. |
| |
| - Layout `[summary 256 | text 768]` @ L1024; prefix always revealed (never |
| masked); masked-CE NELBO on the text region only. `time_conditioning=False`. |
| - 169.6M vendored Duo DiT backbone, GPT-2 tokenizer, vocab 50258 |
| (`[MASK]`=50257, pad=eos=50256). |
| - Data: [`EER6/openwebtext-coarse`](https://huggingface.co/datasets/EER6/openwebtext-coarse) |
| (doc_idx >= 2048; first 2048 held out). |
| - Recipe: 100k steps, global batch 384 (8x GH200 DDP), lr 3e-4 cosine |
| (warmup 500), AdamW(0.9, 0.95), wd 0, bf16, EMA 0.99. |
| - **These are the EMA weights of checkpoint-100000** (DiT backbone state_dict, |
| same layout as mdlm-owt: `model.safetensors` at repo root). |
|
|
| Results / caveats: held-out val NELBO 2.996 (ppl 20.0) vs trash-prefix control |
| 3.293 (26.9) — strong conditioning (samples reproduce ~44% of summary content |
| words, 5.5x the shuffled baseline). NOTE: the hot 100k finetune degraded |
| *sampling* fluency (gen-PPL ~207 @512 steps vs ~59 for the base model); see |
| RESULTS_MDLM_100K.md in the project repo for the full diagnosis (earlier |
| checkpoints sample better; remasking samplers recommended). |
|
|
| Load (project code): `duo_core.load_model("EER6/mdlm-owt-diff1", 1024, 50258, device)` |
| or as `--init_ckpt EER6/mdlm-owt-diff1` in `train/train_big_mdlm.py`. |
|
|
| Companion control: [`EER6/mdlm-owt-trash`](https://huggingface.co/EER6/mdlm-owt-trash). |
|
|
