matol-16/hcdlm

Hierarchical Continuous Diffusion Language Models

H-CoBit checkpoints from Hierarchical Continuous Diffusion Language Models (Mathias Ollu, Nikos Komodakis). Code, evaluation and training: github.com/matol-16/HCDLM.

H-CoBit is CoBit with a second modality: each token's bits are extended with the bits of its cluster (n = 16 clusters of pretrained token embeddings), and both modalities are diffused and denoised jointly by a single model.

Folder	Data	Steps	Bits per token	Noise schedule
hcobit_lm1b	LM1B	600k (of a 1M-step cosine schedule)	15 + 4	entropy_profile_{info,cluster}.pt
hcobit_owt	OpenWebText	750k (of a 1M-step cosine schedule)	16 + 4	entropy_profile_{info,cluster}.pt
hcobit_tinygsm	TinyGSM, evaluated on GSM8K	250k	16 + 4	entropy_*_{info,cluster}.pt, sigma_data.json

model.pt holds the EMA weights. Each folder also holds the per-modality noise schedule it is sampled with.

Usage

git clone https://github.com/matol-16/HCDLM && cd HCDLM/hcobit
python scripts/download_checkpoints.py          # these checkpoints and CoBit-S (CoBit's release)
python evaluate.py genppl lm1b hcobit_pms --nfe 512 --gamma 0.3
python evaluate.py gsm8k tinygsm hcobit --nfe 512 --gamma 0.38

Citation

Coming with the arXiv release.