LLaDA-XDLM-8B-Base

This repository contains the checkpoint of 600 training steps for continual pretraining LLaDA with XDLM.

LLaDA-XDLM with sampling budget of 32. Evaluation of adapting LLaDA-8B to our XDLM formulation (LLaDA-XDLM): (a) LLaDA-XDLM consistently out-performs baselines across diverse benchmarks with 32 sampling steps; (b) Improvements are particularly pronounced in code generation (MBPP), where the model substantially reduces generation failures.

For details and usage see Code

TODO:

• update model_card to support standard huggingface transformers's usage.