|
|
--- |
|
|
license: apache-2.0 |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
datasets: |
|
|
- HuggingFaceFW/fineweb-edu |
|
|
base_model: |
|
|
- GSAI-ML/LLaDA-8B-Base |
|
|
tags: |
|
|
- XDLM |
|
|
- LLaDA |
|
|
--- |
|
|
|
|
|
# [LLaDA-XDLM-8B-Base](https://arxiv.org/pdf/2602.01362) |
|
|
|
|
|
This repository contains the checkpoint of 600 training steps for ***continual pretraining LLaDA with XDLM***. |
|
|
|
|
|
***LLaDA-XDLM with sampling budget of 32.*** |
|
|
Evaluation of adapting LLaDA-8B to our XDLM formulation (LLaDA-XDLM): (a) LLaDA-XDLM consistently out-performs baselines across diverse benchmarks with 32 sampling steps; (b) Improvements are particularly pronounced in code generation (MBPP), where the |
|
|
model substantially reduces generation failures. |
|
|
|
|
|
<div align=center> |
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/65aa76b1cb5b4fb08ecb087c/oPbIv32EgvA1BbCqd2r6E.png" width="80%"> |
|
|
</div> |
|
|
|
|
|
|
|
|
For details and usage see [Code](https://github.com/MzeroMiko/LLaDA-XDLM) |
|
|
|
|
|
## TODO: |
|
|
- [ ] update `model_card` to support standard huggingface transformers's usage. |
|
|
|
|
|
<!-- ## Updates --> |
