Entropy-Valley · LLaDA-8B Chinese→English

📄 Paper (arXiv:2608.22274) • 💻 Code • 🤗 Models & Dataset • 📚 Datasets

Official Chinese→English LoRA adapter for Length-Adaptive Decoding for Masked Diffusion Machine Translation (EMNLP 2026 Main Conference). It turns GSAI-ML/LLaDA-8B-Base into a masked-diffusion MT system for Zh→En.

The adapter is not the method. Entropy-Valley (EV) is a training-free, decoding-time length selector, implemented in ladit/decoding/length_adaptive.py. This adapter is the fixed backbone that EV decodes with — the same weights serve the length-oracle, fixed-ratio, and EV conditions. Only the canvas length handed to the decoder changes.

How Entropy-Valley works

A masked diffusion LM fills a fixed-size canvas: it must be told how many target slots to produce before denoising begins, and there is no autoregressive EOS to stop it. EV asks the frozen backbone which canvas it is most prepared to fill — one all-mask forward pass per candidate length, scored by mean predictive entropy over the first $L-1$ slots (the last is reserved for EOS), then decode the minimum:

$L^{\star} = \arg\min_{L \in \mathcal{C}(\mathbf{x})} \bar{H}(L), \qquad \bar{H}(L) = \frac{1}{L-1}\sum_{i=1}^{L-1} H\big(p_\theta(y_i \mid \mathbf{x}, \texttt{[MASK]}^L)\big)$

Quick facts


Base model	`GSAI-ML/LLaDA-8B-Base` (8.02B, masked diffusion)
Adapter	LoRA `r=64`, `α=128`, dropout `0.05` on `q/k/v/o_proj` + `ff_proj/up_proj/ff_out`
Training data	200k WMT19 zh-en pairs (Entropy-Valley-Datasets, config `enzh`, roles swapped), 3 epochs, bf16, 8×H20
Decoding	MED schedule, $T{=}32$ steps, EOS truncation
EV candidate grid	$\mathcal{R} = {1.00, 1.10, 1.20, 1.30, 1.40}$, fixed for the direction
Prompt template	`Translate Chinese to English.\n\nChinese: {src}\nEnglish:`

Results

WMT22 Zh→En ($N{=}2{,}037$), 32-step MED decoding, mean over three independent training runs.

Length method	COMET-22	sacreBLEU
Fixed ratio 1.2	0.8266	23.65
Entropy-Valley	0.8431	25.28
Length oracle †	0.8519	27.93

† Decodes at the reference target length — an upper bound, not a deployable method. EV closes 65.3% of the gap between the two.

This is the direction with the strongest human-evaluation support in the paper. Significance tests, the expert study, comparisons against DAEDAL and CAL, cross-backbone results, and all ablations are in the paper. This repository releases one of the three training runs behind the means above.

Usage

git clone https://github.com/Entropy-Valley/Entropy-Valley.git && cd Entropy-Valley
pip install -e .

import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

from ladit.data.mt_dataset import set_lang_pair
from ladit.decoding.length_adaptive import entropy_valley_probe, set_mask_token_id as set_ev_mask
from ladit.decoding.translate import translate_single, set_mask_token_id as set_dec_mask

BASE, ADAPTER = "GSAI-ML/LLaDA-8B-Base", "YanZhanPKU/Entropy-Valley-LLaDA-8B-Zh2En"

tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(BASE, trust_remote_code=True,
                                             torch_dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload().eval()

mask_tid = getattr(AutoConfig.from_pretrained(BASE, trust_remote_code=True), "mask_token_id", 126336)
set_ev_mask(mask_tid); set_dec_mask(mask_tid)
set_lang_pair("zh-en")

src = "很抱歉，您点的餐可能会晚到一会。"
n_src = len(tokenizer.encode(src, add_special_tokens=False))
candidates = sorted({max(1, int(n_src * r)) + 1 for r in (1.00, 1.10, 1.20, 1.30, 1.40)})

L_star = entropy_valley_probe(model, tokenizer, src, candidates)["best_length"]
out = translate_single(model, tokenizer, src, target_length=L_star,
                       num_steps=32, schedule_name="med")
print(L_star, out["translation"])

Reproduce the full WMT22 evaluation (decodes all three length methods and scores BLEU + COMET-22):

python scripts/decode_eval.py \
    --model_path /path/to/LLaDA-8B-Base \
    --lora_path  YanZhanPKU/Entropy-Valley-LLaDA-8B-Zh2En \
    --input_file data/wmt22_enzh_test.jsonl \
    --output_dir eval_results/zhen_ev \
    --num_examples 2037 --num_steps 32 --schedule med \
    --methods "oracle,ratio_1.2,entropy_valley" \
    --candidate_ratios "1.00,1.10,1.20,1.30,1.40" \
    --lang_pair zh-en --device cuda

Zh→En reuses wmt22_enzh_test.jsonl — --lang_pair zh-en swaps which key is source and which is target.

Limitations

EV can only choose among the fixed candidate grid. A candidate-width control in the paper shows this direction benefits from a wider window than the deployed default; the grid is kept fixed for protocol consistency.
The adapter is tied to LLaDA-8B-Base and to WMT-style news/web text; high-risk domains should retain human review.
EV operates only at inference time and inherits the safety and bias profile of the backbone and the training data.

Citation

@inproceedings{zhan2026lengthadaptive,
  title         = {Length-Adaptive Decoding for Masked Diffusion Machine Translation},
  author        = {Zhan, Yan and Hou, Mengkai and Zhang, Wanting and Gao, Zhijun},
  booktitle     = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year          = {2026},
  eprint        = {2608.22274},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.22274}
}