Entropy-Valley · LLaDA-8B English→German

📄 Paper (arXiv:2608.22274) • 💻 Code • 🤗 Models & Dataset • 📚 Datasets

Official English→German LoRA adapter for Length-Adaptive Decoding for Masked Diffusion Machine Translation (EMNLP 2026 Main Conference). It turns GSAI-ML/LLaDA-8B-Base into a masked-diffusion MT system for En→De.

En→De is the paper's matched-scale check, not its headline direction. It is included as a typologically more distant test; the gains are smaller here and the paper documents why. See Limitations before deploying. For the strongly supported directions see En→Zh and Zh→En.

The adapter is not the method. Entropy-Valley (EV) is a training-free, decoding-time length selector, implemented in ladit/decoding/length_adaptive.py. This adapter is the fixed backbone that EV decodes with — the same weights serve the length-oracle, fixed-ratio, and EV conditions. Only the canvas length handed to the decoder changes.

How Entropy-Valley works

A masked diffusion LM fills a fixed-size canvas: it must be told how many target slots to produce before denoising begins, and there is no autoregressive EOS to stop it. EV asks the frozen backbone which canvas it is most prepared to fill — one all-mask forward pass per candidate length, scored by mean predictive entropy over the first $L-1$ slots (the last is reserved for EOS), then decode the minimum:

$L^{\star} = \arg\min_{L \in \mathcal{C}(\mathbf{x})} \bar{H}(L), \qquad \bar{H}(L) = \frac{1}{L-1}\sum_{i=1}^{L-1} H\big(p_\theta(y_i \mid \mathbf{x}, \texttt{[MASK]}^L)\big)$

Quick facts


Base model	`GSAI-ML/LLaDA-8B-Base` (8.02B, masked diffusion)
Adapter	LoRA `r=64`, `α=128`, dropout `0.05` on `q/k/v/o_proj` + `ff_proj/up_proj/ff_out`
Training data	200k WMT19 de-en pairs (Entropy-Valley-Datasets, config `ende`), 3 epochs, bf16, 8×H20
Decoding	MED schedule, $T{=}32$ steps, EOS truncation
EV candidate grid	$\mathcal{R} = {1.50, 1.60, 1.70, 1.80, 1.90}$, fixed for the direction
Prompt template	`Translate English to German.\n\nEnglish: {src}\nGerman:`

Results

WMT22 En→De ($N{=}2{,}037$), 32-step MED decoding, mean over three independent training runs.

Length method	COMET-22	sacreBLEU
Fixed ratio 1.8	0.7170	20.73
Entropy-Valley	0.7240	21.55
Length oracle †	0.7382	22.55

† Decodes at the reference target length — an upper bound, not a deployable method. EV closes 33.0% of the gap between the two.

Significance tests, cross-backbone results, and all ablations are in the paper. This repository releases one of the three training runs behind the means above.

Usage

git clone https://github.com/Entropy-Valley/Entropy-Valley.git && cd Entropy-Valley
pip install -e .

import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

from ladit.data.mt_dataset import set_lang_pair
from ladit.decoding.length_adaptive import entropy_valley_probe, set_mask_token_id as set_ev_mask
from ladit.decoding.translate import translate_single, set_mask_token_id as set_dec_mask

BASE, ADAPTER = "GSAI-ML/LLaDA-8B-Base", "YanZhanPKU/Entropy-Valley-LLaDA-8B-En2De"

tokenizer = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(BASE, trust_remote_code=True,
                                             torch_dtype=torch.bfloat16).to("cuda")
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload().eval()

mask_tid = getattr(AutoConfig.from_pretrained(BASE, trust_remote_code=True), "mask_token_id", 126336)
set_ev_mask(mask_tid); set_dec_mask(mask_tid)
set_lang_pair("en-de")

src = "You can come back any time as our chat service window is open."
n_src = len(tokenizer.encode(src, add_special_tokens=False))
candidates = sorted({max(1, int(n_src * r)) + 1 for r in (1.50, 1.60, 1.70, 1.80, 1.90)})

L_star = entropy_valley_probe(model, tokenizer, src, candidates)["best_length"]
out = translate_single(model, tokenizer, src, target_length=L_star,
                       num_steps=32, schedule_name="med")
print(L_star, out["translation"])

Reproduce the full WMT22 evaluation (decodes all three length methods and scores BLEU + COMET-22):

python scripts/decode_eval.py \
    --model_path /path/to/LLaDA-8B-Base \
    --lora_path  YanZhanPKU/Entropy-Valley-LLaDA-8B-En2De \
    --input_file data/wmt22_ende_test.jsonl \
    --output_dir eval_results/ende_ev \
    --num_examples 2037 --num_steps 32 --schedule med \
    --methods "oracle,ratio_1.8,entropy_valley" \
    --candidate_ratios "1.50,1.60,1.70,1.80,1.90" \
    --lang_pair en-de --device cuda

The 1.50–1.90 grid above is the protocol-aligned grid behind the reported numbers. Earlier pilot runs used a narrower grid — do not mix evaluation artifacts from the two when computing gap closure.

Limitations

Read these before using this direction:

Sentence-level evidence is weaker than on En↔Zh. The mean over three runs is positive, but the paired bootstrap on this direction is not significant.
The step budget matters. EV leads at small $T$, but at $T{\ge}64$ the fixed ratio slightly overtakes it. The reported result uses the default $T{=}32$.
The remaining gap is not mainly a length problem. A matched-data autoregressive baseline sits well above even the LLaDA length oracle here, so supplying the reference length does not close it. EV is scoped to canvas selection within a fixed backbone.
The adapter is tied to LLaDA-8B-Base and to WMT-style news/web text; high-risk domains should retain human review. EV operates only at inference time and inherits the backbone's safety and bias profile.

Citation

@inproceedings{zhan2026lengthadaptive,
  title         = {Length-Adaptive Decoding for Masked Diffusion Machine Translation},
  author        = {Zhan, Yan and Hou, Mengkai and Zhang, Wanting and Gao, Zhijun},
  booktitle     = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year          = {2026},
  eprint        = {2608.22274},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.22274}
}