# Raw PDF Table Extracts > Generated from `pdftotext -layout` outputs under `extracted_pdf_text/remote_text`. These blocks preserve raw PDF layout and may need manual cleanup before citation. ## reference/DMax- Aggressive Parallel Decoding for dLLMs ### Table 1: Comparison with the original model and different baselines. For our DMax-Math model, ```text Table 1: Comparison with the original model and different baselines. For our DMax-Math model, we set the decoding threshold to 0.5; for the DMax-Coder model, we set it to 0.65. In addition to TPF, TPS, and accuracy, we also report the AUP score to provide a more comprehensive evaluation of parallel decoding performance. All evaluations are under zero-shot and a batch size of 1. Benchmark Method TPF ↑ TPS ↑ Acc. ↑ AUP Score ↑ Math & Reasoning Benchmarks LLaDA-2.0-mini 2.04 512 92.6% 340 Hierarchical Decoding 2.44 577 91.6% 357 GSM8K dParallel SFT 2.79 721 92.3% 395 Uniform Diffusion Training 2.26 493 68.7% 0 DMax-Math 5.48 1258 92.1% 557 LLaDA-2.0-mini 2.58 626 75.8% 257 Hierarchical Decoding 3.01 669 73.0% 268 MATH500 dParallel SFT 3.42 823 75.8% 310 Uniform Diffusion Training 2.43 530 33.6% 0 DMax-Math 5.94 1286 75.4% 507 LLaDA-2.0-mini 3.01 755 91.4% 363 Hierarchical Decoding 3.40 787 90.6% 382 Minerva-Algebra dParallel SFT 3.91 943 91.4% 430 Uniform Diffusion Training 2.55 551 42.7% 0 DMax-Math 7.03 1492 91.5% 658 LLaDA-2.0-mini 2.03 512 92.8% 354 Hierarchical Decoding 2.43 528 92.5% 366 ASDIV dParallel SFT 2.72 663 93.0% 459 Uniform Diffusion Training 2.51 515 80.8% 0 DMax-Math 5.62 1172 92.5% 556 Code Generation Benchmarks LLaDA-2.0-mini 4.38 1044 84.2% 369 Hierarchical Decoding 4.67 1014 81.1% 379 HumanEval-Instruct dParallel SFT 5.12 1229 76.8% 394 Uniform Diffusion Training 2.93 628 15.2% 0 DMax-Coder 7.36 1557 83.5% 637 LLaDA-2.0-mini 2.71 662 80.6% 276 Hierarchical Decoding 2.88 685 76.6% 241 MBPP-Instruct dParallel SFT 3.66 880 74.7% 273 Uniform Diffusion Training 2.84 608 23.4% 0 DMax-Coder 5.86 1264 79.2% 482 Training Data. We construct all training data through self-distillation. Specifically, we take prompts from public datasets and use LLaDA-2.0-mini to generate responses as training targets. For math, prompts are collected from GSM8K trainset [20], PRM12K [44], a subset of Numina-Math [40], and a subset of OpenThoughts [26]. For code, prompts are drawn from a subset of OpenCodeInstruct [2]. Responses are generated with a confidence threshold of 0.95, a block size of 32, and a maximum generation length of 2048 tokens. We discard incomplete generations that do not finish within the length budget. This yields 0.7M math samples and 1.0M code samples. Notably, we do not use any external high-quality responses, all supervision is obtained from the model’s own generations. Evaluation Details. We evaluate our method on multiple benchmarks. For mathematical reasoning, we use GSM8K [20], MATH500 [44], Minerva-Algebra [29], and ASDIV [55], and prompt the model to produce chain-of-thought [80] reasoning. For code generation, we use the instruction versions of HumanEval [13] and MBPP [5]. All evaluations are conducted with the dInFer [54] framework on 2 H200 GPUs using tensor parallelism. Besides TPF, TPS, and accuracy, we also report AUP Score [62] to measure parallel decoding performance. The generation length for all benchmarks is 2048. Baselines. We compare our method against four baselines in terms of both decoding efficiency and generation accuracy: (1) LLaDA-2.0-mini, the base model, evaluated with its default confidence- threshold-based parallel decoding strategy using a threshold of 0.95; (2) Hierarchical Decoding, an advanced inference strategy that improves parallel decoding via a divide-and-conquer procedure [61]. The low threshold is set as 0.2; (3) dParallel-SFT, for which we use the LLaDA-2.0-mini-CAP model [10], where the certainty-forcing loss proposed in dParallel [16] is incorporated into large-scale 7 (a) Acc-TPF Trade-off on GSM8K (b) Acc-TPF Trade-off on MATH500 (c) Acc-TPF Trade-off on HumanEval (d) Acc-TPF Trade-off on MBPP Figure 4: Comparison of accuracy-TPF trade-off curves between original LLaDA-2.0-mini model and our method. We present curves on GSM8K, MATH500, HumanEval and MBPP benchmarks. ``` ### Table 2: Our proposed new paradigm also improves the model’s accuracy at low parallelism. ```text Table 2: Our proposed new paradigm also improves the model’s accuracy at low parallelism. Benchmarks LLaDA-2.0-mini DMax TPF ↑ Acc. ↑ TPF ↑ Acc. ↑ GSM8K 2.04 92.6% 3.54 (+1.50) 93.4% (+0.8%) MATH500 2.58 75.8% 3.45 (+0.87) 78.0% (+2.2%) Minerva-Algebra 3.01 91.4% 4.96 (+1.95) 93.6% (+2.2%) ASDIV 2.03 92.8% 3.10 (+1.07) 93.5% (+0.7%) HumanEval-Instruct 4.38 84.2% 4.58 (+0.20) 87.2% (+3.0%) MBPP-Instruct 2.71 80.6% 3.58 (+0.87) 83.4% (+2.8%) supervised fine-tuning to improve decoding parallelism; and (4) Uniform Diffusion Training, which continues training the base model using the conventional UDLM objective. In addition to masked noisy sequences, this baseline also replaces tokens with random vocabulary samples to construct uniformly corrupted noisy sequences, while keeping all other training settings identical to those of DMax. During inference, it updates all tokens within a block at every step until convergence. 4.2 Experimental Results Aggressive Parallelism While Preserving Accuracy. As shown in Table 1, compared with the original LLaDA-2.0-mini, our method substantially increases decoding parallelism, improving the average TPF from 2.8 to 6.2 while preserving the original accuracy. In contrast, the other baselines provide only limited gains in parallel decoding. This advantage is further reflected in the AUP Score, where DMax consistently outperforms both the original model and all baselines by a large margin. These results demonstrate that our paradigm enables a much stronger parallel decoding capability than conventional MDLMs. Moreover, on two H200 GPUs, our model achieves a practical inference throughput of over 1000 tokens per second. On-Policy Training as the Cornerstone. Table 1 also compares our method with conventional uniform diffusion training. The latter neither improves decoding speed nor preserves model quality, instead causing a noticeable performance drop. We find that this failure stems from the large mismatch between the randomly sampled noisy sequences used in training and the model’s actual decoding trajectories at inference time. Consequently, the model struggles to revise erroneous predictions while unnecessarily perturbing correct ones, resulting in unstable oscillations within each block. By contrast, our on-policy training samples noisy sequences from the model’s own outputs, effectively bridging this train–inference gap and substantially improving self-revision under parallel decoding. Superior Efficiency–Performance Trade-off. Figure 4 compares the accuracy–TPF trade-off curves of our method and the original model on GSM8K, MATH500, HumanEval, and MBPP. As TPF increases, the original model suffers a sharp accuracy drop, whereas our method maintains stable performance. For instance, on MATH500, at around 6.5 TPF, our method still retains over 71.6% accuracy, while the original model falls to 15.2%. The gap is even larger on code benchmarks: on MBPP, at a similar TPF, our method achieves 79.2%, whereas the original model drops to only 2.3%. This superior trade-off stems from the self-revision capability of our paradigm, which effectively mitigates error accumulation under aggressive parallel decoding. 8 ``` ### Table 3: Ablation on different training and inference strategies with different decoding parallelism. ```text Table 3: Ablation on different training and inference strategies with different decoding parallelism. Train Inference τdec = 0.95 τdec = 0.50 τdec = 0.0 On-Policy Contiguous Hybrid TPF ↑ Acc. ↑ TPF ↑ Acc. ↑ TPF ↑ Acc. ↑ Rollout Prefix Embedding 2.04 92.6% 4.47 78.0% 7.86 0.9% ✓ ✓ 1.04 0.0% 1.73 0.0% 5.39 0.0% ✓ 2.95 92.6% 5.14 90.1% 5.89 68.2% ✓ ✓ 2.85 93.0% 5.28 91.3% 5.89 68.2% ✓ ✓ 3.25 92.8% 5.64 91.4% 6.01 90.4% ✓ ✓ ✓ 3.00 93.3% 5.48 92.1% 6.01 90.4% Improved Performance at Low Parallelism. By enabling dLLMs to revise their own predictions, our method not only mitigates error accumulation under aggressive parallel decoding, but also improves performance in the low-parallelism regime. Through iterative re-evaluation of earlier predictions, the model can recover from reasoning errors that would otherwise remain on the original decoding path. As shown in Table 2, our method consistently improves accuracy by 0.8%–3.0% across multiple benchmarks at low parallelism. Importantly, these gains are obtained using only the model’s own generated responses as training data, without introducing any external supervision. 5 Ablation Study Ablation Study on Training and Inference Strategies. Table 3 presents a comprehensive ablation study of both our training and inference designs. We compare different combinations of training and decoding strategies on GSM8K under three decoding thresholds, τdec ∈ {0.95, 0.5, 0.0}. On-policy rollout is the core of our training method. Even with OPUT alone, the model acquires the ability to revise its own errors, yielding substantial accuracy gains over the original model at τdec = 0.5 and 0.0. Our proposed SPD further improves robustness when many erroneous predictions emerge simultaneously, allowing the model to remain stable under highly parallel decoding and to preserve strong performance even in the extreme case of τdec = 0.0. The key ingredient of SPD is to use soft embeddings, rather than discrete tokens, as intermediate decoding states. Maintaining the non-masked region as a contiguous prefix further improves performance. Another important result is that OPUT is a prerequisite for SPD. As shown in Table 3, directly applying SPD to the original model causes generation to collapse. This is because OPUT trains the model to recover clean tokens from both mask tokens and predicted tokens, making interpolation between their embeddings a meaningful and effective input for denoising. Ablation Study on Convergence ``` ### Table 4: Ablation study on block-level convergence criteria. ```text Table 4: Ablation study on block-level convergence criteria. Criteria. We further study in Ta- The decoding threshold is set to 0.5. ble 4 how different block-level con- vergence criteria affect the efficiency– Convergence GSM8K MBPP performance trade-off. We consider Consistency Confidence TPF ↑ Acc. ↑ TPF ↑ Acc. ↑ two criteria: (1) consistency, where decoding is considered converged if ✓ 5.13 92.1% 5.16 79.9% the model produces the same top-1 ✓ 2.28 92.2% 3.36 80.1% ✓ ✓ 5.48 92.1% 5.86 79.2% prediction for the block in two con- secutive steps; and (2) confidence, where decoding is considered converged if the confidence of every token in the block exceeds 0.9. As shown in Table 4, consistency serves as the primary convergence signal, with most blocks terminating once this condition is met. Adding the confidence criterion can further improve TPF by allowing decoding to stop before two consecutive identical predictions are observed, thereby saving the final forward pass. Importantly, neither criterion affects the accuracy. 6 Related Work Diffusion Language Models. Diffusion models [31, 71] have become dominant in visual generation [64, 60, 65, 99], and recent work has explored their application to text generation. Among existing paradigms, masked diffusion language models (MDLMs) [70, 4, 66, 105, 51] have emerged as a promising alternative to AR-LLMs by modeling language in discrete space through masked token 9 prediction. Building on this formulation, LLaDA [58] and Dream [92] scale MDLMs to the billion- parameter regime with large-scale pretraining, demonstrating their practical potential. LLaDA-2.0 [10] and LLaDA-MoE [110] further show that MDLMs can be effectively scaled with mixture-of- experts architectures. Beyond these developments, dLLMs are also attracting increasing attention in reasoning [109, 59, 86, 63, 74, 57, 103], multimodal tasks [94, 96, 90, 91, 48, 82, 97, 18], code generation [87, 24, 21], long-context modeling [47, 28, 106], and agent [104, 102]. Accelerating Diffusion Language Models. dLLMs are viewed as promising due to their potential for low-cost inference, yet their efficiency remains largely underexplored. Existing efforts improve efficiency from several perspectives. Some methods reduce the cost of each decoding step through techniques including KV caching [53, 49, 84, 35, 45], token dropping [15, 36, 72, 85], and sparse attention [79, 19]. Others design more effective decoding strategies [37, 81, 41, 27, 8, 32, 34, 89, 50, 12, 77, 61, 22] to improve generation efficiency. A separate line of work [73, 62, 101, 7, 14, 33] learns better decoding trajectories so that fewer decoding steps are required. dParallel [16] employs certainty- forcing distillation to accelerate confidence convergence and enable higher parallel decoding. Other methods [83, 17, 78, 3, 52, 75, 46, 23] interpolate between diffusion and autoregressive language models to better balance speed and accuracy. [9, 76, 100] implement uniform training, which trains the model to recover clean tokens from random noisy tokens, thereby enabling token correction during generation. SM [30] and EvoToken [107] introduce soft embeddings into the decoding process, but neither method translates this design into improved decoding efficiency. Further efforts [88] leverage compression techniques to construct lightweight dLLMs. 7 Conclusion In this paper, we present DMax, a novel paradigm for efficient diffusion language models that mitigates error accumulation for parallel decoding. DMax enables aggressive decoding parallelism while preserving the accuracy of the original model. We introduce two key components of our approach, namely On-Policy Uniform Training and Soft Parallel Decoding, and demonstrate their effectiveness through extensive experiments on diverse benchmarks. Our results establish a strong new baseline for parallel decoding in dLLMs and suggest a promising new direction for dLLMs. 10 References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi, Somshubra Majumdar, and Boris Ginsburg. Opencodeinstruct: A large-scale instruction tuning dataset for code llms. arXiv preprint arXiv:2504.04030, 2025. [3] Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025. [4] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993, 2021. [5] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. ``` ## reference/Nemotron-Labs-Diffusion- A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding ### Table 1 | Ablation study of each training technique during continuous pretraining on 25B tokens. ```text Table 1 | Ablation study of each training technique during continuous pretraining on 25B tokens. Technique HumanEval HumanEval+ MBPP MBPP+ GSM8K Minerva Math Avg Block-wise attention 39.02 37.80 53.40 67.72 82.87 44.58 54.23 + Global Loss Avg 42.07 39.02 56.20 71.69 83.78 45.36 56.35 + DP-rank Varying Masking Ratios 45.12 43.29 55.80 70.90 81.58 45.66 57.06 + Two-stage training 58.54 52.44 53.00 73.81 83.17 55.84 62.80 + AR loss 64.02 57.93 65.60 80.95 86.73 66.44 70.28 Global loss averaging. Since the diffusion objective The Noisy→Noisy and Noisy→Clean parts follow involves randomly sampling masked tokens, different the standard block diffusion design [9]: we partition training examples may have different numbers of to- the sequence into 𝐵 contiguous blocks {𝑥𝑏 }𝐵 𝑏=1 ; in kens contributing to the diffusion loss. As a result, the the noisy stream, tokens attend bidirectionally within strategy for averaging token-wise losses matters, anal- each block and causally across blocks; and for denois- ogous to how the choice of aggregation in on-policy ing block 𝑏, noisy tokens additionally attend to the RL objectives (e.g., GRPO [11] vs. DAPO [12]) can clean-prefix blocks 𝑥<𝑏 in the clean stream to achieve affect training stability. We consider two loss averag- clean-context conditioning. ing strategies. Let a batch contain 𝑁 sequences, each The key difference lies in the Clean→Clean mask. of length 𝐿, and let ℓ𝑛,𝑖 denote the token-level loss Prior designs [9, 10, 13] allow the clean stream to for token 𝑖 in sequence 𝑛 based on Eq. 3. One choice attend to future tokens using block-wise attention. In is to first average token losses within each sequence contrast, we enforce a strictly causal mask within the and then average them over sequences: clean stream [14, 7]. This enables us to compute the 𝑁 𝐿 AR objective on 𝑥 in this clean-context part together 1 ∑︁ (︁ 1 ∑︁ )︁ ℒseq-avg = ℓ𝑛,𝑖 . (4) with the diffusion objective on 𝑥 ˜𝑡 in the same forward- 𝑁 𝑛=1 𝐿 𝑖=1 backward pass, without label leakage. Another choice is to treat all tokens across the batch Relationship with prior works. Our attention equally and globally average over the 𝑁 𝐿 token losses: pattern follows the pioneering work of [14], which also 𝑁 1 ∑︁ ∑︁ 𝐿 performs joint diffusion and AR training. The key ℒglobal = ℓ𝑛,𝑖 . (5) differences in our work lie in (1) proposing tri-mode in- 𝑁 𝐿 𝑛=1 𝑖=1 ference, particularly self-speculation decoding, along with post-training enhancements for improved paral- While Eq. 4 and Eq. 5 coincide when every sequence lelism, including the samplers and LoRA-enhanced has the same number of loss-contributing tokens, they drafters introduced in Sec. 3; (2) the overall training differ once masking yields variable numbers of noisy pipeline used to develop the full model family de- tokens across samples, which is common in the dif- scribed in Sec. 5; and (3) the systematic studies and fusion objective. In particular, in Eq. 2, the loss SOL analysis conducted to address critical questions includes a 1𝑡 reweighting, and the number of noisy regarding the true potential of diffusion LMs. tokens is approximately proportional to 𝑡. When 𝑡 is small, each noisy token tends to carry a larger 2.3. Ablation Study on Training Techniques weight (due to 1𝑡 ), but there are fewer such tokens We ablate the contribution of each training tech- in the sample. Sequence-wise averaging can there- nique by progressively adding them during contin- fore amplify the influence of these small-𝑡 samples: uous pretraining on 25B tokens, starting from the their per-token losses are larger, yet the per-sequence official Ministral3-8B base model. Detailed train- normalization assigns them the same weight as other ing/evaluation settings will be elaborated in Sec. 5.1 samples, increasing batch-to-batch fluctuations and and Sec. 6.3. All models are evaluated in diffusion gradient variance. In contrast, global averaging effec- mode on coding and math benchmarks. tively weights each training example in proportion to its number of contributing tokens, preventing samples Observations. As shown in Tab. 1, we progressively with only a few highly weighted noisy tokens from add each technique and observe that (1) block-wise disproportionately influencing the batch loss. attention serves as the baseline at 54.23% average accuracy, following the setting of [10, 4] and provid- 2.2. Attention Pattern ing a functional diffusion LM; (2) global loss averag- ing improves the average by 2.12%, confirming that Following [9], at training time we use a dual-stream treating all tokens equally across the batch reduces input by concatenating a corrupted/noised view and gradient variance from variable masking ratios, as a clean view of the same sequence, and apply a struc- analyzed in Sec. 2.1; (3) DP-rank varying masking tured attention pattern, as shown in Fig. 3. ratios, which applies different noise levels across data- 4 ``` ### Table 2 | Impact of diffusion loss weight 𝛼 during pretraining on top of the two-stage training setting in ```text Table 2 | Impact of diffusion loss weight 𝛼 during pretraining on top of the two-stage training setting in 25B-token continuous pretraining. Tab. 2. We observe that both modes peak at 𝛼=0.3. Human Human Minerva This implies that the two modes do not necessarily 𝛼 Mode Eval Eval+ MBPP MBPP+ GSM8K Math Avg Diff. 56.71 51.83 64.80 81.22 87.64 67.02 68.20 compete with each other or achieve the best perfor- 0.1 AR 58.54 53.05 64.60 80.42 87.64 68.02 68.71 mance at the two extremes; instead, there exists a Diff. 60.37 54.27 63.40 80.69 86.43 66.58 68.62 0.2 AR 60.98 57.93 66.40 83.60 87.04 67.04 70.50 sweet spot where both are well harmonized. Similarly, 0.3 Diff. 61.59 58.54 64.60 80.42 87.64 65.86 69.77 no value of 𝛼 in [0.1, 0.5] improves one mode at the AR 62.80 57.93 65.80 82.54 87.79 66.84 70.62 Diff. 59.76 54.27 64.40 80.16 87.11 66.98 68.78 expense of the other, and the two objectives rise and 0.5 AR 58.54 53.05 65.40 82.80 86.81 67.14 68.96 fall together, indicating that they are complementary Diff. 56.10 48.78 65.00 80.69 84.91 66.12 66.93 1.0 AR 54.27 50.61 64.80 80.69 86.58 66.36 67.22 rather than competing for model capacity. parallel ranks, further improves the average by 0.71%; We also visualize the training loss curves in Fig. 4. (4) two-stage training, which provides a better AR ini- We observe that the aforementioned setting 𝛼=0.3, tialization with 1T-token AR objective training, yields which achieves the best accuracy, provides a good a substantial 5.74% gain, implying that stronger AR balance between the two losses. Setting 𝛼 too small initialization can enable better future planning and or too large leads to increased diffusion or AR loss, ease AR-to-diffusion conversion; (5) the addition of respectively. In addition, without the AR or diffusion AR loss contributes the largest single improvement loss, the corresponding AR or diffusion capabilities of 7.48%, significantly boosting diffusion decoding are degraded or lost. abilities, echoing our analysis in Sec. 2. Diffusion loss preserves AR accuracy. To study Cumulatively, the full pipeline improves the base- whether diffusion loss can hurt or preserve AR ac- line by 16.05% in average accuracy, with the AR loss curacy, we compare models trained w/o and w/ the and two-stage training contributing the most. This diffusion loss (𝛼=0.3) under two settings: continuous validates our core insight that preserving the AR ob- pretraining on 25B tokens on top of the two-stage jective during diffusion training anchors the model training setting in Tab. 1, and further SFT on 45B to- to linguistically coherent trajectories and is a critical kens, following training/evaluation settings in Sec. 5.2 factor for achieving strong diffusion LM accuracy. and Sec. 6.1. We ensure that all settings are trained on the same number of tokens. 2.4. Mutual Impact of AR/Diffusion Losses As shown in Tab. 3, we observe that: (1) In both set- tings, the average AR accuracy is preserved or slightly In this subsection, we examine whether the AR and boosted, with 0.14% and 0.43% improvements for the diffusion objectives compete for model capacity or base and instruct models, respectively, indicating that reinforce each other: the impact of adding the AR diffusion training, when properly integrated, can en- loss on the diffusion mode, and the impact of adding hance the future prediction abilities of the AR mode, the diffusion loss on AR-mode accuracy. similar to observations in DeepSeek-V3 [15]; (2) At the AR loss boosts diffusion accuracy. As studied in per-benchmark level, the instruct model shows gains Sec. 2.3 and Tab. 1, AR loss can significantly boost on coding and math benchmarks, e.g., 4.24% higher on diffusion accuracy. As a complement to this study, LCB-CPP and 1.59% higher on MBPP, but drops on we also vary 𝛼 in Eq. 3 during 25B-token continuous IFEval (3.01% lower) and HumanEval (2.44% lower), 5 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding ``` ### Table 3 | AR-mode accuracy with and without the diffusion loss (𝛼=0.3). Base: 25B-token continuous ```text Table 3 | AR-mode accuracy with and without the diffusion loss (𝛼=0.3). Base: 25B-token continuous pretraining from Ministral3-8B. Instruct: further SFT on 45B tokens. Base Model Training HumanEval HumanEval+ MBPP MBPP+ GSM8K Minerva Math MMLU Hellaswag PIQA Winogrande Avg AR only 60.37 56.10 67.00 81.48 87.64 67.90 76.34 76.54 79.71 71.98 72.50 + Diff. loss 62.80 57.93 65.80 82.54 87.79 66.84 75.99 76.59 79.82 70.32 72.64 Instruct Model Training GPQA IFEval HumanEval MBPP Math500 GSM8K AIME24 AIME25 MMLU LCB-CPP Avg AR only 44.44 71.66 82.93 83.60 87.80 93.63 36.67 26.67 79.77 24.61 63.18 + Diff. loss 44.44 68.65 80.49 85.19 88.00 94.01 33.33 33.33 79.85 28.85 63.61 suggesting that strict instruction-following compli- criterion that we later analyze in Sec. 4. The sam- ance is slightly affected by the diffusion objective. pler architecture, feature engineering, and training trajectory data collection are detailed in Appendix A. These results, together with the 𝛼 sensitivity analysis above, support the conclusion that joint AR–diffusion training is not a zero-sum trade-off: the 3.3. Mode 3: Self-Speculation Decoding diffusion loss enables parallel decoding modes (Sec. 3) Linear self-speculation. The simplest self- at negligible cost to AR-mode accuracy, and the two speculative mode separates diffusion-based drafting objectives share a common optimal operating point. and AR-based verification into two forward passes. Let [𝑥1 , . . . , 𝑥𝑛 ] denote the currently verified prefix 3. Tri-Mode LM Inference and let 𝑘 be the speculative width. Drafting with diffusion. We append 𝑘 mask to- The joint AR and diffusion training enables decoding kens to the verified prefix, forming the input in three modes: AR, diffusion, and self-speculation [𝑥1 , . . . , 𝑥𝑛 , 𝑚1 , . . . , 𝑚𝑘 ]. The model denoises all 𝑘 decoding, as shown in Fig. 5. mask positions in parallel using the diffusion pathway, producing draft tokens {^ ^𝑛+𝑘 }. 𝑥𝑛+1 , . . . , 𝑥 3.1. Mode 1: AR Decoding Verification with AR. We then run a second forward Tri-mode LMs fully preserve standard left-to-right pass over the draft tokens [^ 𝑥𝑛+1 , . . . , 𝑥 ^𝑛+𝑘 ] with generation: at step 𝑖, they sample 𝑥𝑖 ∼ 𝑝𝜃 (· | 𝑥<𝑖 ) causal attention, again reusing the prefix KV cache. with causal attention. This mode is preferred when The AR logits at each position yield next-token pre- 𝑘 serving with high concurrency. dictions {𝑥AR𝑛+𝑗 }𝑗=1 . We accept the longest prefix of draft tokens that passes the verification criterion (e.g., 𝑥AR 𝑛+𝑗 = 𝑥^𝑛+𝑗 ) and commit the accepted tokens 3.2. Mode 2: Block-wise Diffusion Denoising to the verified prefix. As in standard speculative de- coding [17], the AR prediction at the first rejected Confidence-based sampling. Following [13, 10], position provides one additional verified token, so the diffusion decoding mode proceeds block by block. each step produces between 1 and 𝑘+1 tokens. Note For the current block, we initialize its positions as that both the drafting and verification passes can mask tokens and iteratively denoise multiple tokens in reuse the cached prefix KVs from prior verified steps. parallel per step based on a confidence threshold [16]. When a block is completed, its KV cache will be Enhance linear self-speculation w/ LoRA. We refreshed, and decoding proceeds to the next block. further enhance linear self-speculation by tuning a LoRA adapter [18] on top of the diffusion draft path- Sampling with a trained sampler. A fixed con- way to better align its drafts with the AR verifier, fidence threshold is an implicit signal that is not thereby extending the accepted prefix length per step. explicitly optimized during training. We therefore We apply LoRA only to the 𝑜proj layer of the at- train a lightweight sampler that, for every masked po- tention module (rank 128, 𝛼=512, ∼36M trainable sition in the current block, predicts whether the top-1 parameters/∼0.4% of the backbone), leaving the AR prediction at the current denoising step is correct. pathway unchanged. The training loss combines an Here, correct means that the decoded token matches LK-hybrid distribution-matching term [19] with a the token that will eventually be committed at this token-level cross-entropy term, both applied to the position when decoding only the highest-confidence accepted prefix plus the first rejected position of each token at each step. At inference, we commit positions draft block, as shown in Fig. 11 in Appendix B. whose predicted probability from the sampler exceeds a predefined threshold, which trades off TPF against Drafter–verifier setup. For each position 𝑗 ∈ per-token error rate. The sampler can be viewed as a {1, . . . , 𝑘} in the draft block, the LoRA-augmented learned classifier to approach the greedy-acceptance drafter produces logits 𝑧𝑗𝑑 ∈ R|𝒱| over the vocabulary ``` ### Table 5 | Benchmark our Nemotron-Labs-Diffusion-8B instruct model against SOTA AR and diffusion instruct ```text Table 5 | Benchmark our Nemotron-Labs-Diffusion-8B instruct model against SOTA AR and diffusion instruct LMs across scientific QA, instruction following, coding, and math reasoning benchmarks. Qwen2.5 Qwen3 Ministral3-8B LLaDA-8B Dream-7B SDAR-8B Nemotron-Labs-Diffusion-8B Model 7B 8B Instruct-2512 Instruct Instruct Chat (Tri-Mode in One Model) Gen. Mode AR AR AR Diff. Diff. Diff. Diff. AR Diff. Linear SS Quad. SS Scientific QA & Instruction Following GPQA 37.12 49.24 42.87 33.30 33.00 40.20 30.80 44.44 43.94 40.40 44.30 IFEval 74.58 87.38 64.31 59.90 62.50 61.40 60.07 68.65 68.32 69.13 71.00 MMLU 74.86 76.66 73.90 65.50 67.00 78.60 78.83 79.85 78.71 79.01 79.95 Coding HumanEval 77.44 81.71 71.04 49.40 55.50 78.70 79.27 80.49 78.66 81.71 79.27 MBPP 81.55 81.88 78.97 41.00 58.80 72.00 67.32 85.19 83.86 84.92 85.19 LCB-CPP 12.33 21.09 20.76 4.19 1.25 13.44 11.89 28.85 26.16 24.89 27.70 Math Math500 75.10 84.80 83.60 39.20 43.00 78.60 72.40 88.00 85.80 87.60 88.80 GSM8K 91.89 92.42 92.42 79.91 81.00 91.30 88.48 94.01 93.03 93.78 94.16 AIME24 13.75 30.21 27.71 0.00 0.00 16.67 13.33 33.33 46.67 36.67 33.33 AIME25 6.88 22.08 24.58 0.00 3.33 10.00 3.33 33.33 26.67 30.00 36.67 Average over All Tasks Accuracy 54.55 62.75 58.02 37.24 40.54 54.09 50.57 63.61 63.18 62.81 64.04 TPF 1.00 1.00 1.00 1.00 1.00 1.00 1.75 1.00 2.57 5.99 6.38 troduced; the vocabulary and embedding dimensions SOTA AR instruct models (Qwen3-8B, Qwen2.5-7B, remain unchanged. and Ministral3-8B Instruct) and SOTA diffusion in- struct models (LLaDA-8B Instruct [2], Dream-7B Continued SFT. Starting from the merged initial- Instruct [3], and SDAR-8B Chat [4]). We evalu- ization, we finetune the full model (LM backbone, ate all modes of our model: AR, diffusion, and self- vision encoder, and projector) with the same joint speculation, including both linear and quadratic self- AR-diffusion objective used for the text-only instruct speculation modes (denoted as linear SS and quadratic model, on multimodal instruction-following data [25]. SS). We use LoRA-enhanced linear self-speculation Asymmetric dual stream. A straightforward ex- by default. For the diffusion model evaluation of tension doubles all tokens in the noisy half, includ- Nemotron-Labs-Diffusion-8B and SDAR-8B Chat [4], ing vision tokens. However, vision tokens are never we also report tokens per forward (TPF) by selecting masked; only text response tokens are subject to for- different denoising thresholds [16]. All models are ward corruption. Carrying vision tokens in the noisy evaluated in the non-thinking mode. half, therefore, adds FLOPs without contributing to We evaluate across scientific QA and instruction fol- the diffusion loss. For high-resolution images, this lowing (GPQA, IFEval, MMLU), coding (HumanEval, overhead is substantial. We address this with an MBPP, LiveCodeBench-CPP), and math reasoning asymmetric dual-stream layout that strips all vision (Math500, GSM8K, AIME24, AIME25). We use token positions from the noisy half: NeMo-Skills [26] as the evaluation framework for all (text, 𝐿text ) AR baselines and our Nemotron-Labs-Diffusion-8B, | 𝑥(𝐿) , [︀ ]︀ 𝑥 ˜𝑡 𝐿text = 𝐿 − 𝑁vis , (10) and use the official evaluation pipelines provided in the original papers for the diffusion baselines [2, 3, 4]. where 𝑁vis is the number of vision tokens. The clean half retains the full sequence, including vision tokens, Observations. As shown in Tab. 5, we observe that, preserving complete visual context for the AR ob- compared to SOTA AR and diffusion instruct LMs, jective and for cross-stream conditioning. The total our Nemotron-Labs-Diffusion-8B achieves both higher sequence length becomes 𝐿text + 𝐿 instead of 2𝐿, and accuracy and efficiency across all modes. More specif- the reduction of FLOPs of attention scales with the ically, (1) In terms of AR performance, Nemotron- vision tokens 𝑁vis /𝐿. Labs-Diffusion-8B in AR mode delivers +0.86% higher average accuracy than Qwen3-8B and outperforms all other AR baselines, demonstrating that the joint 6. Evaluation and Analysis AR–diffusion training objective effectively preserves strong AR accuracy. In fact, the ablation study under 6.1. Benchmark Instruct Models a controlled setting in Sec. 2.4 indicates that adding the diffusion objective can maintain or slightly im- Baselines and benchmarks. We compare our prove AR accuracy, potentially due to an improved Nemotron-Labs-Diffusion-8B instruct model against 11 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding ``` ### Table 6 | Per-task TPF achieved by linear self-speculation w/ and w/o LoRA tuning across 3B/8B/14B scales. ```text Table 6 | Per-task TPF achieved by linear self-speculation w/ and w/o LoRA tuning across 3B/8B/14B scales. Model Scale Setting GPQA IFEval HumanEval MBPP Math500 GSM8K AIME24 AIME25 MMLU LCB-CPP Avg TPF Avg Acc w/o LoRA 3.07 2.97 4.77 3.74 4.94 4.01 4.33 4.53 2.42 3.34 3.81 55.00 3B w/ LoRA 3.57 3.30 5.56 4.23 5.63 4.55 4.99 5.11 2.74 3.95 4.36 55.00 w/o LoRA 5.10 4.32 4.62 3.44 5.43 4.47 5.38 5.05 3.25 4.14 4.52 62.88 8B w/ LoRA 6.64 5.52 5.82 4.44 7.36 5.89 7.44 6.92 4.08 5.70 5.99 62.81 w/o LoRA 5.31 3.86 6.75 4.42 5.01 4.54 4.47 3.92 4.95 3.42 4.67 66.35 14B w/ LoRA 6.07 4.74 8.11 5.22 6.72 5.79 5.88 5.41 7.22 4.46 5.96 66.36 ability to predict the future. (2) The diffusion mode decodes 2.57× TPF while achieving +0.43% higher av- erage accuracy than Qwen3-8B. Compared to existing 1.3x TPF diffusion LMs, Nemotron-Labs-Diffusion-8B outper- forms SDAR-8B Chat by +9.09% in average accuracy +10.6% Acc and better maintains accuracy under larger decoding parallelism, as shown in Fig. 1 (b). (3) LoRA-tuned linear self-speculation maintains comparable accuracy to the diffusion mode while further boosting TPF to 5.99×, indicating the effectiveness of aligning the diffu- sion drafter with the AR target via lightweight LoRA tuning. (4) Quadratic self-speculation can achieve the Figure 8 | Comparing the accuracy-TPF trade-offs highest TPF of 6.38×, as it prepares the next block for achieved w/ and w/o a sampler. all possible acceptance positions at a quadratic cost. ments from this simple design suggest that part of However, due to the use of FlexAttention with less op- the gap between the realized diffusion-mode TPF and timized kernels for the dedicated attention mask [20], the SOL ceiling in Sec. 4 can be closed by learning the real-device efficiency of quadratic self-speculation the acceptance policy itself. falls behind the linear one according to Fig. 1 (b). As such, we use linear self-speculation by default. The impact of LoRA tuning for linear self- speculation. We perform an ablation study on Remark. The tri-mode design enables Nemotron- the impact of LoRA tuning for aligning diffusion Labs-Diffusion to serve different deployment needs drafters and AR verifiers. As shown in Tab. 6, we within a single model: (1) The AR mode matches observe that (1) even without LoRA adapters, lin- or surpasses SOTA AR LMs in accuracy, meaning ear self-speculation already achieves nontrivial TPF, that Nemotron-Labs-Diffusion can serve as a drop-in e.g., 4.67× for our 14B model, with larger model replacement for any application that currently uses scales generally leading to higher TPF; and (2) adding an AR model, with no pipeline changes required. (2) LoRA tuning consistently improves TPF, yielding The diffusion mode enables one-for-all flexibility: by 14.4%/32.5%/27.6% relative gains at the 3B/8B/14B adjusting the denoising threshold, a single model can scales. We also note that the small accuracy gap be- achieve a range of accuracy-throughput trade-offs, tween different self-speculation settings and the AR as illustrated in Fig. 1 (b). (3) Self-speculation is mode is due to kernel mismatches between 1-token promising for achieving significant inference speedup decoding and multi-token prefilling. through the synergy between AR and diffusion. While it sacrifices the flexibility of the diffusion mode by only accepting prefix tokens, it provides a reliable 6.2. Extend to More Model Scales mechanism to verify diffusion drafts, which can lead Baselines and benchmarks. We extend the eval- to substantial inference acceleration, as demonstrated uation to two additional scales, Nemotron-Labs- in Sec. 6.5. Diffusion-3B/14B, under the same evaluation protocol Improve diffusion decoding with a better sam- as Sec. 6.1, and benchmark against SOTA open-source pler. We evaluate the effectiveness of the proposed AR instruct models at the corresponding scales. sampler in Sec. 3.2. We apply it on top of our instruct Observations. As shown in Tab. 7, we observe that: model with a block size of 32 and report the average (1) Nemotron-Labs-Diffusion maintains consistent im- accuracy across all ten tasks under different denoising provements in accuracy and efficiency across scales thresholds to obtain the accuracy–TPF trade-off. As and generation modes. For example, using LoRA- shown in Fig. 8, the trained sampler shifts the entire tuned linear self-speculation, our Nemotron-Labs- Pareto frontier upward, delivering higher TPF at the Diffusion-3B/14B outperforms the strongest base- same accuracy (e.g., 1.3× TPF) or higher accuracy at lines, Qwen3-4B/14B, by +1.77%/+1.19% in accu- the same TPF (e.g., +10.6% accuracy). The improve- racy while achieving 4.36×/5.96× TPF, respectively. 12 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding ``` ### Table 7 | Benchmark our Nemotron-Labs-Diffusion-3B/14B instruct models against SOTA AR instruct models. ```text Table 7 | Benchmark our Nemotron-Labs-Diffusion-3B/14B instruct models against SOTA AR instruct models. Model Gen. Mode GPQA IFEval HumanEval MBPP Math500 GSM8K AIME24 AIME25 MMLU LCB-CPP Avg Acc Avg TPF 3B Scale Llama-3.2-3B-Instruct AR 27.78 69.69 65.85 66.93 32.60 62.09 0.00 0.00 62.45 9.25 39.90 1.00 Phi-4-mini-Instruct AR 40.91 71.16 77.44 75.13 70.80 76.57 10.00 6.67 53.37 9.91 49.20 1.00 Ministral3-3B-Instruct-2512 AR 28.79 57.55 64.79 68.39 71.90 87.41 12.50 12.71 63.21 12.50 47.97 1.00 Qwen3-4B AR 37.18 72.20 75.91 72.49 68.40 92.19 10.63 10.63 77.05 15.58 53.23 1.00 AR 39.39 69.39 76.22 71.16 77.60 87.87 23.33 16.67 71.70 21.37 55.50 1.00 Nemotron-Labs Diff. 33.84 68.93 74.39 73.54 74.80 88.40 16.67 10.00 72.06 16.30 52.90 1.91 -Diffusion-3B Linear SS 35.86 68.96 75.00 70.11 77.40 87.79 26.67 16.67 71.89 20.04 55.00 4.36 (Tri-Mode) Quad. SS 42.93 71.36 79.27 76.46 78.20 88.25 13.33 16.67 72.12 19.60 55.80 5.42 14B Scale Gemma-3-12B-IT AR 38.32 85.73 58.23 85.45 84.55 90.45 23.75 16.88 76.41 20.54 58.03 1.00 Phi-3-Medium-14B AR 37.56 85.75 70.43 76.65 43.35 89.69 1.88 0.62 76.67 10.79 49.34 1.00 Phi-4-14B AR 56.94 68.96 84.60 83.93 79.95 92.27 19.17 15.83 84.76 21.75 60.82 1.00 Ministral3-14B-Instruct-2512 AR 52.02 71.51 72.56 82.47 86.30 92.80 36.25 29.38 79.88 26.54 62.97 1.00 Qwen3-14B AR 50.51 88.36 83.54 87.30 85.40 94.31 33.33 20.00 81.51 27.42 65.17 1.00 AR 54.55 68.50 86.59 85.19 88.40 91.36 46.67 43.33 82.51 27.48 67.46 1.00 Nemotron-Labs Diff. 48.99 69.03 83.54 82.80 85.80 93.71 43.33 50.00 82.17 25.77 66.51 2.74 -Diffusion-14B Linear SS 47.47 70.06 85.37 84.66 86.60 92.04 50.00 40.00 81.11 26.32 66.36 5.96 (Tri-Mode) Quad. SS 52.02 72.15 87.20 85.45 88.00 92.12 53.33 40.00 82.45 28.74 68.15 6.92 ``` ### Table 8 | Benchmark our Nemotron-Labs-Diffusion-8B base model against SOTA AR and diffusion base LMs ```text Table 8 | Benchmark our Nemotron-Labs-Diffusion-8B base model against SOTA AR and diffusion base LMs across coding, math, knowledge, and commonsense reasoning benchmarks. Human Human Minerva Hella Wino Avg Avg Model Gen. Mode MBPP MBPP+ GSM8K MMLU ARC-E ARC-C PIQA Eval Eval+ Math swag grande Acc TPF Llama-3.1-8B AR 35.37 28.66 48.80 61.90 54.06 18.22 65.15 81.31 53.41 78.93 81.18 77.43 57.04 1.00 Ministral3-8B AR 42.68 38.41 61.60 76.98 80.21 44.58 76.39 86.15 60.75 79.01 80.74 73.48 66.75 1.00 Qwen3-8B AR 64.63 56.71 69.40 83.07 86.73 52.94 76.93 81.90 53.16 78.59 79.22 75.69 71.58 1.00 LLaDA-8B Diff. 32.32 27.44 40.80 51.85 70.96 27.30 65.86 73.78 49.15 71.05 73.88 74.66 54.92 1.00 Dream-7B Diff. 54.88 49.39 56.80 74.60 77.18 39.60 67.00 82.20 59.13 73.73 75.52 73.56 65.30 1.00 AR 60.37 53.05 68.20 82.54 88.25 66.00 74.68 83.38 58.11 76.08 80.09 71.98 71.89 1.00 Nemotron-Labs Diff. 62.80 57.32 67.00 81.75 87.26 65.16 74.68 83.38 58.11 76.08 80.09 71.98 72.13 2.06 -Diffusion-8B Linear SS 63.41 56.10 67.20 81.75 88.17 67.38 74.68 83.38 58.11 76.08 80.09 71.98 72.36 4.67 (Tri-Mode) Quad. SS 62.20 54.88 67.60 81.48 88.48 66.24 74.68 83.38 58.11 76.08 80.09 71.98 72.10 7.04 (2) Based on the performance of Nemotron-Labs- racy compared to the strongest baseline, Qwen3-8B. Diffusion-3B/8B/14B, larger LMs generally more readily unlock parallel diffusion abilities, as the TPF 6.4. Benchmark VLMs of diffusion/self-speculation modes grows broadly with scale. For example, the TPF of linear self-speculation Benchmarks and evaluation settings. We eval- increases from 4.36× to 5.96× when scaling from uate on a diverse set of VLM benchmarks spanning 3B to 14B. We attribute this to the stronger future- two categories. Short-answer benchmarks require prediction abilities of larger models, which yield more brief, factual responses: AI2D [27], ChartQA [28], reliable draft predictions. DocVQA [29], MMMU [30], MathVista [31], and Re- alWorldQA [32]. Long-answer benchmarks require 6.3. Benchmark Base Models extended chain-of-thought reasoning: MMMU-Pro- V [33]. We benchmark against existing diffusion Baselines and benchmarks. We compare VLMs [34, 35, 36, 37]. All benchmarks are evalu- Nemotron-Labs-Diffusion-8B against SOTA AR base ated using VLMEvalKit [38] under the same prompts LMs and two representative diffusion LMs (LLaDA- and post-processing as the AR baseline (Ministral3 8B [2] and Dream-7B [3]). We evaluate on cod- VLM). Throughput (tokens per second, TPS) is mea- ing benchmarks (HumanEval, HumanEval+, MBPP, sured on a single NVIDIA H100 GPU with identical MBPP+), math reasoning (GSM8K, Minerva Math), prompt batching for fair comparison. knowledge (MMLU), and commonsense reasoning Observations. As shown in Tab. 9, we compare (ARC-E, ARC-C, Hellaswag, PIQA, Winogrande). our Nemotron-Labs-Diffusion-VLM against existing Observations. As shown in Tab. 8, we observe diffusion VLMs in three modes: diffusion, AR, and findings consistent with the instruct model results. linear self-speculation decoding. We observe that Our Nemotron-Labs-Diffusion-8B base model achieves (1) In terms of AR performance, our model deliv- both higher accuracy and efficiency across all modes: ers 1.3% higher average accuracy than the strongest (1) the AR mode delivers +5.14%/+0.31% higher baseline, LLaDA-V-8B. (2) The diffusion mode pro- average accuracy than Ministral3-8B/Qwen3-8B; (2) vides 2.46×–3.15× TPF while maintaining compet- the diffusion mode delivers +17.21%/+6.83% higher itive accuracy. (3) Linear self-speculation preserves accuracy than LLaDA-8B/Dream-7B; and (3) self- near-AR accuracy, with only a 0.1% average accuracy speculation achieves 4.67× TPF (linear) and 7.04× drop, while further increasing decoding parallelism to TPF (quadratic) with over 0.5% higher average accu- 3.63×–7.45× TPF, where the higher end is achieved 13 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding ``` ### Table 9 | Benchmarking discrete diffusion VLMs and Nemotron-Labs-Diffusion-VLM across tasks. The ```text Table 9 | Benchmarking discrete diffusion VLMs and Nemotron-Labs-Diffusion-VLM across tasks. The diffusion mode of Nemotron-Labs-Diffusion-VLM uses denoising threshold 𝜏 =0.9. MMMU MMMU Math RealWorld Model Gen. Mode AI2D ChartQA DocVQA MMMU TPF Acc Pro-10c Pro-V-CoT Vista QA MMaDA Diff. 67.4 9.6 9.5 30.2 16.5 8.5 33.4 49.2 1 28.0 LaViDa Diff. 70.0 59.0 64.6 43.3 28.7 10.5 44.8 54.5 1 46.9 Dimple Diff. 74.4 63.4 37.7 45.2 23.8 12.4 42.3 55.4 1 44.3 LLaDA-V-8B Diff. 77.8 78.3 83.9 48.6 35.2 18.6 59.7 63.2 1 58.2 AR 75.0 81.3 89.2 50.3 32.6 24.3 60.4 62.6 1 59.5 2.46 all samples Nemotron-Labs Diff. 74.7 76.6 88.3 50.4 31.7 22.2 58.5 60.3 2.80 tok>100 57.9 -Diffusion 3.15 tok>200 -VLM-8B 3.63 all samples Linear SS 74.9 81.2 89.3 50.0 32.8 24.1 60.7 62.4 6.03 tok>100 59.4 7.45 tok>200 for responses exceeding 200 tokens. This implies that through (1) significantly higher acceptance length, the advantage of our model is most pronounced on and (2) token-parallel drafting that enables better tasks requiring longer reasoning. These results demon- GPU utilization. strate that the joint AR–diffusion training framework Setup. We deploy Nemotron-Labs-Diffusion-8B with extends effectively to the vision-language setting, pre- the SGLang server and profile it on NVIDIA GB200, serving the broad capabilities of the LM backbone RTX Pro 6000, and DGX Spark under different con- while enabling efficient multi-token decoding. currency levels, comparing against Qwen3-8B-Eagle3, with results shown in Fig. 1 (c) and Fig. 9. Eval- 6.5. Inference Efficiency uations are conducted on SPEED-Bench [1] across four categories (math, coding, reasoning, and multilin- We analyze and compare the deployment efficiency of gual), and limiting generation length to 1024 tokens Nemotron-Labs-Diffusion against MTP/Eagle3-style to avoid repetition/hallucinations. We perform a speculative decoding. grid search over hyperparameters for Eagle3, whereas Self-speculation vs. MTP. MTP methods such for Nemotron-Labs-Diffusion we only vary the block as Eagle3 [8] have become the default choice for effi- length. We additionally report a SOL throughput cient LLM deployment at low concurrency, where a estimate mentioned in Sec. 4, providing a reference small model is used to draft multiple future tokens, ceiling for current self-speculation infrastructure. and then a single forward pass of a larger AR model Observations. As shown in Fig. 1 (c) and Fig. 9, we verifies and accepts some of them. This schedule is observe that: (1) Linear self-speculation consistently more efficient at low concurrency because the memory improves user throughput over the AR mode across all transfer cost is similar between token-by-token gen- three GPUs, achieving up to 3.3× speedup over AR eration and verification passes, while the latter can on GB200 (3.97× speedup and 1015 tok/sec with an accept multiple tokens in a single pass. The two main optimized kernel), as shown in Fig. 9 (c), and pushing bottlenecks of Eagle3 are: (1) the draft model has the absolute throughput at batch size 1 to 277/525 limited capacity and is less reliable beyond a short tok/sec on RTX Pro 6000 (3.46×/2.35× over AR) and horizon; and (2) proposals are generated recursively, 77.5/112.5 tok/sec on DGX Spark (3.14×/2.69× over so even if the draft model is tiny, it still incurs the AR) under FP8/INT4 quantization, demonstrating cost of the embedding layer and LM head. In contrast, its effectiveness as a drop-in low-concurrency accel- Nemotron-Labs-Diffusion provides unique advantages c=128 4000 Autoregressive FP8 INT4 SOL FP8 12.36× 1750 SOL INT4 SOL Linear SS INT4-AWQ-Marlin INT4-AWQ-Marlin Decode Throughput at c=1 (tok/sec) 989 250 GPU Throughput at c=1 (tok/sec) 3500 Qwen3-8B-Eagle3 1000 5.75× 9.03× 1500 1471 223.1 Throughput at c=1 (tok/sec) GPU Throughput (tok/sec) BL=8 3000 c=64 c=64 FP8 SOL 800 200 7.13× c=64 ``` ### Table 10 | Per-category acceptance length on SPEED- up work has further explored alternative diffusion ```text Table 10 | Per-category acceptance length on SPEED- up work has further explored alternative diffusion Bench [1]. Comparing Native / LoRA for Nemotron- LM paradigms [48, 49, 6], and scaled them to larger Labs-Diffusion-8B and Qwen3-8B-Eagle3 / Qwen3- scales [50, 51, 52] or domain-specific specialists such 9B-MTP, all with draft length 31. as coding agents [53, 54, 55, 56], explored dedicated reinforcement learning schemes [57, 58], and extended Category Native LoRA Eagle3 MTP them to more modalities [36, 34]. Compared to AR coding 6.61 8.57 3.14 5.97 LMs, diffusion LMs have been demonstrated to be math 6.24 8.14 2.79 4.80 better learners under data-constrained settings [59] reasoning 6.18 7.99 3.40 3.68 multilingual 7.96 10.06 1.91 4.47 and show improved performance in planning [3] and humanities 5.01 6.31 3.12 3.76 text embedding [60]. qa 4.01 4.65 2.63 3.50 Diffusion language model acceleration. De- rag 5.07 6.15 3.06 4.75 spite the acceleration potential of large diffusion roleplay 4.66 5.54 2.10 2.32 stem 5.55 7.02 2.92 4.45 LMs [2, 3], the gap between bidirectional attention summarization 4.47 5.48 2.66 3.69 and KV caching, along with the one-token-per-step writing 4.28 5.07 2.81 3.21 denoising process, limits their achievable speed-up. To address these challenges, dedicated caching strate- Average 5.46 6.82 2.75 4.24 4 category avg 6.75 8.69 2.81 4.73 gies [61, 62, 16] have been developed to reuse com- putations and approximate bidirectional attention. In addition, to realize the potential of parallel token eration scheme. (2) Compared with Eagle3, linear generation, confidence-based sampling [16], guidance self-speculation delivers a 2.4×/2.3×/1.8× speedup from AR models [63], and adaptive decoding with cer- at batch size 1 on GB200/RTX Pro 6000/DGX tainty and positional priors [64] have been proposed. Spark and achieves better trade-offs between system Beyond these training-free methods, [65, 3] propose throughput and per-user throughput, as shown in initializing diffusion LMs from AR models with token Fig. 1 (c) and Fig. 9 (a). This indicates that diffusion shifts to accelerate diffusion LM training. Block Dif- drafting paired with AR verification is a more effec- fusion [9] combines AR and diffusion by performing tive acceleration mechanism than auxiliary-head MTP block-wise AR and in-block diffusion to support na- due to its higher acceptance length. (3) The SOL tive KV caching. Follow-up works [13, 10, 4, 66, 67] ceiling reveals substantial remaining headroom: on also convert pretrained AR models or diffusion LMs RTX Pro 6000, the projected SOL throughput reaches into block-wise ones. [14, 7] further explore combin- 7.09×/12.36× over AR under FP8/INT4 quantiza- ing AR and diffusion through either joint training or tion, roughly 2× above linear self-speculation. LoRA modules dedicated to diffusion. Acceptance length per category. As shown in ``` ### Tab. 10, Nemotron-Labs-Diffusion achieves signifi- 8. Insights and Future Directions ```text Tab. 10, Nemotron-Labs-Diffusion achieves signifi- 8. Insights and Future Directions cantly higher acceptance length than both Eagle3 and MTP across all categories, with average accep- We deliver Nemotron-Labs-Diffusion, a tri-mode lan- tance lengths of 5.46/6.82 for Native/LoRA-tuned guage model family trained via joint AR-diffusion Nemotron-Labs-Diffusion versus 2.75/4.24 for Ea- optimization that unifies AR, diffusion, and self- gle3/MTP. The gap further widens to 6.75/8.69 vs. speculation within a single model. The resulting 2.81/4.73 on the four diffusion-friendly categories base, instruct, and vision-language models outper- (coding, math, reasoning, multilingual), implying that form SOTA open-source AR/diffusion LMs in both diffusion drafting yields more reliable multi-token pro- accuracy and efficiency. The training and analysis of posals, especially on structured tasks with strong tri-mode LMs reveal the following insights: syntactic or semantic constraints. 1. Tri-mode generation arises naturally from 7. Related Work joint AR-diffusion training. By enabling both AR and non-AR parallel token prediction within Diffusion language models. To overcome the a single model, the joint training objective simul- token-by-token decoding nature of AR LMs, dif- taneously produces three inference modes without fusion LMs, both continuous [39, 40, 41] and dis- any mode-specific architectural modifications. crete [42, 43, 44, 45, 46, 47], have been proposed 2. AR and diffusion losses are complementary, to perform non-AR decoding and thus enable par- not competing. The two objectives mutually allel token generation. Among them, masked diffu- benefit each other and peak at the same loss co- sion LMs [43, 45, 44, 2, 3] have been successfully efficient (𝛼=0.3). Adding the AR loss induces scaled up (e.g., LLaDA [2] and Dream [3]). Follow- left-to-right linguistic priors for diffusion, and the 15 Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding diffusion loss preserves or slightly improves AR 3. Beyond prefix-only AR verification. Current accuracy through better future planning. AR verification accepts drafted tokens only in a 3. Self-speculation outperforms MTP methods. prefix-wise manner, which does not fully exploit Instead of relying on auxiliary prediction heads, the non-AR nature of diffusion drafts. A promising self-speculation leverages diffusion to generate high- direction is to explore diffusion-mode verification, quality multi-token drafts and uses AR verification potentially using another diffusion verifier, to vali- to ensure correctness, achieving higher acceptance date multiple non-contiguous drafted tokens and rates and better efficiency. further improve the effective acceptance rate. 4. Variance reduction is critical for diffusion 4. Enabling higher-level parallelism in diffu- training. The diffusion loss introduces intrinsi- sion generation. Although diffusion decoding en- cally high variance due to random masking with ables parallel token prediction, its generation order variable noise levels. More sufficiently trained AR still exhibits a strong left-to-right tendency and starting points (e.g., via two-stage training) or mainly provides token-level parallelism. Future variance-reduction training techniques (e.g., global training algorithms that encourage segment-level loss averaging) can improve training effectiveness. or paragraph-level parallelism could better unlock 5. Linear self-speculation is currently the most the global planning ability and efficiency potential efficient mode. Linear self-speculation achieves of diffusion-mode generation. the best efficiency in the current infrastructure. Quadratic self-speculation achieves higher TPF per step, making it more promising at batch size 1 References with improved infrastructure support. [1] Talor Abramovich, Maor Ashkenazi, Benjamin 6. Diffusion-mode decoding has substantial Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran headroom. Our SOL analysis shows the poten- Zilberstein, Yonatan Geifman, et al. Speed-bench: A tial to correctly predict 76.5% more tokens per unified and diverse benchmark for speculative decod- forward pass than the current best strategy (linear ing. arXiv preprint arXiv:2604.09557, 2026. self-speculation), indicating a more promising up- [2] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, per bound for parallel decoding than speculative Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong decoding based only on prefix decoding. Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025. Looking forward, these insights shed light on several promising directions for further improvements: [3] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv 1. Closing the gap between practical diffusion preprint arXiv:2508.15487, 2025. decoding and its SOL upper bound. Our SOL analysis suggests that diffusion-mode decod- [4] Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng ing could offer a more attractive path toward par- Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, allel decoding than linear decoding, due to its Qipeng Guo, Kai Chen, Biqing Qi, et al. Sdar: non-prefix acceptance pattern and therefore higher A synergistic diffusion-autoregression paradigm for upper bound. However, current confidence-based scalable sequence generation. arXiv preprint samplers remain far from this upper bound. De- arXiv:2510.06303, 2025. veloping optimized samplers that more reliably [5] Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian identify correct tokens, or more advanced train- Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. ing schemes that enable more aggressive parallel Scaling up masked diffusion models on text. arXiv ``` ## reference/TAD- Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM ### Table 1: Comparison of TAD-LLaDA with other LLaDA-based models. The best results among ```text Table 1: Comparison of TAD-LLaDA with other LLaDA-based models. The best results among acceleration methods are highlighted in bold and the second best are underlined. GSM8K-CoT MATH MBPP HumanEval Average Method Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ LLaDA 72.6 1.00 72.6 32.2 1.00 32.2 41.7 1.00 41.7 38.3 1.00 38.3 46.2 1.00 46.2 Fast-dLLM 74.7 2.77 182.6 30.8 1.97 45.7 38.6 2.13 56.6 37.8 2.56 54.0 45.5 2.36 84.7 D2F 73.2 2.88 189.2 28.7 2.38 47.2 38.0 1.94 53.0 36.6 2.69 62.0 44.1 2.47 87.9 dParallel 72.6 5.14 308.7 30.2 3.17 61.3 40.0 2.35 60.5 39.0 4.93 83.7 45.5 3.90 128.6 d3LLM 73.1 9.11 539.2 30.4 5.74 100.1 40.6 4.21 88.4 39.6 5.95 96.6 45.9 6.25 206.1 TAD-Q 79.9 6.32 503.8 42.7 4.49 190.4 41.6 3.38 89.3 42.1 6.11 117.4 51.6 5.08 225.2 TAD-S 78.8 8.47 649.0 40.4 5.17 191.0 40.6 3.42 80.3 39.6 5.96 108.0 49.9 5.76 257.1 ``` ### Table 2: Comparison of TAD-Dream with other Dream-based models. The best results among ```text Table 2: Comparison of TAD-Dream with other Dream-based models. The best results among acceleration methods are highlighted in bold and the second best are underlined. GSM8K-CoT MATH MBPP HumanEval Average Method Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ Acc↑ TPF↑ AUP↑ Dream 83.9 1.00 83.9 39.6 1.00 39.6 57.2 1.00 57.2 55.2 1.00 55.2 59.0 1.00 59.0 Fast-dLLM 79.0 1.44 116.5 38.3 1.78 55.2 53.2 1.20 63.6 54.3 1.33 63.5 56.2 1.44 74.7 Fast-dLLM-v2 77.5 2.21 156.0 48.7 2.61 126.7 50.1 2.04 81.9 61.7 2.58 128.9 59.5 2.36 123.4 dParallel 82.1 3.02 245.7 38.7 2.94 77.9 55.4 2.24 108.0 54.3 2.57 98.8 57.6 2.69 132.6 d3LLM 81.4 4.94 391.3 38.2 3.92 97.5 55.6 2.96 141.4 57.1 3.20 129.5 58.1 3.76 189.9 TAD-Q 81.4 3.26 255.6 42.8 3.96 144.2 56.8 2.32 122.8 64.1 2.35 134.3 61.3 2.97 164.2 TAD-S 81.0 4.59 359.5 41.3 5.30 161.3 53.8 2.38 112.9 62.2 2.90 147.5 59.6 3.79 195.3 Baselines. Our baselines fall into three categories: (1)The original dLLMs, LLaDA-Insturct [5] and Dream-Instruct [8]; (2)Training-free methods: Fast-dLLM [7]; (3)Training-based methods, including Fast-dLLM-v2 [27], D2F [15], dParallel [10]and d3LLM [9]. For a fair and consistent comparison, we directly report the baseline results disclosed in the d3LLM paper [9]. 4.2 Main Results Results on LLaDA Model. Table 1 demonstrates that the TAD framework successfully improve the accuracy-parallelism trade-off across two flexible configurations. Across all benchmarks, the Quality model (TAD-Q) achieves the highest average accuracy of 51.6%, substantially outperforming the 46.2% baseline average. On the complex MATH dataset where baselines typically degrade, TAD-Q improves accuracy to 42.7% while decoding 4.49 tokens per forward (TPF). The Speed model (TAD-S) attains the highest average AUP score of 257.1. On GSM8K-CoT, TAD-S reaches 8.47 TPF with 78.8% accuracy, achieving a better trade-off compared to d3LLM [9] and dParallel [10]. Both models also enhance code generation, with TAD-Q surpassing the baseline HumanEval accuracy while decoding approximately 6 tokens per forward. These results confirm the effectiveness of our trajectory collection mechanism and temporal-aware distillation framework. Results on Dream Model. Applying TAD to the Dream architecture (Table 2) confirms its strong generalization. TAD-Q achieves the highest average accuracy of 61.3%, peaking at 64.1% on HumanEval to significantly outperform the original model. On MATH, TAD-Q maintains a strong 42.8% accuracy while TAD-S achieves the peak AUP score of 161.3 at 5.30 TPF. Both configurations avoid the severe performance drops on GSM8K-CoT and MBPP typical of training-free methods. Although baselines like d3LLM occasionally exhibit marginally higher raw throughput, TAD-S delivers the highest average AUP score of 195.3. These results confirm that our TAD framework improves the accuracy-parallelism trade-off. 4.3 Ablation Study We conduct ablation studies on the LLaDA-8B architecture to validate the design choices within the TAD framework. Effect of Decoupled Distillation Objectives. Table 3 evaluates the individual objective components using a fixed window of δ = 8. Applying hard cross-entropy globally forces early confidence on 7 ``` ### Table 3: Ablation study on the distillation objectives. ‘Hard CE’ refers to cross-entropy with reference ```text Table 3: Ablation study on the distillation objectives. ‘Hard CE’ refers to cross-entropy with reference tokens from trajectories, and ‘Soft KL’ refers to KL divergence with teacher distributions. Objectives MATH HumanEval Near Distant TPF ↑ Acc ↑ AUP ↑ TPF ↑ Acc ↑ AUP ↑ Hard CE Hard CE 4.34 40.3 159.1 5.37 32.9 72.1 Soft KL Soft KL 2.08 41.4 90.0 2.90 35.4 55.4 Hard CE None 5.12 34.8 142.9 6.47 39.6 110.7 Hard CE Soft KL 4.49 42.7 190.4 6.11 42.1 117.4 ``` ### Table 4: Ablation on data collection methods and distillation strategies. We report the average ```text Table 4: Ablation on data collection methods and distillation strategies. We report the average Accuracy, TPF, and AUP score across the four evaluated benchmarks. Data Source and Training Strategy Acc (%) TPF AUP Standard SFT (Ground Truth + Random Mask) 46.2 3.24 105.8 Trajectory (with privileged information) + Random Mask 47.9 4.26 152.8 Trajectory (w/o privileged information) + TAD Distillation 47.9 5.03 187.1 Trajectory (with privileged information) + TAD Distillation (Ours) 51.6 5.08 225.2 distant tokens, lowering HumanEval accuracy to 32.9%. Conversely, global soft KL divergence lacks deterministic targets to anchor the generation path, resulting in over-smoothed predictions and minimal acceleration (2.08 TPF on MATH). Restricting hard cross-entropy solely to the near subset accelerates generation but reduces MATH accuracy to 34.8%, demonstrating the necessity of distant supervision. Combining near-term hard targets with distant soft supervision optimally resolves this problem, yielding the highest accuracy and AUP scores across both benchmarks. Impact of Privileged Information-Guided Trajectories. Table 4 evaluates four training paradigms to confirm the necessity of privileged information. Standard supervised fine-tuning with random masking ignores sequential dependencies and yields the lowest performance. Applying random masking to the final text of the privileged information-guided trajectory improves accuracy but still omits natural state transitions. Distilling from valid trajectories generated without privileged information increases TPF but limits accuracy. In contrast, distilling from trajectories generated with privileged ground-truth context achieves the highest average accuracy (51.6%) and optimal AUP score (225.2), confirming that these paths provide essential high-quality targets. Sensitivity to the Spatial Partition Window (δ). Table 5 presents average performance under varying window sizes. A conservative window (δ = 4) yields the highest accuracy (52.1%) but restricts decoding speed. Expanding δ to 20 provides deterministic supervision to a larger sequence portion, accelerating generation to 5.76 TPF and achieving the peak AUP score (257.1) with only a minor accuracy decline. However, an extreme window (δ = 256) mimics global cross-entropy, compelling the model to predict distant tokens without sufficient context and severely degrading performance. These observations justify our dual-model strategy, assigning a moderate window for robust reasoning (Quality model) and a larger window for maximized throughput (Speed model). ``` ### Table 5: Average performance across four bench- Table 6: Average performance across four bench- ```text Table 5: Average performance across four bench- Table 6: Average performance across four bench- marks under varying partition window sizes (δ). marks under varying KL weights (λ). δ Acc (%) TPF AUP λ Acc (%) TPF AUP δ=4 52.1 4.33 190.3 λ = 0.0 47.2 5.35 192.1 δ=8 51.6 5.08 225.2 λ = 0.5 51.1 4.90 215.9 δ = 12 49.9 4.64 196.4 λ = 1.0 51.6 5.08 225.2 δ = 16 48.9 5.57 229.2 λ = 1.5 50.5 4.89 207.5 δ = 20 49.9 5.76 257.1 λ = 2.0 50.4 4.74 199.5 δ = 24 48.9 5.78 237.1 δ = 256 (Global CE) 46.3 5.10 208.4 8 Sensitivity to the weight of Ldistant (λ). Table 6 evaluates the balance between near cross-entropy and distant Kullback-Leibler divergence. Removing distant supervision (λ = 0) maximizes speed (5.35 TPF) but causes the lowest average accuracy (47.2%), confirming that omitting distant depen- dencies damages generation quality. Integrating the distant constraint at λ = 1.0 restores reasoning capabilities, achieving the optimal average accuracy (51.6%) and AUP score (225.2). Increasing the weight further (λ ≥ 1.5) overemphasizes the soft objective, reducing the impact of near-term certainty forcing and subsequently decreasing. 5 Related Work 5.1 Diffusion Large Language Models Recent research has extended diffusion modeling from continuous domains [28] to discrete text generation [1, 3, 2, 4]. Unlike traditional autoregressive models that rely on left-to-right sequential generation [29, 30, 31], Diffusion Large Language Models (dLLMs) feature bidirectional context attention and parallel decoding capabilities [32]. Recent models, including the LLaDA series [5, 33, 34, 35], Dream [8], and SDAR [36], achieve performance competitive with leading autoregressive models across various benchmarks. They also demonstrate advantages in reverse reasoning tasks that require global planning [5]. Beyond these developments, the research community has increasingly focused on enhancing reasoning capabilities [37, 38, 39, 40, 41], building agent systems [42, 43], and accelerating inference [16] for dLLMs. In this paper, we focus on further accelerating dLLM inference by increasing the parallelism of these models. 5.2 Inference Acceleration for dLLMs The inference speed of dLLMs is primarily hindered by the incompatibility of traditional KV caching with bidirectional attention and the severe quality degradation during highly parallel decoding [7]. To alleviate the caching bottleneck, recent studies [44, 45, 46, 7] exploit the temporal consistency of KV states across decoding iterations to develop approximate caching mechanisms, significantly reducing redundant computations. To enhance parallelism, current approaches are categorized into training- free [7, 11, 12, 47, 48, 49] and training-based [10, 15, 50, 9, 14, 51, 52, 53] methods. Training-free strategies accelerate inference by dynamically adapting decoding schedules, but their effectiveness is bounded by the capacity of the model. Alternatively, training-based methods finetune the model for parallel generation. While they achieve higher throughput, this acceleration typically comes at the expense of generation quality. Building upon the training-based paradigm, our work improves this trade-off through a privileged-information strategy to acquire high-quality trajectories and a temporal-aware distillation framework. 6 Limitations As a training-based method, TAD mainly has three limitations. First, the framework depends on high- quality ground-truth responses to generate trajectory, which restricts its applicability in unsupervised settings. Second, the token-by-token rollout during trajectory collection introduces overhead prior to distillation. Third, the partition window δ requires empirical tuning across architectures. We leave dynamic window sizing and data-efficient trajectory generation for future work. 7 Conclusion We present TAD, a temporal-aware trajectory self-distillation framework that improves the accuracy- parallelism trade-off. The framework collects high-quality trajectories via a teacher conditioned on privileged information and partitions masked positions by their decoding steps, applying cross-entropy to near tokens for throughput and KL divergence loss to distant tokens for dependency preservation. Experiments on mathematical reasoning and code generation confirm that this design improves both accuracy and decoding speed, offering a practical path toward deploying efficient dLLMs. 9 References [1] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981–17993, 2021. ``` ### Table 7: Training hyperparameters for TAD. ```text Table 7: Training hyperparameters for TAD. Hyperparameter TAD-LLaDA TAD-Dream Training Optimizer AdamW AdamW Learning Rate 2 × 10−5 2 × 10−5 LR Scheduler Constant Constant Warmup Steps 0 0 Weight Decay 0.01 0.01 Max Gradient Norm 1.0 1.0 Training Epochs 1 1 Per-Device Batch Size 4 4 Gradient Accumulation Steps 8 8 Global Batch Size 256 256 Precision bfloat16 bfloat16 DeepSpeed Stage ZeRO-2 ZeRO-2 Max Sequence Length Dynamic Dynamic LoRA Rank (r) 128 128 Alpha (α) 128 128 Dropout 0.05 0.05 Bias None None Target Modules q,k,v,o,gate,up,down_proj Distillation λ 1.0 1.0 Temperature (τ ) 1.0 1.0 C.2 Evaluation Details C.2.1 Evaluation Metrics We adopt the AUP (Accuracy Under Parallelism) [9] score as our primary evaluation metric. AUP is defined as a weighted area under the accuracy–parallelism curve, where parallelism is measured by Tokens Per Forward pass (TPF). Given a set of parallelism-accuracy pairs S = {(ρi , yi )}m i=1 sorted 15 by increasing parallelism ρ1 < ρ2 < · · · < ρm , AUP is computed as: m   X yi W (yi ) + yi−1 W (yi−1 ) AUP ≜ ρ1 y1 + (ρi − ρi−1 ) , (26) i=2 2 where the weighting function W (y) = min(e−α(1−y/ymax ) , 1) penalizes accuracy degradation rel- ative to the best accuracy ymax achieved on that task. We use α = 3 as the default penalty factor. A minimum accuracy threshold ymin = y1 − 5 is applied to exclude regimes of severe accuracy degradation. AUP rewards methods that increase parallelism without sacrificing accuracy, while suppressing contributions from low-accuracy regimes. Importantly, AUP is hardware-independent since it relies on TPF rather than tokens per second (TPS), providing a fair comparison of algorithmic parallelism across different hardware setups. C.2.2 Evaluation Configurations We evaluate on four downstream benchmarks covering math reasoning and code generation. All evaluations are conducted using the lm-evaluation-harness framework. Due to differences in model architecture and instruction-following capabilities, TAD-LLaDA and TAD-Dream use slightly different task variants, as summarized in Table 8. During inference, we set the maximum generation length to 256 tokens for all tasks. We use greedy decoding with temperature set to 0.0. The block size is fixed at 32 tokens for all experiments. For our TAD framework with multi-block generation, the block-add threshold is set to 0.1, and the decoded token threshold is 0.95. ``` ### Table 8: Evaluation configurations for TAD-LLaDA and TAD-Dream. ```text Table 8: Evaluation configurations for TAD-LLaDA and TAD-Dream. Model Benchmark Few-shot Gen Length Block Size Entropy threshold GSM8K 0-shot 256 32 0.5 MATH (Minerva) 4-shot 256 32 0.5 TAD-LLaDA HumanEval 0-shot 256 32 0.5 MBPP 3-shot 256 32 0.5 GSM8K 0-shot 256 32 0.5 MATH (Minerva) 4-shot 256 32 0.5 TAD-Dream HumanEval-Instruct 0-shot 256 32 0.5 MBPP-Instruct 0-shot 256 32 0.5 D More Experiment Results D.1 Impact of Privileged Information on Trajectory Collection In this section, we investigate the quality of the intermediate trajectories collected during the data construction phase. Specifically, we evaluate the generative accuracy of the teacher models under two distinct rollout conditions: (1) standard self-generation, where the model is conditioned solely on the input question without access to the ground truth (w/o GT), and (2) generation with privileged information, where the model receives both the input question and the ground-truth response. As presented in Table 9, the inclusion of ground-truth information yields a substantial and consistent improvement in trajectory accuracy across all evaluated benchmarks for both LLaDA-8B-Instruct and Dream-7B-Instruct architectures. For instance, on the GSM8K Training Set, the privileged information guidance improves LLaDA’s accuracy from 68.88% to 89.16%, and Dream’s from 75.17% to 94.79%. This performance confirms that leveraging ground truth as privileged information is beneficial to sample the high-quality intermediate states required for effective temporal-aware self-distillation. D.2 Throughput Analysis To further validate the practical deployment efficiency of the TAD framework, we conduct a wall- clock throughput analysis on the GSM8K-CoT benchmark using NVIDIA H200 GPUs. We measure 16 ``` ### Table 9: Performance comparison of dLLMs with and without Ground Truth (GT). ```text Table 9: Performance comparison of dLLMs with and without Ground Truth (GT). Model Condition GSM8K KodCode w/o GT 68.88 8.22 LLaDA-8B-Instruct with GT 89.16 77.28 w/o GT 75.17 14.05 Dream-7B-Instruct with GT 94.79 75.08 the generation speed in tokens per second (TPS) and compare it against the base models and strong acceleration baselines. The results for LLaDA and Dream architectures are presented in Table 10. Results on LLaDA Architecture As shown in Table 10, TAD demonstrates a remarkable improve- ment over the baseline and existing acceleration methods on the LLaDA architecture. The Quality mode (TAD-LLaDA-Q) achieves 339.4 TPS, a 10.9-fold speedup over LLaDa, while simultaneously improving the accuracy from 72.6% to 79.9%. Furthermore, the Speed mode (TAD-LLaDA-S) maximizes hardware utilization, reaching a peak throughput of 451.8 TPS (a 14.5-fold acceleration) while maintaining a robust accuracy of 78.8%. Results on Dream Architecture The Dream base model inherently possesses a highly optimized initial state, achieving 83.9% accuracy on GSM8K-CoT. As illustrated in Table 10, all acceleration methods incur a minor accuracy penalty on this architecture. TAD-Dream-Q achieves 205.7 TPS (a 5.1-fold speedup) while preserving a highly competitive accuracy of 81.4%. TAD-Dream-S pushes the throughput to 288.4 TPS (a 7.2-fold speedup) with a marginal drop to 81.0% accuracy. We observe that while the absolute peak TPS of TAD-Dream-S is slightly lower than that of d3LLM [9] (295.0 TPS), TAD matches its accuracy in the Quality mode. ``` ### Table 10: Throughput comparison of TAD-LLaDA and TAD-Dream on GSM8K-CoT using H200 ```text Table 10: Throughput comparison of TAD-LLaDA and TAD-Dream on GSM8K-CoT using H200 GPUs. We report tokens per second (TPS) and accuracy (%). Speedup ratios relative to the respective base models (LLaDA and Dream) are shown in parentheses. Method H200 TPS ↑ Acc (%) ↑ Method H200 TPS ↑ Acc (%) ↑ LLaDA 31.2 (1.0×) 72.6 Dream 40.1 (1.0×) 83.9 Fast-dLLM-LLaDA 137.3 (4.4×) 74.7 Fast-dLLM-Dream 117.8 (2.9×) 79.0 D2F-LLaDA 121.1 (3.9×) 73.2 Fast-dLLM-v2 169.3 (4.2×) 77.5 dParallel-LLaDA 214.3 (6.9×) 72.6 dParallel-Dream 189.4 (4.7×) 82.1 d3LLM-LLaDA 328.2 (10.5×) 73.1 d3LLM-Dream 295.0 (7.4×) 81.4 TAD-LLaDA-Q 339.4 (10.9×) 79.9 TAD-Dream-Q 205.7 (5.1×) 81.4 TAD-LLaDA-S 451.8 (14.5×) 78.8 TAD-Dream-S 288.4 (7.2×) 81.0 17 ``` ## reference/beyond single token/Beyond Single Tokens- Distilling Discrete Diffusion Models via Discrete MMD ### Table 1 | Overview of D-MMD generators and base model sample quality for different NFEs on ```text Table 1 | Overview of D-MMD generators and base model sample quality for different NFEs on CIFAR10. Results are measured in FID with 50K samples compared to the train dataset. D-MMD models substantially outperform their teacher while using a fraction of the NFEs. Model 8 16 32 64 128 256 512 Uniform Teacher ( 𝑝 = 0.50) 0.375 0.326 0.330 0.324 0.313 Uniform D-MMD ( 𝑝 = 0.70/0.70) 0.337 0.310 0.307 0.316 Masked Teacher ( 𝑝 = 0.85) 0.402 0.307 0.297 0.275 0.275 Masked D-MMD ( 𝑝 = 0.85) 0.456 0.236 0.225 0.231 AR Baseline 0.061 ``` ### Table 2 | Text D-MMD generators and base model sample quality for different NFEs measured in ```text Table 2 | Text D-MMD generators and base model sample quality for different NFEs measured in GPT-2 GM (see section 5). D-MMD models outperform their teacher at a fraction of the NFEs. 6.1. CIFAR-10 In this first set of experiments we train diffusion models to generate unconditional images. The models are trained on the 32x32x3 images in the CIFAR10 dataset. We train a model directly on the {0, . . . , 255}32×32×3 pixel values, resulting in a total of 3072 tokens that need to be generated. We evaluate the performance using the FID metric, which notwithstanding the flaws, is still one of the better metrics to measure distances between distributions of (generated) images. On this dataset we train a masked and uniform diffusion model. These models tend to perform worse than standard diffusion models because there is no inductive bias: every pixel value is a unique token in the vocabulary. The uniform diffusion teacher achieves an FID of 7.5 and the masked diffusion teacher an FID of 6.4 using 1024 denoising steps.2 Impressively, D-MMD is able to distill much better generators at only a fraction of the denoising steps compared to the original teacher (Table 1). For uniform diffusion models an FID of 3.7 is achieved in 32 steps versus an FID of 7.5 for a 1024-step teacher. For Masked diffusion models, the distilled generator outperforms the teacher with 16 steps, and obtains an FID of 3.5 with only 64 uniform denoising steps. In conclusion, both uniform and masked D-MMD achieve a substantially better Pareto front of steps vs FID than their teachers. 6.2. Text For text generation we train on Open Web Text (OWT) and take the last 2% as a validation set. Because generative perplexity can be gamed by lower temperature sampling (either intentionally or unintentionally through biased samplers), we use the GPT-2 GM metric to measure distance from the distribution. 2 Note: continuous (standard) diffusion models easily obtain an FID of around 3 (Ho et al., 2020). 9 Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD Model 16 256 256-Block Uniform Teacher ( 𝑝 = 0.9) - 0.225 256-Block Uniform D-MMD ( 𝑝 = 0.7) 0.225 - ``` ### Table 3 | Block auto-regressive diffusion model with block size 256. 16-step D-MMD matches the ```text Table 3 | Block auto-regressive diffusion model with block size 256. 16-step D-MMD matches the performance of the 256-step teacher. Method NFE FID Di4C Teacher 40 8.0 Di4C (hybrid) 20 9.5 Di4C 10 20.6 Uniform Teacher 512 7.6 64 10.7 Uniform D-MMD 8 5.0 16 4.1 32 3.7 Masked Teacher 512 6.7 64 20.0 Masked D-MMD 16 5.3 32 3.8 64 3.5 ``` ### Table 4 | Comparison with literature on CIFAR10. D-MMD considerably outperforms existing methods ```text Table 4 | Comparison with literature on CIFAR10. D-MMD considerably outperforms existing methods using fewer NFEs. Similar to image experiments, we train masked and uniform diffusion teacher models and measure their performance by generating 1024 tokens unconditionally using increasing number of denoising steps. We tune the top-p value for the best GPT-2 GM. The results are in Table 2. The Masked D-MMD generator already outperforms the teacher using only 16 steps, achieving 0.236 GPT-2 GM. Similar to the results for images, both the uniform and masked generators consistently outperform their teacher counterparts and improve the whole Pareto front. 6.3. Block autoregressive diffusion Rather than generating an entire sequence at once, a more realistic setup would be to use a diffusion model to generate a limited block of tokens conditioned on an auto-regressive encoder. This combines the training efficiency and efficient inference of an AR model with the parallel sampling of diffusion. In this experiment, the 16-step D-MMD generator matches the performance of the 256-step teacher (see Table 3). 6.4. Comparison related work In this section we compare to the discrete diffusion distillation literature. For Di4C, results in the main paper are available on CIFAR10. Note that Di4C is actually at an advantage here, because its teacher model is trained using a discrete process that mimics the destruction of a Gaussian process. As a result, Di4C is able to achieve a teacher FID of 8.0 using only 40 steps. Nevertheless, because D-MMD outperforms the teacher models it still outperforms Di4C with 5.0 using only 8 steps with 10 Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD Method NFE GPT-2 GM ↓ GPT2 Perplexity ↓ Sample entropy Duo + DCD 4 108.2 4.82 Duo + Di4C 4 150.7 4.81 MDLM + SDTT 4 339.7 5.38 MDLM + Di4C 4 239.3 5.40 FMLM 4 76.4 5.05 Masked Teacher 256 0.275 22.5 5.13 128 0.295 23.9 5.17 64 0.307 26.0 5.19 SDTT (reimpl.) 64 0.293 26.9 5.17 32 0.340 30.4 5.18 Masked D-MMD 4 0.820 20.3 4.60 16 0.236 17.2 5.00 32 0.225 19.4 5.05 Data 0.000 15.4 5.44 Masked Teacher 256 0.672 85.9 5.59 ( 𝑝 = 1.0) 128 0.711 91.1 5.61 64 0.781 101.0 5.63 Masked D-MMD 4 0.719 66.1 5.44 ( 𝑝 = 1.0) 16 0.558 67.7 5.57 32 0.578 72.1 5.57 ``` ### Table 5 | Comparison with literature on OWT measured in GPT-2 GM (lower is better), generative ```text Table 5 | Comparison with literature on OWT measured in GPT-2 GM (lower is better), generative perplexity (should not be too high) and sample entropy (should not be too low). D-MMD is able to achieve even better results in fewer steps. the uniform generator (see Table 4). Recall that a metric such as generative perplexity is roughly measuring your distance from a mode, and collapsed models can easily score generative perplexities near 1.03 (the optimum). Instead, we measure performance with GPT-2 Gradient Moment (GPT-2 GM), which is somewhat more robust to this. Here we do see that even though SDTT improves upon the teacher model, it still degrades over repeated distillation rounds and is outperformed by D-MMD (see Table 5). Especially the GPT-2 GM metric highlights this degradation. The optimal top-p was chosen at 𝑝 = 0.85 by sweeping, measuring GPT-2 GM on the masked teacher. SDTT (reimpl.) and D-MMD use the same teacher. For completeness, we also show results without top-p 𝑝 = 1.0. For other related works, (Sahoo et al., 2025; Roos et al., 2026) the results were taken from (Lee et al., 2026). 6.5. Conditioning the generator on input noise In theory the generator should have access to a noise source to be able to generate a distribution. However, in Salimans et al. (2024) it was noted that in practice no input noise is required for Gaussian diffusion distillation. However, in the case of 1-step masked generation (Zhu et al., 2025) noise conditioning turned out to be important. For images we learn a projection of a 2D Gaussian noise pyramid to be added to the residual. For text we learn a projection of plain Gaussian noise. In our case we find that masked distillation performs much better with an extra noise source (see 3 For example the sentence "hahahahahahaha" repeated also has a perplexity near 1.0 11 Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD D-MMD Masked 4 8 16 32 64 without noise (FID) 151.0 37.0 14.7 7.7 6.0 (generator output entropy) 1.26 1.37 1.57 1.86 1.91 with noise (FID) 22.3 12.7 5.3 3.8 3.5 (generator output entropy) 1.01 1.29 1.53 1.76 1.83 ``` ### Table 6 | Noise input conditioning is important for masked distillation. Fewer steps require more ```text Table 6 | Noise input conditioning is important for masked distillation. Fewer steps require more generator output collapse, and generators with noise conditioning can collapse their factorized output distribution more. ``` ### Table 6). In that case, the generator is able to collapse its output distribution more and achieves ```text Table 6). In that case, the generator is able to collapse its output distribution more and achieves much better sample quality. In contrast, for uniform diffusion we did not observe any meaningful improvements. As is the case with Gaussian diffusion, for uniform diffusion there may already be sufficient noise in 𝑧𝑡 that the generator is able to use. All other masked distillation experiments in this paper condition on input noise. 6.6. Discussion on students outperforming teachers It may seem counterintuitive that students can outperform their teachers. However, teachers are trained using maximum likelihood which is known to be mode-covering. Mode-collapsing behavior is often induced by reducing temperature or top-p sampling. Many distillation approaches such as D-MMD have an adversarial component and generate samples based on the student, which both are reminiscent of reverse-KL optimization. D-MMD may move more density towards modes without fully collapsing, which is typically desired for samples from an image or language generator. A paradoxical side-effect is the following: suppose the student is better than the teacher for a certain number of steps. Then, the student’s performance will degrade at some point even as sampling steps increase, as that performance will converge to the teacher’s at high step counts. 7. Conclusions In summary, D-MMD is a new technique that allows for a principled way to distill discrete diffusion processes into few-step generators. In experiments, generators tend to outperform their teachers considerably, using only a fraction of the denoising steps. 12 Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD References M. Arriola, A. Gokaslan, J. T. Chiu, Z. Yang, Z. Qi, J. Han, S. S. Sahoo, and V. Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025. J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg. Structured denoising diffusion models in discrete state-spaces. CoRR, abs/2107.03006, 2021. L. Azzopardi, M. Girolami, and C. J. Van Rijsbergen. Investigating the relationship between language model perplexity and ir precision-recall measures. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 369–370, 2003. A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799, 2020. T. Chen, R. Zhang, and G. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022. J. Deschenaux and C. Gulcehre. Beyond autoregression: Fast llms via self-distillation through time. In The Thirteenth International Conference on Learning Representations, ICLR. OpenReview.net, 2025. I. Gat, T. Remez, N. Shaul, F. Kreuk, R. T. Q. Chen, G. Synnaeve, Y. Adi, and Y. Lipman. Discrete flow matching. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS, 2024. S. Hayakawa, Y. Takida, M. Imaizumi, H. Wakaki, and Y. Mitsufuji. Distillation of discrete diffusion through dimensional correlations. CoRR, abs/2410.08709, 2024. J. Heek, E. Hoogeboom, and T. Salimans. Multistep consistency models. Technical report, GDM, 2024. URL https://arxiv.org/abs/2403.06807. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, volume 30, 2017. J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS, 2020. E. Hoogeboom, D. Nielsen, P. Jaini, P. Forré, and M. Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. CoRR, abs/2102.05379, 2021. D. Kim, C. Lai, W. Liao, N. Murata, Y. Takida, T. Uesaka, Y. He, Y. Mitsufuji, and S. Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. CoRR, abs/2310.02279, 2023. C. Lee, J. Yoo, M. Agarwal, S. Shah, J. Huang, A. Raghunathan, S. Hong, N. M. Boffi, and J. Kim. One-step language modeling via continuous denoising. arXiv preprint arXiv:2602.16813, 2026. D. Li, N. Gushchin, D. Abulkhanov, E. Moulines, I. Oseledets, M. Panov, and A. Korotin. Idlm: Inverse-distilled diffusion language models. arXiv preprint arXiv:2602.19066, 2026. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda. OpenReview.net, 2023. 13 Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD ``` ## reference/dflash/DFlash- Block Diffusion for Flash Speculative Decoding ### Table 1. Decoding speedup over baseline and average acceptance length (τ ) on Qwen3 models with thinking mode disabled and a ```text Table 1. Decoding speedup over baseline and average acceptance length (τ ) on Qwen3 models with thinking mode disabled and a maximum of 2048 generated tokens. Parenthesized values indicate the draft tree size for EAGLE-3 and the diffusion block size for DFlash. M ATH C ODE C HAT Model Method GSM8K MATH-500 AIME25 HumanEval MBPP LCB MT-Bench Avg. Temperature = 0 Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ EAGLE-3 (16) 1.99× 3.30 1.83× 3.08 1.79× 3.05 1.84× 3.05 1.78× 2.95 1.73× 2.91 1.74× 3.02 1.81× 3.05 Q3-4B EAGLE-3 (60) 2.27× 3.77 2.10× 3.52 2.13× 3.51 2.12× 3.47 2.02× 3.38 1.90× 3.22 2.04× 3.49 2.08× 3.48 DFlash (16) 5.15× 6.53 6.09× 7.84 5.68× 7.27 5.21× 6.64 4.78× 6.09 5.41× 7.09 2.85× 4.35 4.91× 6.54 EAGLE-3 (16) 1.94× 3.23 1.81× 3.02 1.79× 3.00 1.89× 3.17 1.69× 2.82 1.57× 2.65 1.63× 2.83 1.76× 2.96 Q3-8B EAGLE-3 (60) 2.23× 3.71 2.05× 3.49 2.05× 3.44 2.17× 3.65 1.93× 3.25 1.81× 3.03 1.90× 3.26 2.02× 3.40 DFlash (16) 5.15× 6.54 6.08× 7.87 5.62× 7.08 5.14× 6.50 4.65× 5.95 5.51× 7.27 2.75× 4.24 4.86× 6.49 Temperature = 1 Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ Speedup τ EAGLE-3 (16) 1.89× 3.22 1.75× 2.99 1.64× 2.79 1.74× 3.01 1.69× 2.89 1.63× 2.77 1.70× 2.95 1.72× 2.95 Q3-4B EAGLE-3 (60) 2.12× 3.68 1.97× 3.44 1.83× 3.20 1.94× 3.39 1.92× 3.33 1.82× 3.14 1.91× 3.36 1.93× 3.36 DFlash (16) 4.71× 6.00 5.09× 6.67 3.73× 4.92 4.74× 6.04 4.42× 5.66 4.90× 6.50 2.67× 4.07 4.24× 5.69 EAGLE-3 (16) 1.87× 3.12 1.73× 2.91 1.63× 2.74 1.75× 3.05 1.64× 2.74 1.56× 2.57 1.58× 2.70 1.68× 2.83 Q3-8B EAGLE-3 (60) 2.07× 3.59 1.94× 3.38 1.84× 3.18 2.05× 3.54 1.85× 3.16 1.72× 2.92 1.70× 3.05 1.88× 3.26 DFlash (16) 4.67× 5.98 4.84× 6.40 3.57× 4.73 4.32× 5.52 4.04× 5.21 4.93× 6.69 2.47× 3.80 4.03× 5.48 5. Experiments Li et al., 2025a; Sandler et al., 2025) due to lack of open- source implementation. For comparisons with EAGLE-3 Models and Evaluations. We conduct experiments on on Qwen3 models (Section 5.1), we use the checkpoints LLaMA-3.1 Instruct (8B) and Qwen3 (4B, 8B, Coder-30B- released by AngelSlim (Tencent, 2025); for LLaMA-3.1- A3B-Instruct) pre-trained models. We evaluate tasks in Instruct (Section 5.4.1), we use the official checkpoint re- three categories: Math: GSM8K (Cobbe et al., 2021), leased by EAGLE-3 team. MATH (Lightman et al., 2023), and AIME25 (MAA, 2025); Code: HumanEval (Chen, 2021), MBPP (Austin et al., 5.1. Instruct Models 2021), and LiveCodeBench (Jain et al., 2024); Chat: MT- Bench (Zheng et al., 2023) and Alpaca (Taori et al., 2023). In this section, we evaluate DFlash against EAGLE-3 on For each task, we assess the performance of the draft models Qwen3 models with thinking mode disabled, using the using average acceptance length (τ ) and end-to-end decod- Transformers backend. For EAGLE-3, we consider two ing speedup over the autoregressive baseline. We conduct settings: a tree size of 16, which matches DFlash with block all experiments on NVIDIA H200 GPUs unless otherwise size 16 for a fair drafting-budget comparison, and a tree size specified. of 60, as used in the EAGLE-3 paper to maximize accep- tance length with higher verification cost. In both cases, the Datasets. To provide a diverse set of training data, we draft steps and top-k are set to 7 and 10, respectively. collect a mixture of around 800K samples from NVIDIA Nemotron Post-Training Dataset V2 (Nathawani et al., 2025) As shown in Table 1, DFlash consistently outperforms and CodeAlpaca (Chaudhary, 2023). Instead of directly EAGLE-3 across all tasks and settings. Under greedy decod- using the original dataset, we construct our training set with ing (temperature = 0), DFlash achieves an average speedup the responses generated by the target model for better target of 4.9× over the autoregressive baseline, corresponding alignment. to a 2.4× improvement over EAGLE-3 (16). Under non- greedy sampling (temperature = 1), DFlash maintains a Implementation. For DFlash draft models, we set the 4.1× speedup over baseline and a 2.2× improvement over number of layers to 5 (8 for Qwen3 Coder) and use a block EAGLE-3. Notably, DFlash also surpasses EAGLE-3 with size of 16 (10 for LLaMA 3.1). The target hidden features tree size 60, achieving higher acceptance length while incur- are extracted from 5 layers uniformly selected between the ring substantially lower verification overhead. These results second layer and the third-to-last layer of the target model. demonstrate the effectiveness and efficiency of diffusion- More training details are presented in Section A.1. based drafting in DFlash. Baselines. We compare DFlash with the vanilla autore- gressive decoding (baseline) and state-of-the-art speculative 5.2. Reasoning Models decoding method EAGLE-3 (Li et al., 2025b). We did not include comparisons with other dLLM-based speculative In this section, we evaluate DFlash for Qwen3 models with decoding methods (Liu et al., 2025; Samragh et al., 2025; thinking mode enabled using Transformers. The draft mod- 6 DFlash : Block Diffusion for Flash Speculative Decoding els are trained on target-model outputs with reasoning traces. GPU with the FlashAttention-4 (FA4) backend. We en- able Spec-v2 scheduling overlap to maximize achievable As shown in Table 2, DFlash maintains the high acceptance throughput. length, achieving speedups of roughly 4.5× and 3.9× over the baseline. This efficiency gain is particularly valuable for As shown in Table 3, DFlash consistently provides speedups the practical deployment of reasoning models, given their across all three models over concurrency levels ranging prolonged generation time. from 1 to 32, achieving up to a 5.1× speedup on Qwen3- ``` ### Table 2. Decoding speedup over baseline and average acceptance in real-world serving scenarios, where it can substantially ```text Table 2. Decoding speedup over baseline and average acceptance in real-world serving scenarios, where it can substantially length (τ ) with thinking mode enabled. reduce serving cost. GPQA MATH-500 AIME25 Model Temp. Speedup τ Speedup τ Speedup τ 5.4. Ablation Study 0 4.23× 5.23 4.59× 5.74 4.39× 5.54 In this section, we ablate the impact of training data and Q3-4B 1 3.67× 4.55 3.93× 4.89 3.64× 4.68 several key design choices of the DFlash draft model. Un- 0 4.17× 5.17 4.64× 5.82 4.51× 5.74 Q3-8B 1 3.75× 4.65 4.03× 5.06 3.70× 4.69 less otherwise specified, all ablation models are trained on 100K samples randomly drawn from the full data mixture. All experiments are conducted on a single H200 GPU with 5.3. Performance on SGLang greedy decoding, except those evaluated on SGLang. ``` ### Table 3. Throughput (tok/s), speedup over baseline, and average 5.4.1. T RAINING DATA ```text Table 3. Throughput (tok/s), speedup over baseline, and average 5.4.1. T RAINING DATA acceptance length τ on SGLang (FA4 backend). ``` ### Table 4. Speedup over baseline and average acceptance length ```text Table 4. Speedup over baseline and average acceptance length Concurrency τ for LLaMA-3.1-8B-Instruct on SGLang (Flashinfer backend, Task Method Avg. τ 1 4 8 16 32 single B200 GPU). Baseline reports absolute throughput (TPS; tokens/s). EAGLE-3 uses 7 draft steps with top-k=10 and either Qwen3-4B 10 or 60 draft tokens. DFlash uses block size 10. Baseline 316 1145 2201 4100 7136 – Math500 1531 4943 9066 14477 20417 Concurrency DFlash 4.8× 4.3× 4.1× 3.5× 2.9× 8.01 Method Avg. τ 1 4 8 16 32 Baseline 312 1162 2217 4184 7143 – GSM8K Human- Eval 1247 4147 6997 11234 15703 DFlash 4.0× 3.6× 3.2× 2.7× 2.2× 6.63 Baseline (TPS) 249 923 1739 3245 5349 – EAGLE-3 (10) 1.6× 1.5× 1.4× 1.2× 1.0× 3.49 Qwen3-8B EAGLE-3 (60) 1.9× 1.6× 1.3× 0.9× 0.6× 4.55 Baseline 230 861 1666 3133 5694 – DFlash (10) 2.4× 2.2× 2.1× 1.8× 1.6× 4.32 Math500 1175 3884 7485 12268 16076 HumanEval DFlash 5.1× 4.5× 4.5× 3.9× 2.8× 8.01 Baseline (TPS) 245 922 1778 3336 5854 – Baseline 229 868 1649 3253 5462 – EAGLE-3 (10) 2.0× 1.9× 1.8× 1.5× 1.2× 3.62 Human- Eval 955 3092 6010 9919 13116 EAGLE-3 (60) 2.0× 1.7× 1.3× 0.9× 0.6× 4.65 DFlash 4.2× 3.6× 3.6× 3.0× 2.4× 6.50 DFlash (10) 2.8× 2.6× 2.5× 2.1× 1.8× 4.91 Qwen3-Coder-30B-A3B Alpaca Baseline 229 686 1068 1681 2713 – Baseline (TPS) 245 906 1745 3237 5434 – Human- Eval 802 2078 3442 5429 8314 EAGLE-3 (10) 1.5× 1.4× 1.4× 1.1× 0.9× 3.11 DFlash 3.5× 3.0× 3.2× 3.2× 3.1× 8.09 EAGLE-3 (60) 1.8× 1.5× 1.2× 0.8× 0.5× 4.07 Baseline 220 681 1112 1733 2823 – DFlash (10) 2.2× 2.0× 1.8× 1.5× 1.4× 3.73 LCB 569 1621 2554 4160 6401 DFlash 2.6× 2.4× 2.3× 2.4× 2.3× 6.42 We compare DFlash against EAGLE-3 on LLaMA-3.1- Baseline 228 682 1057 1697 2735 – 8B-Instruct. DFlash is trained on UltraChat (Ding et al., MBPP 720 2052 3360 5522 8538 2023) and ShareGPT, using the exactly same training data DFlash 3.2× 3.0× 3.2× 3.3× 3.1× 7.23 as EAGLE-3, and is evaluated against the official EAGLE-3 checkpoints. The DFlash draft model uses a block size of In this section, we evaluate the performance of DFlash on 10, with other configurations matching those of the DFlash the popular open-source inference framework SGLang us- Qwen3-8B draft model. All experiments are conducted us- ing Qwen3-4B, Qwen3-8B, and Qwen3-Coder-30B-A3B- ing SGLang with Spec-v1 (without scheduling overlap), as Instruct. All experiments are conducted on a single B200 Spec-v2 does not support tree-based drafting for EAGLE-3. 7 DFlash : Block Diffusion for Flash Speculative Decoding Evaluations are performed on a single B200 GPU. Table 7. Ablation study of training–inference block size (BS) mismatch. All draft models use 8 layers and 5 target hidden As shown in Table 4, DFlash consistently outperforms features. EAGLE-3 across all tasks, concurrency levels, and EAGLE- Train Test Math500 HumanEval MT-Bench 3 tree-size configurations. This performance gap holds for math, code, and chat benchmarks, demonstrating the robust- BS BS Speedup τ Speedup τ Speedup τ ness and efficiency advantages of DFlash over autoregres- b16 b16 4.64x 6.33 3.96x 5.29 2.23x 3.50 sive tree-based speculative decoding. b16 b8 3.87x 5.09 3.39x 4.44 2.12x 3.18 b8 b16 3.78x 5.02 3.24x 4.28 2.09x 3.09 5.4.2. N UMBER OF D RAFT L AYERS b8 b8 3.97x 5.21 3.53x 4.61 2.22x 3.29 ``` ### Table 5. 5-layer draft model has the best average speedup. All ```text Table 5. 5-layer draft model has the best average speedup. All DFlash draft models are trained with block size 16 and hidden features extracted from 5 layers of the target model. trained DFlash model can generalize from its training-time block size to different block sizes during inference. To study Math500 HumanEval MT-Bench Setting this, we train two draft models with block sizes 8 and 16 Speedup τ Speedup τ Speedup τ on the same data and evaluate their inference-time scaling 3-L 4.69× 5.64 3.90× 4.61 2.38× 3.18 behavior, as shown in Table 7. 5-L 4.71× 5.99 3.96× 4.94 2.35× 3.37 8-L 4.64× 6.33 3.96× 5.29 2.23× 3.50 When training and inference block sizes match (8→8 and 16→16), the block-size-16 model achieves substantially One advantage of DFlash is that acceptance length scales higher acceptance lengths on math and coding tasks. Accep- effectively with the depth of the draft model. However, this tance histograms on Math500 reveal that the block-8 model comes with a trade-off between drafting cost and draft qual- frequently fully accepts entire blocks (35.7%), suggesting ity. Deeper draft models are more expressive and achieve that block size 8 is often underutilized. In contrast, the higher acceptance lengths, but they also incur higher draft- block-16 model exhibits a more spread-out acceptance dis- ing latency. As a result, the optimal number of layers de- tribution with higher average acceptance length, indicating pends on the deployment setting. As shown in Table 5, more effective use of larger blocks. while the 8-layer draft model achieves longer acceptance We further examine cross-block-size generalization at infer- lengths, the 5-layer model attains higher overall speedup ence time and observe a clear asymmetry. A model trained due to a better balance between drafting cost and quality. with a larger block size generalizes well to smaller inference- time block sizes: using block size 8 with a model trained at 5.4.3. N UMBER OF TARGET H IDDEN F EATURES block size 16 yields acceptance lengths close to those of a ``` ### Table 6. More hidden features from target model increases the model trained and evaluated at block size 8. However, the ```text Table 6. More hidden features from target model increases the model trained and evaluated at block size 8. However, the acceptance length. All DFlash draft models use 3 draft layers and reverse does not hold. are trained with block size 16. Overall, DFlash models trained with larger block sizes Math500 HumanEval MT-Bench generalize well to smaller inference-time block sizes. This Setting property enables dynamic block-size scheduling during in- Speedup τ Speedup τ Speedup τ 3-H 4.49× 5.38 3.80× 4.47 2.32× 3.07 ference to improve end-to-end efficiency. In practical serv- 5-H 4.69× 5.64 3.90× 4.61 2.38× 3.18 ing scenarios, large blocks can increase verification cost under compute-bound settings (e.g., large batch sizes); re- ducing the block size in such cases can therefore yield better The number of target hidden features affects both accep- overall speedup. We leave adaptive block-size scheduling tance length and end-to-end speedup. Extracting features to future work. from more target layers provides richer semantic and future- token information, improving draft quality. As shown in ``` ### Table 6, conditioning on five hidden features consistently 6. Conclusion ```text Table 6, conditioning on five hidden features consistently 6. Conclusion outperforms using three. However, this benefit comes at In this paper, we present DFlash, a diffusion-based specula- higher training cost: in offline training, the storage required tive decoding framework that rethinks the role of diffusion to cache target hidden states increases linearly with the num- language models in accelerating autoregressive LLM infer- ber of extracted features. ence. By confining diffusion models to the drafting stage, DFlash exploits their inherent parallelism while avoiding 5.4.4. T RAINING -I NFERENCE T IME B LOCK S IZE the quality degradation that has limited their standalone Block size is a critical design choice for the DFlash draft use. Conditioning the diffusion drafter on rich target-model model. An equally important question is whether a pre- context enables high acceptance rates, allowing DFlash to 8 DFlash : Block Diffusion for Flash Speculative Decoding significantly push inference speed beyond prior speculative Cheng, S., Bian, Y., Liu, D., Zhang, L., Yao, Q., Tian, decoding methods. Z., Wang, W., Guo, Q., Chen, K., Qi, B., and Zhou, B. Sdar: A synergistic diffusion-autoregression paradigm Beyond empirical improvements, DFlash suggests a new for scalable sequence generation, 2025. URL https: development paradigm for diffusion LLMs. Rather than //arxiv.org/abs/2510.06303. 1, 3 competing with autoregressive models in end-to-end gen- eration quality, diffusion models can serve as lightweight, Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., specialized drafters optimized for fast and accurate block Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, prediction. This reframing permits aggressive reduction in R., et al. Training verifiers to solve math word problems. denoising steps to maximize parallelism, while speculative arXiv preprint arXiv:2110.14168, 2021. 6 verification provides a principled guarantee of output qual- ity. We hope DFlash establishes diffusion-based drafting as Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, a practical and scalable paradigm for speculative decoding, Z., Sun, M., and Zhou, B. Enhancing chat language mod- advancing more efficient and accessible LLM deployment. els by scaling high-quality instructional conversations, 2023. URL https://arxiv.org/abs/2305.14233. 7 Acknowledgements Dong, J., Feng, B., Guessous, D., Liang, Y., and He, H. Flex attention: A programming model for generating The authors would like to express their sincere gratitude to optimized attention kernels, 2024. URL https://arxiv. David Wang for leading the fast and high-quality SGLang org/abs/2412.05496. 5 integration for DFlash, and to Richard Gong and other mem- bers of the Modal team for their strong engineering support. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, Their efforts were truly instrumental in enabling the practi- A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., cal, production-grade deployment of DFlash. Vaughan, A., et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. 2 We also gratefully acknowledge Modal, Yotta Labs, Eigen AI, and InnoMatrix for providing the compute resources Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., that made this work possible. Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. Deepseek-r1 incentivizes reasoning in llms through References reinforcement learning. Nature, 645(8081):633–638, September 2025. ISSN 1476-4687. doi: 10.1038/ An, Z., Bai, H., Liu, Z., Li, D., and Barsoum, E. Pard: Accel- s41586-025-09422-z. URL http://dx.doi.org/10. erating llm inference with low-cost parallel draft model 1038/s41586-025-09422-z. 1 adaptation, 2025. URL https://arxiv.org/abs/2504. 18583. 2, 4 Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Livecodebench: Holistic and contamination free eval- Han, J., Sahoo, S. S., and Kuleshov, V. Block diffusion: uation of large language models for code. arXiv preprint Interpolating between autoregressive and diffusion lan- arXiv:2403.07974, 2024. 6 guage models, 2025. URL https://arxiv.org/abs/ 2503.09573. 1, 2, 5 Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding, 2023. URL Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, https://arxiv.org/abs/2211.17192. 1, 2 H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv Li, G., Fu, Z., Fang, M., Zhao, Q., Tang, M., Yuan, C., preprint arXiv:2108.07732, 2021. 6 and Wang, J. Diffuspec: Unlocking diffusion language models for speculative decoding, 2025a. URL https: Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., //arxiv.org/abs/2510.02358. 2, 3, 6 and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024. URL Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle-2: Faster https://arxiv.org/abs/2401.10774. 1, 2 inference of language models with dynamic draft trees, 2024. URL https://arxiv.org/abs/2406.16858. 1, Chaudhary, S. Code alpaca: An instruction-following 2 llama model for code generation. https://github.com/ sahil280114/codealpaca, 2023. 6 Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle-3: Scaling up inference acceleration of large language models via Chen, M. Evaluating large language models trained on code. training-time test, 2025b. URL https://arxiv.org/ arXiv preprint arXiv:2107.03374, 2021. 6 abs/2503.01840. 1, 2, 6 9 ``` ### Table 8. A 5-layer block diffusion draft model without target ```text Table 8. A 5-layer block diffusion draft model without target context features. The draft model achieves only modest acceptance length speedup. GSM8K Math500 AIME24 AIME25 Temp Speedup / τ Speedup / τ Speedup / τ Speedup / τ 0 2.83 / 3.38 3.73 / 4.61 3.43 / 4.12 3.35 / 4.07 1 2.76 / 3.29 3.31 / 4.12 2.66 / 3.23 2.65 / 3.24 A.3. Further Ablations A.3.1. L OSS D ECAY We ablate the position-dependent loss decay introduced in Section 4.2. Specifically, we compare the default setting with exponentially decaying token weights against a vari- ant trained with uniform token weighting within each draft block. This study isolates the effect of emphasizing early- token accuracy during training. Results in Figure 5 show that applying loss decay leads faster and better convergency. Acceptance Length vs. Training Epoch 6.5 Acceptance Length (Math500) 6.0 5.5 5.0 4.5 With loss decay Without loss decay 1 2 3 4 5 6 7 8 9 Epoch Figure 5. The loss decay makes training converge faster and better. 11 ``` ## reference/dmax/Back on Track- Aligning Rewards and States for Reasoning in Diffusion Large Language Models ### Table 1: Performance Comparison on Four Benchmarks. Bold numbers indicate the top performance in each ```text Table 1: Performance Comparison on Four Benchmarks. Bold numbers indicate the top performance in each category. Underlined numbers denote the best performance among methods without additional Supervised Fine- Tuning (SFT). Green values show the absolute improvement of our method over the LLaDA-8B-Instruct baseline. “–” denotes unreported results, “×” denotes unsupported tasks, and † indicates our own re-implementation. Without additional SFT, our PAPO demonstrates superior performance. Figure 3: Reward Curves during RL Training. Comparison of our method (PAPO) and diffu-GRPO across four benchmarks. PAPO consistently achieves higher reward trajectories, demonstrating superior sample efficiency. GSM8K MATH500 Countdown Sudoku Model / Seq Len 128 256 512 128 256 512 128 256 512 128 256 512 LLaDA-8B-Instruct 68.7 76.7 78.2 26.0 32.4 36.2 20.7 19.5 16.0 11.7 6.7 5.5 + HR 73.8 80.1 80.2 32.2 36.8 36.4 51.9 56.6 55.5 25.3 22.6 18.7 + HR + SPR 74.0 81.3 79.8 32.2 37.6 38.0 51.6 60.6 58.2 25.8 23.6 18.9 + EHR + SPR 73.8 82.4 80.8 33.4 35.6 40.0 52.0 65.6 65.2 27.2 25.0 20.2 ``` ### Table 2: Ablation Study on the contributions of Step-Aware Process Rewards (SPR) and Entropy-Guided Historical ```text Table 2: Ablation Study on the contributions of Step-Aware Process Rewards (SPR) and Entropy-Guided Historical Re-enactment (EHR), where HR denotes Historical Re-enactment without entropy guidance. tablishing state-of-the-art results among direct RL 5.3 Ablation Study methods on the majority of tasks. The gains are especially pronounced on planning tasks, reach- We conduct a thorough ablation study to exam- ing up to 42.2% on Countdown and 16.1% on Su- ine the contribution of each component, as shown doku, where structured, multi-step credit assign- in Table 2. First, removing SPR and reverting to ment proves most critical. sparse, terminal-only rewards leads to a substantial performance drop, underscoring the necessity of dense, process-aware feedback for effective credit Training Dynamics. As shown in Figure 3, assignment. Second, replacing EHR with a sim- PAPO exhibits markedly superior training dynam- pler Historical Re-enactment (HR) that samples ics compared to the diffu-GRPO baseline, achiev- states uniformly also yields a noticeable perfor- ing a consistently higher reward trajectory with mance decline, demonstrating that training on au- a steeper ascent and lower volatility. This rapid thentic states alone is insufficient: prioritizing high- and stable improvement reflects the superior sam- entropy states where the policy is most uncertain is ple efficiency of PAPO, indicating that our method critical for maximizing learning efficiency. The full more effectively leverages the learning signal to PAPO model achieves the best overall performance, accelerate policy optimization. confirming that the strength of our approach lies in 6 Figure 4: Training Dynamics of Step-Aware Process Rewards. The consistent upward trend of the process reward across all benchmarks confirms that the policy is learning to generate higher-quality intermediate reasoning steps. (a) (b) (c) Figure 5: Analysis of Entropy-Guided Historical Re-enactment. (a) Average token-level entropy exhibits a clear downward trend as the number of sampling steps increases. (b) The impact of the entropy-weighting hyperparameter α on the reward trajectory. (c) Completion length dynamics of PAPO and diffu-GRPO on the GSM8K benchmark. directing granular, process-aware rewards toward 1-step 4-step 8-step 16-step the most informative historical states. Seq Len=128 73.8 74.2⋄0.4% 74.7⋄0.9% 75.4⋄1.6% GPU Hours 177 187⋄5.6% 200⋄13.0% 255⋄44% 6 Analysis ``` ### Table 3: Multi-step SPR Comparison on GSM8K. ```text Table 3: Multi-step SPR Comparison on GSM8K. 6.1 Effect of Step-Aware Process Rewards Increasing the lookahead steps yields diminishing accu- racy gains at prohibitive computational cost. ⋄ denotes Process reward drives structured reasoning. To the relative improvement over 1-step. validate the effectiveness of SPR, we track the av- erage process reward during training (Figure 4). The consistent upward trend confirms that the pol- icy progressively learns to produce higher-quality intermediate reasoning steps, rather than merely arriving at correct final answers by chance. This dense, step-level supervision directly addresses the “unstructured refinement” problem by incentivizing (a) Multi-step comparison (b) Early reward clipping coherent, step-by-step reasoning paths. Figure 6: SPR Fidelity-Efficiency Analysis. (a) Multi- Fidelity-Efficiency trade-off favors one-step step SPR training dynamics on GSM8K. (b) Clipping SPR. A natural question is whether one-step de- early-stage rewards (t < 32) degrades performance, noised predictions faithfully reflect generation qual- confirming the importance of early supervision. ity. We take GSM8K as a representative case to ex- amine the fidelity-efficiency trade-off. As shown in ``` ### Table 3 and Figure 6 (a), increasing the lookahead 6.2 Effect of Entropy-Guided Historical ```text Table 3 and Figure 6 (a), increasing the lookahead 6.2 Effect of Entropy-Guided Historical from 1-step to 16-step yields only marginal accu- Re-enactment racy gains (+1.6%) while incurring substantial over- Entropy-guided selection improves learning ef- head (+44% GPU hours), confirming that 1-step ficiency. The generative process exhibits a non- SPR achieves the optimal fidelity-efficiency trade- uniform entropy distribution across the denoising off. We further examine the role of early-stage pro- trajectory (Figure 5 (a)), with entropy decreasing as cess rewards. As shown in Figure 6 (b), clipping denoising progresses. Uniform timestep selection rewards for steps t < 32 leads to noticeable per- is thus inefficient, over-sampling low-entropy states formance degradation, confirming that early-stage where the policy is confident while under-sampling process rewards are critical for stable convergence. high-entropy states with the greatest learning po- 7 tential. EHR addresses this by probabilistically GSM8K MATH500 Countdown Sudoku Model / Seq Len 128 256 512 128 256 512 128 256 512 128 256 512 prioritizing updates on high-entropy states. LLaDA-1.5 69.8 79.4 81.1 29.0 32.4 35.4 21.5 21.1 20.7 12.4 8.7 7.3 Impact of hyperparameter α in EHR. We fur- diffu-GRPO 73.0 78.9 83.1 29.8 36.2 40.2 38.7 29.7 39.1 23.5 18.3 13.1 ther validate this design through an analysis of α Ours 75.9 81.7 81.8 31.2 36.6 39.4 40.6 68.8 66.0 44.8 26.8 24.2 (Figure 5 (b)). Without entropy guidance (α = 0), ``` ### Table 4: Generalization to LLaDA-1.5. PAPO consis- ```text Table 4: Generalization to LLaDA-1.5. PAPO consis- training yields the lowest reward and later-stage in- tently outperforms diffu-GRPO across all benchmarks, stability, as uniform sampling keeps updating well- with the largest gains on planning tasks. learned states. With entropy weighting (α > 0), re- ward and convergence stability both improve, con- HumanEval MBPP firming that prioritizing high-entropy states benefits Model / Seq Len 128 256 512 128 256 512 learning efficiency and training stability. LLaDA + SFT 21.3 32.3 32.9 40.1 39.7 41.2 Token Efficiency. The efficiency gain also man- diffu-GRPO 31.1 32.9 37.8 40.5 44.7 42.8 Ours 31.7 34.8 40.2 42.1 43.2 47.1 ifests in the generated outputs. As observed on the GSM8K benchmark (Figure 5 (c)), PAPO con- Table 5: Generalization to Code Task. PAPO achieves verges to shorter completions than diffu-GRPO competitive or superior performance on HumanEval and while achieving higher task performance, indicat- MBPP with execution-based rewards. ing improved token efficiency. 6.3 Cross-Domain Generalization Generalization across Backbones. We apply PAPO to the LLaDA-1.5 (Zhu et al., 2025a) model to assess cross-backbone transfer (Table 4). PAPO consistently outperforms diffu-GRPO on all bench- marks, with pronounced gains on planning tasks, (a) Impact of µ (b) Efficiency on code tasks demonstrating that the process-aligned optimiza- Figure 7: Training Efficiency Analysis. (a) Impact tion transfers effectively across model scales. of policy update steps µ on reward convergence. (b) Generalization across Task Domains. We fur- Reward convergence comparison on code generation ther evaluate PAPO on code generation (Table 5), tasks under the same GPU budgets. a domain involving more complex, execution- based reward functions. We train a model on the based rewards (Figure 7 (b)), PAPO still reaches KodCode-Light-RL-10K dataset (Xu et al., 2025). a higher reward in less wall time despite the extra PAPO achieves consistently competitive or superior cost per step. This result indicates that the process- results on both HumanEval and MBPP benchmarks, aligned learning signal more than compensates for confirming that our framework is not restricted to the per-step overhead and improves overall training mathematical and planning tasks. efficiency. A broader efficiency analysis across all 6.4 Efficiency tasks is provided in Appendix B. A practical concern is whether step-aware process 7 Conclusion rewards computation introduces prohibitive over- head. We analyze training efficiency along two In this paper, we identify two fundamental mis- dimensions. Firstly, Figure 7 (a) shows how the alignments when applying reinforcement learn- number of policy optimization updates per rollout ing to diffusion large language models (dLLMs): batch µ affects sample efficiency and stability. Rais- process-reward misalignment due to the reliance ing µ from 2 to 24 accelerates reward convergence on sparse, terminal rewards, and inefficient state under the same wall time, but µ = 24 becomes selection caused by training on unfaithful contexts unstable late in training and the reward declines. with uniform step sampling. To address these limi- With µ = 12, convergence remains stable, reaches tations, we propose Process Aligned Policy Opti- a higher final reward with faster wall-clock conver- mization (PAPO), a novel framework that holisti- gence. Secondly, relative to diffu-GRPO, PAPO cally aligns the RL update process with the model’s incurs additional per-step overhead from process authentic generative trajectory. At its core, PAPO reward computation. However, even in the most ex- features two synergistic modules: Step-Aware Pro- pensive setting of code generation with execution- cess Rewards (SPR) for dense, step-specific credit 8 assignment, and Entropy-Guided Historical Re- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, enactment (EHR) for efficient, authentic state se- Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro lection. Extensive experiments on four reasoning Nakano, and 1 others. 2021. Training verifiers benchmarks demonstrate that PAPO significantly to solve math word problems. arXiv preprint outperforms strong baselines. We believe our pro- arXiv:2110.14168. cess aligned approach offers a more stable and ef- Ziyun Dai, Xiaoqiang Li, Shaohua Zhang, Yuanchen ficient path to enhancing the complex reasoning Wu, and Jide Li. 2025. See different, think better: capabilities of dLLMs. Visual variations mitigating hallucinations in lvlms. In Proceedings of the 33rd ACM International Con- Limitations ference on Multimedia, MM ’25, page 3310–3319. ACM. While PAPO achieves strong performance on text- Huilin Deng, Hongchen Luo, Yue Zhu, Long Li, based reasoning, two extensions remain open. Pro- Zhuoyue Chen, Xinghao Zhao, Ming Li, Jihai Zhang, cess reward relies on a one-step denoised predic- Mengchang Wang, Yang Cao, and Yu Kang. 2026. tion x̂0 (t) under multi-step mask diffusion rollout. Iib-lpo: Latent policy optimization via iterative infor- As few-step distillation (e.g. T3D (Zhang et al., mation bottleneck. Preprint, arXiv:2601.05870. 2026)), and training-free fast decoders (e.g. Fast- Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang dLLM (Wu et al., 2025)) reduce the number of Cao, and Yu Kang. 2025. Boosting the general- ``` ## reference/dmax/Few-Step Diffusion Language Models via Trajectory Self-Distillation ### Table 1: Few-step accuracy comparison across baselines on SDAR-1.7B-Chat and SDAR-4B- ```text Table 1: Few-step accuracy comparison across baselines on SDAR-1.7B-Chat and SDAR-4B- Chat [8]. Few-step performance is evaluated using tokens-per-step (TokPS): for example, Block Size= 4 and TokPS= 2 means decoding uses blocks of 4 tokens while generating 2 tokens per diffusion step, resulting in 4/2 = 2 diffusion steps per block. SD means Self-Distillation methods. T3D is consistently among the strongest methods, demonstrating the effectiveness of trajectory- level distillation for few-step generation. Block Size = 4 Block Size = 8 TokPS Method SD AVG. Gains (%) MATH500 GSM8K MBPP HumanEval MATH500 GSM8K MBPP HumanEval SDAR-1.7B-Chat Original Model - 39.40 63.00 30.40 32.93 33.60 55.88 27.80 37.20 40.03 - SFT ✗ 43.00 61.79 30.00 34.76 36.80 62.55 27.20 37.80 41.74 ↑ 4.28 2 ReDi ✓ 40.60 63.99 13.20 16.46 36.40 62.17 12.80 13.41 32.38 ↓ 19.11 dParallel ✓ 43.40 68.23 22.20 24.39 45.20 67.70 23.20 26.83 40.14 ↑ 0.29 T3D (Ours) ✓ 47.00 70.96 27.20 30.49 47.80 68.84 26.60 25.61 43.06 ↑ 7.59 Original Model - 5.00 13.34 10.60 12.20 4.80 12.74 10.20 10.37 9.91 - SFT ✗ 22.40 36.62 6.20 5.49 20.00 39.65 4.40 7.93 17.84 ↑ 80.05 4 ReDi ✓ 15.00 32.45 3.40 5.49 12.80 29.72 4.00 4.88 13.47 ↑ 35.95 dParallel ✓ 22.80 45.26 10.20 12.20 25.40 42.91 10.40 11.59 22.60 ↑ 128.09 T3D (Ours) ✓ 25.60 42.91 9.40 15.24 24.40 37.38 9.20 14.02 22.27 ↑ 124.79 SDAR-4B-Chat Original Model - 54.40 78.77 34.20 49.39 49.60 72.33 33.40 46.95 52.38 - SFT ✗ 54.60 54.60 26.80 37.20 54.44 77.41 25.60 29.88 46.76 ↓ 10.73 2 ReDi ✓ 41.00 73.62 20.00 21.95 23.60 71.87 19.20 23.17 36.80 ↓ 29.74 dParallel ✓ 52.60 76.57 23.80 39.63 51.20 75.97 18.20 28.66 45.83 ↓ 12.51 T3D (Ours) ✓ 60.00 83.85 38.80 51.83 61.60 81.96 37.00 56.10 58.89 ↑ 12.43 Original Model - 13.80 41.09 14.00 18.29 16.80 41.02 10.00 16.46 21.43 - SFT ✗ 39.00 48.14 9.00 15.85 40.20 55.42 8.80 11.59 28.50 ↑ 32.98 4 ReDi ✓ 25.40 53.30 5.00 7.32 20.20 47.84 6.80 6.71 21.57 ↑ 0.65 dParallel ✓ 34.20 45.94 13.20 20.73 40.80 53.83 9.60 20.12 29.80 ↑ 39.05 T3D (Ours) ✓ 47.80 69.90 22.60 23.78 44.80 63.99 21.20 23.17 39.66 ↑ 85.02 Final objective. Our full method, T3D (Trajectory self-Distillation via DDO), first collects teacher- generated trajectories and then trains a few-step student using DDO together with path-consistency regularization. The final training objective is LT3D (θ) = Ltraj-DDO (θ) + λ Lpath (θ), (13) where λ controls the strength of the path-consistency regularization. Figure 3 provides an overview of the framework, and the full training algorithm is given in Appendix A. 5 Experiments 5.1 Experimental Settings Baselines. We compare T3D with representative few-step diffusion language model baselines: ReDi [46], dParallel [6], and SFT on real data as a supervised reference. For LLaDA experiments, we additionally include CDLM [18], which accelerates diffusion language models through system-level and training-based designs. All training-based baselines and T3D are trained until convergence. Models and Benchmarks. We evaluate T3D on both block-diffusion and full-diffusion language models. For block diffusion, we use SDAR-1.7B-Chat and SDAR-4B-Chat [8]; for full diffusion, we use LLaDA-8B-Instruct [27]. We evaluate on four reasoning and code-generation benchmarks: MATH500 [21], GSM8K [9], MBPP [3], and HumanEval [5]. These tasks require multi-step reasoning, making them sensitive to quality degradation under aggressive step compression. Metrics. For few-step decoding and full-decoding preservation, we report Accuracy. For dynamic decoding, we additionally report throughput and averaged tokens per decoding steps. For LLaDA coding tasks, we report Extraction Rate (ER), following the limited executable-solution extraction ability of the base model. Training Data and Implementation. For self-distillation methods, we collect teacher-generated trajectories from the corresponding training sets: MATH [13] for mathematical reasoning and 7 ``` ### Table 2: Few-step accuracy comparison on LLaDA. Following the same protocol as Table 1, we ```text Table 2: Few-step accuracy comparison on LLaDA. Following the same protocol as Table 1, we compare T3D with existing few-step decoding and self-distillation baselines under different TokPS settings. T3D achieves the best average accuracy at both TokPS = 4 and TokPS = 8. TokPS Method MATH500 GSM8K MBPP HumanEval AVG. Gains (%) Original Model 24.80 70.43 91.80 87.80 68.71 - ReDi 25.20 68.39 93.80 91.50 69.72 ↑ 1.48 4 dParallel 28.40 71.49 94.80 91.50 71.55 ↑ 4.13 CDLM 30.00 71.70 85.40 85.98 68.27 ↓ 0.64 T3D (Ours) 30.40 75.89 98.20 94.50 74.75 ↑ 8.79 Original Model 3.00 18.04 40.80 50.00 17.11 - ReDi 6.80 31.24 70.80 66.50 28.71 ↑ 67.80 8 dParallel 15.60 53.90 70.60 67.70 36.58 ↑ 113.76 CDLM 11.60 46.50 46.40 46.95 27.43 ↑ 60.29 T3D (Ours) 25.20 70.13 86.60 73.80 47.13 ↑ 175.47 PrimeIntellect [17] for code generation. Unless otherwise specified, trajectories are generated with static decoding and low-confidence remasking. During T3D training, the DDO reference model is periodically updated from the current student, and we mix random tokens into training inputs to improve robustness. All trainable methods are fine-tuned using full-parameter training on 8× NVIDIA A100-40GB GPUs. More implementation details, including trajectory construction, decoding settings, and training cost, are provided in Appendix C. 5.2 Improving Performance of Few-Step Decoding by Self-Distillation Settings. We evaluate few-step decoding un- Table 3: Preserving diffusion performance under der high Tokens Per Step (TokPS) settings, full decoding. We revert few-step distilled mod- where larger TokPS corresponds to more ag- els to full diffusion decoding using static decoding gressive parallel decoding. For SDAR, we (one token per step) without additional training. Re- evaluate two block sizes, 4 and 8, with TokPS sults are reported under block size 4 and 4 steps per = 2 and 4. For LLaDA, we set the maximum block, showing that T3D preserves diffusion perfor- generation length to 1024, use block size 32, mance. Bold numbers denote the best result among and evaluate TokPS = 4 and 8. These settings self-distillation methods. cover both moderate and highly compressed decoding regimes. Method MATH500 GSM8K MBPP HumanEval Results. Table 1 and Table 2 report few- SDAR-1.7B-Chat step accuracy across SDAR and LLaDA mod- Original Model 59.40 80.59 45.20 59.76 els. Overall, T3D is consistently among SFT 52.00 73.09 44.20 60.37 the strongest self-distillation methods and ReDi 47.00 73.77 27.60 31.10 achieves the best average performance in dParallel 0.40 0.23 34.60 43.29 most settings. The gains are especially pro- TD 49.80 72.40 35.20 32.93 T3D (Ours) 56.80 78.01 41.20 57.32 nounced under more aggressive decoding bud- gets, where competing methods often degrade SDAR-4B-Chat substantially. These results show that T3D bet- Original Model 68.00 89.84 58.60 71.95 ter preserves generation quality when the dif- SFT 60.20 86.05 50.20 69.51 fusion process is compressed to only a few ReDi 50.40 82.03 34.00 37.80 steps. dParallel 13.20 2.88 34.00 48.17 TD 57.40 82.11 37.60 43.90 5.3 Preserving Diffusion T3D (Ours) 70.00 89.31 54.20 73.78 Performance under Full Decoding Settings. In this experiment, we investigate whether few-step distillation leads to diffusion property forgetting, i.e., whether a model optimized for compressed decoding degrades when reverted to the original full diffusion process. To evaluate this, we take models distilled for few-step generation and directly restore them to full diffusion decoding using static decoding strategy, decoding one token per step without any additional training. Results. Table 3 reports the results. Across both SDAR-1.7B-Chat and SDAR-4B-Chat, our methods preserve strong performance under full decoding. In particular, T3D achieves performance nearly identical to the original pretrained model on all benchmarks, and in some cases slightly outperforms it. In contrast, prior baselines such as ReDi and dParallel exhibit substantial degradation. 8  9 7   +10.8% ACC ``` ### Table 5: Dynamic decoding results with block size 4, 4 steps per block, confidence threshold 0.9, and ```text Table 5: Dynamic decoding results with block size 4, 4 steps per block, confidence threshold 0.9, and temperature 0.1. We report throughput (TPS), per-sample latency (Latency), average decoding steps and sequence length (Avg Steps and Avg Length), and accuracy (Acc). Bold numbers indicate the best performance among baseline methods. All experiments are done using SDAR-4B-Chat model. Dataset Method TPS↑ Latency↓ Avg Steps↓ Avg Length Acc↑ Original 657.72 1.10 196.19 721.90 39.00 ReDi 715.71 1.04 198.24 757.05 27.00 MATH500 dParallel 692.08 0.95 170.22 653.98 45.80 FKL 693.85 0.97 177.99 678.55 44.00 T3D (Ours) 791.23 0.66 137.95 525.50 49.40 Original 580.60 0.43 71.12 249.52 61.56 ReDi 636.58 0.49 84.63 311.99 54.89 GSM8K dParallel 805.02 0.39 83.23 310.58 67.02 FKL 696.99 0.47 89.78 330.82 62.40 T3D (Ours) 843.05 0.37 83.03 312.48 72.40 Original 262.66 0.36 27.25 93.64 23.40 ReDi 298.83 0.21 17.11 62.57 10.00 MBPP dParallel 215.65 0.63 36.03 135.16 8.40 FKL 314.99 0.31 26.43 98.80 9.80 T3D (Ours) 313.18 0.19 16.94 61.62 23.60 Original 175.48 0.73 36.56 127.54 33.54 ReDi 163.77 0.47 21.23 76.75 10.00 HumanEval dParallel 130.34 0.48 17.41 62.19 23.78 FKL 216.39 0.29 17.15 62.10 23.17 T3D (Ours) 222.68 0.26 16.21 58.10 29.27 ``` ### Table 6: Pass@k results on MATH500 using SDAR-4B-Chat with block size 4 and TokPS 2. T3D ```text Table 6: Pass@k results on MATH500 using SDAR-4B-Chat with block size 4 and TokPS 2. T3D achieves larger gains as k increases, suggesting that it preserves output diversity and benefits from test-time scaling. Model pass@5 pass@10 pass@20 Teacher 81.9 85.6 88.2 Forward-KL 53.1 63.3 71.6 T3D (ours) 66.0 74.2 80.4 We provide two additional analyses to examine whether T3D suffers from diversity collapse. • Entropy and output diversity. As shown in Fig. 6 (b), T3D does not exhibit uniform diver- sity collapse relative to Forward-KL. Instead, it shows a stage-wise exploration–exploitation pattern: higher entropy at early decoding stages (mask ratio = 1.0), indicating broader exploration, and lower entropy at later stages, enabling sharper refinement. This behavior is desirable for reasoning, where the model should explore possible solution paths early and refine toward a coherent answer later. • Exploration behavior on reasoning tasks. As shown in Table 6, we compare pass@k on MATH500 for the teacher, Forward-KL baseline, and T3D. T3D outperforms Forward-KL at every k. Moreover, the gap between T3D and the teacher narrows as k increases, indicating that T3D preserves meaningful exploration ability rather than collapsing to a narrow set of outputs. Overall, these results suggest that T3D does not simply reduce diversity in an indiscriminate way. Instead, it preserves useful exploration for reasoning while promoting sharper refinement during decoding. This helps explain why T3D benefits from test-time scaling and consistently improves over Forward-KL under larger pass@k budgets. 17 ``` ### Table 7: Multi-seed results on MATH-500 and MBPP. We report accuracy over three seeds, together ```text Table 7: Multi-seed results on MATH-500 and MBPP. We report accuracy over three seeds, together with the mean, standard deviation, and sample variance. MATH-500 MBPP Method Seed 1 Seed 2 Seed 3 Mean ± Std Var. Seed 1 Seed 2 Seed 3 Mean ± Std Var. Original 14.80 14.20 12.60 13.87 ± 1.14 1.29 13.00 15.00 13.40 13.80 ± 1.06 1.12 ReDi 24.20 24.20 23.00 23.80 ± 0.69 0.48 6.40 6.80 6.60 6.60 ± 0.20 0.04 dParallel 37.80 38.80 36.20 37.60 ± 1.31 1.72 7.00 6.20 7.00 6.73 ± 0.46 0.21 T3D (ours) 46.20 46.00 46.60 46.27 ± 0.31 0.09 21.60 21.60 22.40 21.87 ± 0.46 0.21 ``` ### Table 8: Results on WinoGrande. T3D outperforms prior few-step decoding baselines, suggesting ```text Table 8: Results on WinoGrande. T3D outperforms prior few-step decoding baselines, suggesting that its benefits extend beyond structured math and coding benchmarks. Model Original dParallel ReDi T3D (ours) Accuracy 1.0 17.4 29.7 31.5 D.3 Experiments with Multiple Seeds To evaluate the stability of T3D, we repeat the main few-step experiments on MATH-500 and MBPP with three random seeds. As shown in Table 7, T3D consistently outperforms all baselines across both benchmarks. On MATH-500, T3D achieves an average accuracy of 46.27 with a standard deviation of only 0.31, indicating both strong performance and low variance across seeds. On MBPP, T3D obtains an average accuracy of 21.87, substantially outperforming the original model and prior few-step DLLM baselines. These results suggest that the gains of T3D are stable and not due to seed-specific variation. D.4 Generalization to Open-Ended Language Tasks Our main experiments evaluate T3D on math and coding benchmarks, which test structured reasoning and executable generation under aggressive few-step decoding. To further examine whether T3D remains effective beyond these structured tasks, we additionally evaluate it on WinoGrande, a broader NLP benchmark that requires commonsense language understanding. As shown in Table 8, T3D outperforms prior few-step decoding baselines, improving over both dParallel and ReDi. This suggests that the benefits of T3D are not restricted to math or coding tasks, but also extend to more open-ended language tasks. Overall, the WinoGrande results provide additional evidence that T3D is not only effective on structured reasoning and coding benchmarks, but can also improve few-step generation on broader open-ended language tasks. E Ablation Study In this section, we present ablation studies for our proposed T3D. In Appendix E.1, we analyze the effect of the regularization coefficient λ. In Appendix E.2, we examine how different components of our method contribute to preserving the full diffusion decoding behavior. Finally, in Appendix E.3, we present ablations under few-step generation settings to evaluate the contribution of each component to the overall performance of our method. E.1 The Effectiveness of λ in Training Objective We conduct an ablation study on the regularization weight λ in Eqn. 13. We run these experi- ments using the SDAR-4B-Chat model and evaluate it on MATH500 benchmark. Table 9 reports performance under different decoding configurations with varying Tokens Per Step (TokPS), block sizes, and decoding steps. Results. Overall, moderate regularization consistently yields the best or near-best performance across most settings. In particular, λ = 0.2 achieves the strongest results in the majority of configura- tions, especially under more aggressive few-step decoding regimes (e.g., higher TokPS). In contrast, 18 ``` ### Table 9: Ablation study on the effect of the regularization weight λ under different decoding ```text Table 9: Ablation study on the effect of the regularization weight λ under different decoding configurations. We report the model performance across varying Tokens Per Step (TokPS), block sizes, and decoding steps. All experiments are done using MATH500 dataset. TokPS Block Size Decoding Steps λ = 0.05 λ = 0.2 λ = 0.5 1 4 4 67.80 69.00 69.20 1 8 8 62.60 64.80 65.40 2 8 4 57.20 58.60 56.20 4 4 1 47.00 47.20 46.00 4 8 2 40.20 45.20 42.00 8 8 1 7.20 7.60 6.20 ``` ### Table 10: Ablation results under full-step diffusion decoding on MATH500. All variants are trained ```text Table 10: Ablation results under full-step diffusion decoding on MATH500. All variants are trained for few-step distillation and then evaluated by reverting to the original full-step diffusion process with block size 4 and decoding steps 4 per block. Method Objective / Variant Acc. Original – 68.00 SFT Supervised Fine-Tuning 60.20 ReDi Endpoint-Style Distillation 50.40 TD Ltraj 22.00 TD + Path Loss Ltraj + λLpath 58.00 DDO Ltraj-DDO 12.00 T3D (Ours) Ltraj-DDO + λLpath 69.00 a smaller regularization weight (λ = 0.05) is often insufficient to stabilize training, while overly strong regularization (λ = 0.5) can lead to degraded performance in several settings. Based on these observations, we fix λ = 0.2 for all experiments reported in the main results. E.2 Preserving Full-Step Diffusion Properties We first examine whether few-step distillation preserves the original full-step diffusion behavior. After training each variant under the few-step distillation setting, we revert the model to the original full-step diffusion decoding process without any additional fine-tuning. This evaluation tests whether the learned model still retains the fine-grained denoising capability of the pretrained diffusion model. Results. As shown in Table 10, directly applying few-step distillation can substantially degrade full-step diffusion behavior. Both TD and DDO alone perform poorly when the distilled model is reverted to the original full-step decoding process, indicating that optimizing only for compressed decoding may damage the model’s fine-grained denoising capability. Adding the path-consistency loss substantially improves preservation under full-step decoding. For TD, adding Lpath improves accuracy from 22.00 to 58.00, showing that path-level supervision helps retain intermediate denoising behavior. The full T3D objective achieves the best result, reaching 69.00 accuracy and slightly surpassing the original model. These results suggest that T3D improves few-step decoding while preserving the intrinsic diffusion behavior of the pretrained model. E.3 Ablation Study on Few-Step Generation We further study how each component affects few-step generation performance. We evaluate SDAR- 4B-Chat on MATH500 with block size 8 under two decoding budgets: 4 decoding steps per block and 2 decoding steps per block. The latter corresponds to a more aggressive few-step decoding regime. Results. As shown in Table 11, trajectory-level distillation is the key factor behind the improvement in few-step generation. TD improves over the original model under both decoding budgets, especially in the more aggressive setting with only 2 decoding steps per block, where accuracy increases from 19 ``` ### Table 11: Component-wise ablation on few-step generation using SDAR-4B-Chat on MATH500. We ```text Table 11: Component-wise ablation on few-step generation using SDAR-4B-Chat on MATH500. We evaluate block size 8 with two decoding budgets: 4 and 2 decoding steps per block. Higher accuracy is better. BS = 8 BS = 8 Method Objective / Variant DS = 4 DS = 2 Original – 49.60 16.80 SFT Supervised fine-tuning 54.44 40.20 ReDi Endpoint-style distillation 23.60 20.20 TD Ltraj 52.60 38.80 TD + Path Loss Ltraj + λLpath 49.40 37.20 DDO Ltraj-DDO 52.22 36.40 T3D (Ours) Full objective 60.60 45.00 16.80 to 38.80. This supports our main claim that matching teacher trajectories helps reduce the factorization error induced by large denoising jumps. DDO further improves few-step performance by replacing the forward-KL trajectory objective with a mode-seeking trajectory-matching objective. Under the aggressive setting with 2 decoding steps per block, the full T3D objective performs best, reaching 45.00 accuracy. This suggests that the benefit of each component becomes more apparent as the decoding budget becomes tighter. Overall, these results show that trajectory supervision provides the main gain, DDO improves the quality of trajectory matching, and path consistency further stabilizes generation under compressed decoding. 20 NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The main claims match the paper’s contributions, theoretical analysis, and empirical results, without going beyond the demonstrated scope. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of our method, particularly those inherent to self- distillation. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs ``` ## reference/dmax/Multi-Block Diffusion Language Models ### Table 1 Evaluation results across math and code benchmarks. SingleBD (Native) denotes the native single- ```text Table 1 Evaluation results across math and code benchmarks. SingleBD (Native) denotes the native single- block diffusion inference of each BD-LM; MultiBD (training-free) applies multi-block decoding without retraining; MBD-* denotes the corresponding MultiTF-post-trained MBD-LM. AUP (Accuracy Under Parallelism) combines accuracy and TPF, reported in the Average column as an aggregate across four benchmarks. MBD-LMs consistently improve TPF over SingleBD. In most settings, MultiTF recovers or improves the quality lost by training-free MultiBD, leading to a better accuracy–parallelism trade-off. GSM8K MATH500 MBPP+ HumanEval+ Average Model Acc ↑ TPF ↑ Acc ↑ TPF ↑ Acc ↑ TPF ↑ Acc ↑ TPF ↑ Acc ↑ TPF ↑ AUP ↑ LLaDA2-Mini-DMax (bufsz=2, blksz=32) SingleBD (Native) 91.89 5.70 76.80 6.13 72.22 6.14 77.44 7.44 79.59 6.35 459.54 MultiBD (training-free) 89.84 8.76 73.80 9.08 72.22 8.44 76.83 10.96 78.17 9.31 651.98 MBD-LLaDA2-Mini-DMax 91.74 8.95 75.00 9.31 70.11 8.34 77.44 10.78 78.57 9.34 661.28 LLaDA2-Mini (bufsz=2, blksz=32) SingleBD (Native) 91.89 2.27 74.20 2.83 75.66 3.25 78.05 5.53 79.95 3.47 247.41 MultiBD (training-free) 92.65 2.76 73.60 3.53 72.49 3.97 75.61 7.37 78.59 4.41 301.81 MBD-LLaDA2-Mini 91.96 5.55 79.20 6.02 72.49 5.35 80.49 7.85 81.03 6.19 449.18 SDAR-8B-Chat-b32 (bufsz=4, blksz=32) SingleBD (Native) 90.07 2.52 65.60 3.81 52.65 1.83 67.68 2.00 69.00 2.54 141.64 MultiBD (training-free) 89.01 2.78 60.60 5.06 52.12 1.97 65.85 2.24 66.89 3.01 156.35 MBD-SDAR-8B-Chat-b32 89.16 3.08 68.00 5.08 58.99 4.87 62.80 4.82 69.74 4.46 210.42 SDAR-8B-Chat-b4 (bufsz=4, blksz=4) SingleBD (Native) 91.05 1.33 72.80 1.46 64.80 1.13 73.70 1.07 75.59 1.25 85.46 MultiBD (training-free) 90.45 2.39 70.60 2.68 65.80 1.55 74.39 1.47 75.31 2.00 129.59 MBD-SDAR-8B-Chat-b4 91.81 2.28 72.40 2.52 64.29 2.62 72.56 2.24 75.27 2.42 148.65 (a) Training-free MultiBD transfers to additional model variants. (b) Ablation of MultiTF training components SingleBD (Native) denotes each model’s native single-block dif- averaged over HumanEval+ and GSM8K with fusion inference. LLaDA2-Mini-DMax. GSM8K MATH500 Average Configuration Acc ↑ TPF ↑ AUP ↑ Acc ↑ TPF ↑ Acc ↑ TPF ↑ Acc ↑ TPF ↑ AUP ↑ SingleBD (Native) 84.67 6.57 536.89 LLaDA2-Mini-CAP (bufsz=2, blksz=32) noise-group layouts construction SingleBD (Native) 91.74 3.08 77.80 3.71 84.77 3.40 247.30 + systematic layouts 83.22 9.71 774.03 MultiBD (training-free) 91.21 4.00 77.20 4.94 84.21 4.47 319.17 + random layouts 82.72 9.42 747.46 systematic + random layouts (ours) 84.59 9.87 805.34 LLaDA2.1-Mini (bufsz=2, blksz=32) SingleBD (Native) 93.03 4.12 81.40 4.87 87.22 4.50 390.64 block-level noise-scheduler MultiBD (training-free) 92.27 5.80 81.00 7.20 86.63 6.50 558.52 D2F-style monotonic scheduler 79.34 8.76 657.74 random scheduler 83.14 9.70 771.74 sorted-uniform scheduler 81.28 9.73 748.73 chain-uniform scheduler (ours) 84.59 9.87 805.34 ``` ### Table 2 Transfer and ablation results. (a) Training-free MultiBD transfers to additional model variants on math ```text Table 2 Transfer and ablation results. (a) Training-free MultiBD transfers to additional model variants on math benchmarks. (b) MultiTF component ablations averaged over HumanEval+ and GSM8K. All reported metrics are higher-is-better. naive dynamic MultiBD, the optimized MultiBD, block-state transitions, threshold rules, and prefix-cache analysis, are provided in Appendix C. The realized speedup is validated by the TPS results in Table 3. 4 Experiments 4.1 Experimental Setup Models and training. We evaluate MultiTF on representative BD-LMs from the LLaDA2.x (Bie et al., 2025, 2026) and SDAR (Cheng et al., 2025) families, including variants enhanced with DMax (Chen et al., 2026). For each base model, MultiTF post-training constructs multiple group-layouts per sample, including systematic shifted layouts and random layouts, to approximate the MultiBD running-set states described in Section 3.1. The resulting models are denoted as MBD-* models, e.g., MBD-LLaDA2-Mini and MBD-SDAR- 8B-Chat. We also evaluate training-free MultiBD, which directly applies MultiBD inference to the original BD-LMs without post-training. Benchmarks and metrics. We evaluate mathematical reasoning on GSM8K (Cobbe et al., 2021) and MATH500 (Hendrycks et al., 2021), and code generation on MBPP+ and HumanEval+ (Liu et al., 2023). We report Accuracy, Tokens Per Forward pass (TPF), and Accuracy Under Parallelism (AUP). Accuracy is exact match for math and pass@1 for code. TPF measures decoding parallelism, while AUP summarizes the accuracy–parallelism trade-off following d3LLM (Qian et al., 2026). Given a set of decoding configurations C, we sort them by TPF and compute AUP as the trapezoidal area under the accuracy–TPF curve: |C|−1 X Aci + Aci+1 (4.1)  AUP = Pci+1 − Pci , i=1 2 where Aci and Pci denote the accuracy and TPF of configuration ci , respectively. For multi-benchmark evaluation, we report the average AUP across benchmarks. Experimental details. Detailed training hyperparameters, inference hyperparameters, hardware settings, and training costs are provided in Appendix D. 4.2 Main Results We first evaluate whether MBD-LMs can improve decoding parallelism without sacrificing generation quality. The analysis focuses on four questions: (i) whether MultiTF-post-trained MBD-LMs improve the TPF– accuracy trade-off over native SingleBD; (ii) whether MultiTF is complementary to T2T-enhanced decoding methods such as DMax; (iii) whether train–inference alignment is necessary beyond training-free MultiBD; and (iv) whether the gains generalize across different BD-LM backbones. Baselines and configurations.Table 1 reports results across four benchmarks. For each base BD-LM, we compare three configurations: (1) SingleBD (Native), the model’s native single-block diffusion inference; (2) MultiBD (training-free), MultiBD inference applied without post-training; and (3) MBD-*, the corresponding MultiTF-post-trained model using MultiBD inference. Main analysis.MBD-LMs improve decoding parallelism while preserving generation quality. Compared with LLaDA2-Mini under SingleBD (Native), MBD-LLaDA2-Mini increases average TPF from 3.47 to 6.19 (+78.4%) and improves average accuracy from 79.95% to 81.03%. Notably, even without DMax, MBD- LLaDA2-Mini reaches a TPF comparable to LLaDA2-Mini-DMax under SingleBD (6.19 vs. 6.35), while achieving higher average accuracy (81.03% vs. 79.59%). This shows that MultiTF can turn a standard BD-LM into an MBD-LM with DMax-level decoding parallelism. Compatibility with T2T-enhanced decoding.MultiTF is complementary to DMax, a Token-to-Token (T2T) en- hanced acceleration method. When combined with DMax, MBD-LLaDA2-Mini-DMax further increases aver- age TPF from 6.35 to 9.34 (+47.1%) over LLaDA2-Mini-DMax under SingleBD, with only a 1.02 percentage- point average accuracy drop. This indicates that MBD-LMs can stack with existing T2T-enhanced recipes. The comparison between training-free MultiBD and MultiTF-post-trained Effect of train–inference alignment. MBD-LMs highlights the importance of train–inference alignment. Directly applying MultiBD already in- creases TPF, confirming that multi-block decoding relaxes the single-block bottleneck. However, it can de- grade accuracy because the original BD-LMs are not trained on practical MultiBD states. MultiTF reduces this mismatch: on LLaDA2-Mini, accuracy improves from 78.59% under training-free MultiBD to 81.03% af- ter MultiTF post-training, while average TPF further increases from 4.41 to 6.19. On LLaDA2-Mini-DMax, MultiTF improves average accuracy from 78.17% to 78.57% while preserving high TPF. Generalization across BD-LM backbones. MBD-LMs also generalize beyond the LLaDA2 family. On SDAR- 8B-Chat-b32, MBD-SDAR-8B-Chat-b32 increases average TPF from 2.54 to 4.46 (+75.6%) and improves average accuracy from 69.00% to 69.74%. With block size 4, MBD-SDAR-8B-Chat-b4 reaches the best average AUP among the three SDAR configurations. These results suggest that the MBD-LM formulation and MultiTF post-training are not tied to a specific BD-LM backbone. In addition, Table 2a shows that training-free MultiBD transfers to additional Transfer of training-free MultiBD. ``` ### Table 3 Throughput and single-step latency comparison. Results are measured for single-sample decoding on two H100 ```text Table 3 Throughput and single-step latency comparison. Results are measured for single-sample decoding on two H100 GPUs with tensor parallelism degree 2 (TP=2). Step latency denotes the average wall-clock latency of one decoding forward pass. TPF and TPS gains are computed relative to LLaDA2-Mini, while latency cost reports the relative increase in per-step latency. Forward-step statistics Realized throughput Model Avg. TPF ↑ TPF Gain ↑ Step Lat. (ms) ↓ Lat. Cost ↓ GSM8K TPS ↑ MATH500 TPS ↑ MBPP+ TPS ↑ HumanEval+ TPS ↑ Avg. TPS ↑ TPS Gain ↑ LLaDA2-Mini 3.47 – 7.07 1.00× 344.05 403.45 496.19 824.94 517.16 – MBD-LLaDA2-Mini 6.19 +78.39% 8.78 1.24× 687.87 707.89 646.73 941.18 745.92 +44.24% LLaDA2-Mini-DMax 6.35 +83.00% 9.02 1.28× 700.82 730.60 754.97 931.55 779.49 +50.73% MBD-LLaDA2-Mini-DMax 9.34 +169.16% 11.20 1.58× 834.52 851.07 896.65 1124.43 926.67 +79.19% where Πeff is the effective compute throughput, BHBM is the effective HBM bandwidth, Tcomm includes fixed-configuration tensor-parallel communication, and Tlaunch denotes launch and runtime overhead. This expression shows that the realized throughput depends on both the useful-token numerator and the roofline- limited per-step cost denominator. The attention arithmetic intensity further explains why MultiBD can still be efficient despite processing more tokens per step. Ignoring lower-order terms, the attention arithmetic intensity is approximately   dQs Ps Qs AIattn ≈ =Θ sdtype ρ(Qs )Ps d sdtype ρ(Qs ) when Ps ≫ Qs . Therefore, increasing Qs through a larger Block Buffer makes the extend-attention step more compute intensive. Prefix KV reads, weight reads, and kernel-launch overheads are amortized over more query tokens. However, the gain is useful only to the extent that these processed tokens lead to committed tokens, as captured by ηtok . The measurements in Table 3 match this analysis. For LLaDA2-Mini, MBD increases the average TPF from 3.47 to 6.19, a 1.78× improvement, while the step latency increases from 7.07 ms to 8.78 ms, a 1.24× cost increase. The expected throughput scaling is therefore approximately 1.78/1.24 = 1.44×, closely matching the measured Avg. TPS improvement from 517.16 to 745.92, i.e., 1.44×. Similarly, for LLaDA2-Mini-DMax, MBD increases the average TPF from 6.35 to 9.34, a 1.47× improvement, while the step latency increases from 9.02 ms to 11.20 ms, a 1.24× cost increase. This predicts a throughput scaling of 1.47/1.24 = 1.18×, which closely matches the measured Avg. TPS improvement from 779.49 to 926.67, i.e., 1.19×. Thus, the observed gap between TPF gain and TPS gain is primarily explained by the increased per-forward cost of processing the larger static Block Buffer. Overall, MultiBD improves wall-clock throughput by increasing the number of useful tokens committed per forward pass and by making each extend-attention step more compute intensive. At the same time, static- shape execution introduces extra processed tokens through resident blocks and dummy slots, reducing token efficiency relative to the ideal case. The final TPS gain is therefore determined by the balance among TPF improvement, token efficiency, and roofline-limited step latency. 5 Related Work 5.1 Diffusion Language Models Diffusion Language Models (DLMs) generate text through iterative denoising and enable parallel token refinement as an alternative to autoregressive generation. Representative models include LLaDA (Nie et al., 2025), Dream (Ye et al., 2025), and LLaDA2.x (Bie et al., 2025, 2026), which improve scaling, initialization, and editable refinement. However, fully bidirectional DLMs are difficult to serve efficiently because they do not naturally support KV caching or flexible-length generation. Block Diffusion Language Models (BD-LMs) (Arriola et al., 2025; Bie et al., 2025; Cheng et al., 2025) address these limitations by introducing block-causal generation. Their native Single-Block Diffusion (SingleBD) inference decodes one noisy block conditioned on a clean cached prefix, enabling KV caching and intra-block parallel decoding. Nevertheless, SingleBD still processes blocks sequentially, leaving inter-block parallelism underused. Our work studies Multi-Block Diffusion (MultiBD) as a broader inference regime for BD-LMs, where a bounded running-set of consecutive blocks can be refined concurrently. 5.2 Efficient DLM Inference and Training Efficient DLMs have been studied through distillation, scheduling, caching, and parallel decoding. D2F (Wang et al., 2025) introduces noisy-block visibility during training and demonstrates the potential of MultiBD-style pipelined decoding. DMax (Chen et al., 2026), d3LLM (Qian et al., 2026), LightningRL (Hu et al., 2026), and dParallel (Chen et al., 2025) improve the accuracy–parallelism trade-off through training objectives or decoding schedules. Fast-dLLM (Wu et al., 2025) and LoPA (Xu et al., 2025) accelerate inference through caching and lookahead parallelism. Our work is complementary to these efforts but focuses on a different level of parallelism. Instead of only increasing token-level parallelism or applying MultiBD as an inference-time heuristic, we treat MultiBD as a target inference regime for BD-LMs. We identify the bounded running-set structure and heterogeneous slot-wise noise patterns as key train–inference alignment factors, and propose MultiTF to post-train BD-LMs into MBD-LMs with inference-like multi-block states. We further provide Block Buffer inference support so that MultiBD preserves prefix-cache reuse and static-shape execution. 6 Conclusion We proposed Multi-Block Diffusion Language Models (MBD-LMs), a unified formulation of BD-LMs for reliable MultiBD inference. Starting from the sequential bottleneck of SingleBD, we showed that MultiBD ``` ### Table 5. A theoretical coverage view is provided in Appendix A. ```text Table 5. A theoretical coverage view is provided in Appendix A. B.2 Chain-uniform Noise-Scheduler For each noise-group Hm = (j1 , . . . , jnm ), MultiTF applies the chain-uniform noise-scheduler used in Algo- rithm 1. We first define an effective upper bound teff = thigh − ρ(thigh − tlow ), (B.3) where ρ is the noise-transition margin ratio, corresponding to noise_transition_margin_ratio in the im- plementation. This parameter is independent of the random noise-scheduler power-law bias γrand , which is used only for the random noise-scheduler ablation. For each group, a group-level floor ℓ is first sampled from the lower part of the noise range. Then each block samples its mask ratio from the interval between the current floor and the effective upper bound, and the sampled ratio becomes the floor for the next block: ℓ ∼ U (tlow , teff ), tji ∼ U(ℓ, teff ), ℓ ← tji , i = 1, . . . , nm . (B.4) This construction produces monotonic but randomized slot-wise mask ratios inside each noise-group. Com- pared with the fixed-step D2F schedule over a long noisy sequence, the resulting groups have larger and more variable block-level noise-ratio gaps, matching the heterogeneous active blocks observed during MultiBD inference. For each block with mask ratio tji , MultiTF replaces ⌊B · tji ⌋ randomly selected token positions in bji with [M]. For a layout λ, the resulting noisy sequence is denoted as xλt . B.3 Group-Aware Dual-Stream Mask Following the TF-style construction of Block Diffusion, MultiTF concatenates the noisy and clean sequences into the input sequence Xλ = [xλt ; x0 ]. (B.5) The attention mask has the block form   MGD MGOC Aλ = , (B.6) 0 MBC where MGD is the group-aware diagonal mask on the noisy part, MGOC is the group-aware offset-causal mask from noisy tokens to clean tokens, and MBC is the standard block-causal mask on the clean part. Let Nλ and C denote token positions in the noisy and clean parts, respectively. Let g(i) be the noise-group index of token i, β(i) be its block index, and α(i) be the first block index of the noise-group containing i. The three masks are defined as [MGD ]ij = 1 ⇐⇒ i, j ∈ Nλ , g(i) = g(j), β(j) ≤ β(i), (B.7) [MGOC ]ij = 1 ⇐⇒ i ∈ Nλ , j ∈ C, β(j) < α(i), (B.8) [MBC ]ij = 1 ⇐⇒ i, j ∈ C, β(j) ≤ β(i), (B.9) and all other entries are zero. Thus, noisy tokens can attend to same-noise-group noisy tokens from the same or preceding blocks, each noise-group can condition on clean prefix blocks before it, and clean tokens never attend to noisy tokens. This implements the visibility pattern required by Equation 3.1 without information leakage. B.4 MultiTF Objective and Model-specific Training Recipes MultiTF defines the training-state construction: the layout λ, the noisy sequence xλt , the clean sequence x0 , and the Group-Aware Dual-Stream Mask Aλ . Different base BD-LMs can reuse the same MultiTF input sequences while keeping their own model-specific training recipes. B.4.1 Default MultiTF CE Objective The default MultiTF objective is masked-token cross-entropy on masked positions in the noisy part of Xλ . Let Mλ = {i : xλt [i] = [M]} (B.10) denote the masked positions. The objective is   1 X i  LMultiTF (θ) = −Eλ,t,x0 log pθ x0 | Xλ , Aλ . (B.11) |Mλ | i∈Mλ This objective is used for BD-LMs whose original training recipe is standard masked-token CE. Algorithm 2 DMax OPUT Self-Denoising Branch Require: Model θ; input sequence Xλ = [xλ t ; x0 ]; noisy length N ; mask token id m. 1: Run a no-gradient forward pass on the noisy part: L ← θ(Xλ ):N . 2: Compute argmax predictions x̂ ← arg max L. 3: Replace masked positions in xλ t with x̂. 4: return the partially self-denoised input sequence. B.4.2 DMax-enhanced Models: OPUT Self-denoising ``` ### Table 4 Inference hyperparameters for all evaluated configurations. τadd controls when a future block is activated; ```text Table 4 Inference hyperparameters for all evaluated configurations. τadd controls when a future block is activated; τsemi controls semi-completion or fallback progress; τstable controls activation stability; τM2T and τT2T are confidence thresholds for mask-to-token filling and token-to-token revision. Configuration Task Buffer Block Max Len Max New Max NFE τadd τsemi τstable τM2T τT2T LLaDA2-Mini-DMax SingleBD (Native) Math 1 32 4096 4096 1024 — — — 0.50 — SingleBD (Native) Code 1 32 4096 4096 1024 — — — 0.65 — MultiBD (training-free) Math 2 32 4096 4096 1024 0.10 0.90 0.50 0.50 — MultiBD (training-free) Code 2 32 4096 4096 1024 0.90 0.90 0.50 0.65 — MBD-LLaDA2-Mini-DMax Math 2 32 4096 4096 1024 0.10 0.90 0.50 0.50 — MBD-LLaDA2-Mini-DMax Code 2 32 4096 4096 1024 0.90 0.90 0.50 0.65 — LLaDA2-Mini SingleBD (Native) Math 1 32 4096 4096 1024 — — — 0.95 — SingleBD (Native) Code 1 32 4096 4096 1024 — — — 0.95 — MultiBD (training-free) Math 2 32 4096 4096 1024 0.10 0.90 — 0.95 — MultiBD (training-free) Code 2 32 4096 4096 1024 0.90 0.90 — 0.95 — MBD-LLaDA2-Mini Math 2 32 4096 4096 1024 0.10 0.90 — 0.95 — MBD-LLaDA2-Mini Code 2 32 4096 4096 1024 0.90 0.90 — 0.95 — SDAR-8B-Chat-b32 SingleBD (Native) Math 1 32 4096 4096 1024 — — — 0.95 — SingleBD (Native) Code 1 32 4096 4096 1024 — — — 0.95 — MultiBD (training-free) Math 4 32 4096 4096 1024 0.10 0.90 — 0.95 — MultiBD (training-free) Code 4 32 4096 4096 1024 0.90 0.90 — 0.95 — MBD-SDAR-8B-Chat-b32 Math 4 32 4096 4096 1024 0.10 0.90 — 0.95 — MBD-SDAR-8B-Chat-b32 Code 4 32 4096 4096 1024 0.90 0.90 — 0.95 — SDAR-8B-Chat-b4 SingleBD (Native) Math 1 4 4096 4096 1024 — — — 0.95 — SingleBD (Native) Code 1 4 4096 4096 1024 — — — 0.95 — MultiBD (training-free) Math 4 4 4096 4096 1024 0.10 0.25 — 0.95 — MultiBD (training-free) Code 4 4 4096 4096 1024 0.75 0.75 — 0.95 — MBD-SDAR-8B-Chat-b4 Math 4 4 4096 4096 1024 0.10 0.25 — 0.95 — MBD-SDAR-8B-Chat-b4 Code 4 4 4096 4096 1024 0.75 0.75 — 0.95 — LLaDA2-Mini-CAP SingleBD (Native) Math 1 32 4096 4096 1024 — — — 0.95 — SingleBD (Native) Code 1 32 4096 4096 1024 — — — 0.95 — MultiBD (training-free) Math 2 32 4096 4096 1024 0.10 0.90 — 0.95 — MultiBD (training-free) Code 2 32 4096 4096 1024 0.90 0.90 — 0.95 — LLaDA2.1-Mini SingleBD (Native) Math 1 32 4096 4096 1024 — — — 0.70 0.50 SingleBD (Native) Code 1 32 4096 4096 1024 — — — 0.70 0.50 MultiBD (training-free) Math 2 32 4096 4096 1024 0.10 0.90 — 0.70 0.50 MultiBD (training-free) Code 2 32 4096 4096 1024 0.90 0.90 — 0.70 0.50 refinement, while future dummy slots remain invisible until activated. As shown in Figure 7(3), this separates cached prefix blocks from active Block Buffer slots, enabling prefix KV reuse while still refining multiple active blocks in parallel. D Experimental Details This appendix reports the inference and MultiTF post-training hyperparameters used in our experiments. “—” indicates that the corresponding hyperparameter is not applicable. SingleBD (Native) denotes the original single-block inference of each BD-LM; MultiBD (training-free) denotes MultiBD inference without post-training; MBD-* denotes the corresponding MultiTF-post-trained model. ``` ### Table 5 MultiTF post-training hyperparameters. tlow and thigh denote the mask-ratio range; ρ is the margin ratio ```text Table 5 MultiTF post-training hyperparameters. tlow and thigh denote the mask-ratio range; ρ is the margin ratio used to determine the effective upper bound teff ; Nrand is the number of random group-layouts per sample. The random-scheduler ablation uses a separate power-law bias γrand , which is independent of ρ and is not used in the chain-uniform scheduler. Target Model Task Objective Data Seq Len Block Max Group tlow thigh ρ Nrand Steps MBD-LLaDA2-Mini-DMax Math MultiTF + DMax OPUT 60k 2048 32 2 0.001 1.00 ρcfg 0 15000 MBD-LLaDA2-Mini-DMax Code MultiTF + DMax OPUT 60k 2048 32 2 0.001 1.00 ρcfg 2 4000 MBD-LLaDA2-Mini Math MultiTF CE 60k 2048 32 2 0.001 1.00 ρcfg 0 15000 MBD-LLaDA2-Mini Code MultiTF CE 60k 2048 32 2 0.001 1.00 ρcfg 0 6500 MBD-SDAR-8B-Chat-b32 Math MultiTF CE 20k 2048 32 4 0.001 1.00 ρcfg 3 3125 MBD-SDAR-8B-Chat-b32 Code MultiTF CE 10k 2048 32 4 0.001 1.00 ρcfg 3 1670 MBD-SDAR-8B-Chat-b4 Math MultiTF CE 20k 2048 4 4 0.001 1.00 ρcfg 2 1250 MBD-SDAR-8B-Chat-b4 Code MultiTF CE 10k 2048 4 4 0.001 1.00 ρcfg 2 200 ``` ## reference/dmax/Revise, Don’t Freeze- Sampler-Matched Training for Self-Correcting Masked Diffusion Language Models ### Table 1: Main results with full step scaling. Strategy denotes the inference rule: “std unmask” is monotonic ```text Table 1: Main results with full step scaling. Strategy denotes the inference rule: “std unmask” is monotonic low-confidence remasking with committed tokens frozen, while “D3IM” is clean-slate top-K remasking with token revision enabled. GSM8K: 1319 problems, 8-shot CoT. MATH-500: 500 problems, sympy-verified. HumanEval: 164 problems, pass@1. MBPP: 500 problems, pass@1 (lm-evaluation-harness 3-shot). SCOPE: LoRA r=64, 500 steps on FineWeb-Edu. Bold = best in column. GSM8K MATH-500 HumanEval MBPP Model Strategy 64 128 256 64 128 256 64 128 256 64 128 256 Original LLaDA std unmask 55.3 58.4 59.5 18.8 21.6 22.4 14.0 26.8 32.9 20.4 29.4 39.2 Original LLaDA D3IM 48.0 51.6 55.3 17.8 18.6 19.2 5.5 4.3 7.3 6.8 8.8 12.0 SCOPE std unmask 52.8 55.3 59.7 19.2 21.2 27.0 6.7 20.1 31.1 16.4 27.6 36.0 SCOPE D3IM 68.3 71.9 73.8 23.6 27.6 30.0 29.3 30.5 34.8 30.8 36.8 36.6 units (“hour”→“minute”), incorrect digits, or mis- ing that continued training on the standard MDM matched operators, and positions are typically re- objective alone shifts the model away from D3IM’s vised only once. decision boundary without improving its revision ability. 4.5 Ablations and Additional Analysis Design choice ablation. We report leave-one- out ablations from the full SCOPE recipe in Ta- D3IM channel ablation. Table 4 isolates ble 3. Each row removes one design choice while D3IM’s two correction channels at 64 steps on keeping the LoRA rank, data, training steps, and bounded subsets (GSM8K-200; MBPP-50). On D3IM inference fixed. The MDM-only row dis- the SCOPE checkpoint, both channels contribute: ables the entire self-conditioning branch and serves disabling either token-to-token revision or token- as a matched-budget continued-training baseline to-mask demotion reduces accuracy, and disabling rather than a strict leave-one-out variant. The both recovers standard unmasking performance. two largest contributors are focused loss (−10.8) On the Original checkpoint, the pattern reverses and confidence-based commit (−3.2): supervis- for token-to-token revision: disabling it raises ing all positions reduces the relative weight of GSM8K from 47.0% to 63.0%, indicating that with- the error-correction gradient, while random com- out SCOPE the model’s direct token overwrites are mit removes the model’s ability to lock in high- destructive. After SCOPE training, the same chan- confidence tokens. Notably, MDM-only continued nel becomes productive (full D3IM reaches 70.5%), training (46.2%) performs even worse than Origi- confirming that preservation bias, not sampler de- nal+D3IM without any training (48.0%), suggest- sign, is the bottleneck. ``` ### Table 2: D3IM correction examples on GSM8K. Left: Table 4: D3IM channel ablation at 64 steps on GSM8K- ```text Table 2: D3IM correction examples on GSM8K. Left: Table 4: D3IM channel ablation at 64 steps on GSM8K- Original model on problem #29 (answer = 23); 274 cor- 200 and MBPP-50. “No-t2t” disables token-to-token rections total, the same positions oscillate between con- overwrite; “no-t2m” disables token-to-mask demotion; tent and whitespace. Right: SCOPE model across sev- “masked-only” disables both. Bold = best per model. eral problems (all answered correctly); each correction targets a specific semantic error. Original SCOPE Sampler GSM MBPP GSM MBPP Original SCOPE std unmask 59.5 8.0 59.5 6.0 Content → whitespace oscil- Unit corrections: D3IM masked-only 59.5 8.0 59.0 6.0 lation: D3IM no-t2t 63.0 4.0 66.0 10.0 “the” → \n “hour” → “minute” D3IM no-t2m 39.5 0.0 64.0 4.0 “the” → \n “week” → “hours” D3IM (full) 47.0 2.0 70.5 10.0 “the” → \n “div” → “ounces” \n → “the” (reversal) Numerical recalculation: “the” → \n (re-erased) “1”→“2”, “5”→“0” Table 5: Inference-only revokable samplers vs. Number erasure: “=”→“*” (operator fix) SCOPE+D3IM at N =64 steps. SCOPE training “.” → \n Concept / entity fixes: strongly improves D3IM but does not transfer to “0” → \n “distance”→“time” ReMDM or Tolerator. “0” → \n “groups”→“boxes” HE GSM MATH ``` ### Table 3: Leave-one-out ablation of SCOPE design Sampler Orig SCOPE Orig SCOPE Orig SCOPE ```text Table 3: Leave-one-out ablation of SCOPE design Sampler Orig SCOPE Orig SCOPE Orig SCOPE choices. All variants use LLaDA-8B, LoRA r=64, 500 std unmask 14.0 6.7 55.3 52.8 18.8 19.2 steps, FineWeb-Edu, D3IM-64. Drop is relative to full D3IM 5.5 29.3 48.0 68.3 17.8 23.6 SCOPE. MDM-only is a matched-budget baseline, not ReMDM-conf 9.8 6.1 54.8 52.5 16.2 14.2 a strict leave-one-out row. ReMDM-cap 6.7 5.5 52.1 47.8 14.6 14.4 Tolerator 21.3 21.3 65.4 65.7 19.6 17.2 Variant GSM8K Drop Original + D3IM, no training 48.0 −20.3 MDM-only continued training 46.2 −22.1 SCOPE, but SCOPE+D3IM surpasses it (68.3%). Full SCOPE 68.3 – Additional ablations covering temperature (τ ) − Focused loss (all-position CE) 57.5 −10.8 and commit-rate schedule (ρ) are reported in Ap- − Confidence commit (random) 65.1 −3.2 pendix C. 5 Related Work Sampler-specificity. SCOPE training is matched to D3IM’s clean-slate top-K decision rule. Table 5 Training on self-generated states. Several re- applies three inference-only revokable samplers cent methods reduce the train-inference gap by ex- (ReMDM-conf, ReMDM-cap (Wang et al., 2025), posing diffusion language models to self-generated and Tolerator (Tian et al., 2025)) on both Original states. DMax (Chen et al., 2026) introduces On- and SCOPE-trained checkpoints. D3IM gains +6 Policy Uniform Training and pairs it with Soft to +24 pp from SCOPE training across the three Parallel Decoding in an interpolated embedding benchmarks, while ReMDM is consistently hurt space. ProSeCo (Schiff et al., 2026) trains a cor- (−1 to −4 pp) and Tolerator is essentially neutral. rector from the model’s own unmasking errors and This is expected: SCOPE specifically trains the adds corrective refinement steps during generation. model’s token → token revision ability by exposing LLaDA2.1 (Bie et al., 2026) jointly trains Mask- it to self-generated visible errors and teaching it to-Token and Token-to-Token objectives, enabling to directly replace wrong tokens with correct ones. threshold-based token editing at inference. These D3IM exploits this ability through its global top- methods share with SCOPE the idea of training on K re-ranking. ReMDM, however, only uses the self-generated predictions, but differ in interface mask → token pathway: it first demotes a token to and goal: DMax targets soft embedding-space de- [MASK] and then re-predicts, so the token-to-token coding, ProSeCo adds explicit correction passes, capability that SCOPE improves is never exercised. and LLaDA2.1 relies on thresholded token edit- Tolerator uses a verify-and-refine loop independent ing. SCOPE training instead aligns the confidence of the confidence ranking, so it is neither helped nor ranking with D3IM sampling’s clean-slate top-K hurt; it already reaches 65.4% on GSM8K without rule. Inference-time revokable decoding. A comple- the position needs a similar token (e.g. “hour” → mentary line of work makes committed tokens re- “minute”), while a demotion to [MASK] signals that visable at inference time without retraining the base the model lacks a reliable candidate and should re- model. ReMDM (Wang et al., 2025) derives a predict from updated context. Retaining both gives stochastic remasking process for absorbing-state D3IM more expressive revision than mask-only MDMs; WINO (Hong et al., 2026) uses draft-and- methods. verify decoding; Tolerator (Tian et al., 2025) inter- Exposure bias in bidirectional models. In bidi- leaves fill-up with refinement; CoRe (Zhai et al., rectional MDLMs, a wrong committed token is 2026) probes context sensitivity to detect brittle visible to all positions, so subsequent generation tokens; and T2M (Yao, 2026) resets suspect to- may condition on the error and propagate it. This kens to [MASK] rather than directly overwriting effect can be self-reinforcing: in autoregressive them. These methods typically decide which ex- models, a committed token’s confidence is fixed isting tokens to revisit using remasking schedules, once generated, since only later positions attend to local detection rules, or verification passes. D3IM it; in bidirectional models, the wrong token also at- differs by constructing each intermediate state di- tends to subsequently generated context, and as rectly from current predictions through a global surrounding tokens accommodate the error, the top-K ranking. Our channel ablation further shows model’s confidence in the wrong token may in- that the same revision channels are harmful with- crease rather than decrease. Revision offers a sec- out SCOPE but beneficial after sampler-matched ond option: overwrite the wrong token instead of training. building around it. Our answer-first stress test (Ap- Confidence, calibration, and token-quality sig- pendix Table 10) is consistent with this hypoth- nals. Many remasking methods depend on confi- esis: SCOPE+D3IM achieves 68.0% vs. 66.0% dence or quality estimates to decide which tokens reasoning-first, suggesting that revision mitigates should survive. PC-Sampler (Huang et al., 2025) error propagation even when answer tokens are calibrates position-dependent confidence biases, likely committed with sparse context. RCR (He et al., 2025) adjusts remasking thresholds, and learned-score methods such as RemeDi (Huang 7 Conclusion et al., 2026) and PRISM (Kim et al., 2025) intro- We showed that self-correcting inference in duce token-quality or confidence signals for refine- MDLMs requires aligning the sampler with the ment. These approaches are complementary to model’s confidence behavior. D3IM re-evaluates ``` ### Table 6: EOS/EOT policy ablation (SCOPE+D3IM, 64 dition on the full prompt context. Only response ```text Table 6: EOS/EOT policy ablation (SCOPE+D3IM, 64 dition on the full prompt context. Only response steps). Task accuracy on bounded subsets (GSM8K positions participate in the mask–commit–retrain 200, HumanEval 50, MBPP-smoke 100). First EOS pos. cycle. and Early EOS <128 are pooled over 350 generations; Trunc. = fraction with no EOS at all. Mask-rate clamping. Per-sample mask rates rb ∼ Beta(2, 2) are clamped to [0.1, 0.9] to avoid Policy GSM HE 1st EOS Early Trunc. pos. <128 trivially easy (nearly unmasked) or trivially hard (nearly fully masked) training examples. No suppress. 64.5 36.0 208.8 20.0 0.0 Conf. (chosen) 70.5 46.0 233.2 8.0 0.0 Per-sample top-K commit. Because each sam- Logit, non-final 65.0 14.0 254.0 0.3 22.9 Logit, all steps 64.0 14.0 256.0 0.0 100.0 ple b has a different mask count |Mb |, we compute Kb = max(1, ⌊ρ S· |Mb |⌋) independently, then take the union C = b Cb as the committed set. Final-step accept. At step t = T , K(T ) = L Standard-MDM mixing. With probability 1 − and the entire block is committed regardless of con- pself-cond , we skip the self-conditioning branch en- fidence (no positions remain for the next iteration). tirely and fall back to the standard MDM training This guarantees the algorithm terminates with a objective: mask random positions and compute fully-decoded sequence. the CE loss only on masked positions. This pre- B.2 EOS / EOT Policy Ablation serves the model’s ordinary mask-filling ability and prevents catastrophic forgetting of the base distri- D3IM lets all tokens compete for top-K at every bution. step, so EOS/EOT tokens can win early and cause premature termination. We compare four policies Temperature branching. When (Table 6): (1) No suppression: EOS competes τ > 0, self-predictions are drawn from normally; 20% of generations place EOS before Categorical(softmax(z/τ )); when τ = 0, position 128. (2) Confidence suppression (cho- we fall back to arg max. Confidence ci is always sen): the model may still predict EOS, but during computed from the τ =1 logits regardless of the non-final steps its confidence is set to −∞ so it sampling temperature, so that the commit ranking cannot survive top-K ranking. EOS only appears reflects calibrated probabilities. at the final step when all positions are accepted. Why three design choices in one recipe. Ta- (3) Logit suppression, non-final: EOS logits are ble 3 ablates the three departures from a vanilla zeroed before softmax at non-final steps, prevent- self-conditioning recipe: (i) predicting from a ing EOS from being predicted at all. This is more masked context rather than a clean one (so the self- aggressive: early EOS nearly vanishes, but 22.9% prediction sees the same masked context the infer- of generations never produce EOS (truncated), and ence loop would), (ii) committing the top-ρ fraction HumanEval drops to 14.0%. (4) Logit suppres- by confidence rather than replacing all masked posi- sion, all steps: EOS logits are zeroed at every step tions, and (iii) focusing the CE loss on C∪M rather including the final one. All generations are trun- than supervising all positions. Beta(2,2) mask-rate cated. Confidence suppression achieves the best sampling and temperature sampling are sub-design accuracy (GSM8K 70.5, HumanEval 46.0) by de- choices: Beta(2,2) keeps the mask rate near 0.5 laying EOS without removing it from the model’s (avoiding trivially easy or trivially hard samples), output distribution. and τ =1.5 pushes the empirical error rate from ∼5% (argmax) to ∼20–45%, better matching the B.3 SCOPE Training: Implementation Details error rate D3IM encounters at inference (Appendix Algorithm 2 gives the simplified one-step proce- Table 7). dure. The full implementation adds the follow- ing details for batched training with prompts. Hy- Monitoring metrics. During training we track perparameter defaults are ρ = 0.3, τ = 1.5, two split losses on the committed set C: Lwrong pself-cond = 0.5. over the subset Cw = {i ∈ C : x̂i ̸= xi } mea- sures error correction, and Lright over Cr = {i ∈ Prompt-mask handling. Prompt positions P = C : x̂i = xi } measures preservation of correct {i : i < ℓp } are never masked, so the self- commits. A healthy run shows Lwrong decreas- prediction and the training forward pass both con- ing (model learns to correct) while Lright stays low ``` ### Table 7: Temperature ablation for SCOPE’s self- Table 8: Extended sampler comparison including ```text Table 7: Temperature ablation for SCOPE’s self- Table 8: Extended sampler comparison including prediction step (GSM8K, 64 steps, LoRA r=64). inference-only revokable baselines, at 64 steps on Higher sampling temperature improves standard un- GSM8K-200 and MBPP-50. Bold = best per model masking and modestly improves D3IM. column. D3IM channel ablation rows are repeated from ``` ### Table 4 for reference. † WINO without the second verifi- ```text Table 4 for reference. † WINO without the second verifi- τ std unmask D3IM D3IM − std cation pass (draft-only). 0 (argmax) 42.2 66.0 +23.8 1.0 41.7 66.8 +25.1 Original SCOPE Sampler 1.5 (paper) 52.8 68.3 +15.5 GSM MBPP GSM MBPP 2.0 55.2 68.7 +13.5 std unmask 59.5 8.0 59.5 6.0 D3IM masked-only 59.5 8.0 59.0 6.0 D3IM no-t2t 63.0 4.0 66.0 10.0 (preservation is easy because the gold token was D3IM no-t2m 39.5 0.0 64.0 4.0 committed). On 50% mask rate, training begins D3IM (full) 47.0 2.0 70.5 10.0 with Lwrong ≈ 6.3 and Lright ≈ 0.8; both decrease ReMDM-conf 59.0 4.0 60.0 6.0 over 500 steps to Lwrong ≈ 1.5 and Lright ≈ 0.05. ReMDM-cap 54.0 4.0 47.5 2.0 WINO† 57.0 8.0 57.0 8.0 Tolerator 67.0 8.0 70.0 4.0 C Additional Ablations C.1 Temperature C.4 Why Not Standard Calibration? Sweeping τ ∈ {0, 1, 1.5, 2} on the full GSM8K set (Table 7) reveals two effects: higher τ lifts Post-hoc methods such as temperature scaling or std-unmasking accuracy by 13 points (42.2% → Platt scaling rescale logits on a fixed validation dis- 55.2%), but D3IM attenuates this effect (only tribution, but D3IM fails under a shifted input distri- +2.7), because D3IM’s clean-slate rule can re- bution containing the model’s own wrong commits. resolve mistakes from poor training. We use Label smoothing similarly changes the target dis- τ =1.5; τ =2.0 is marginally stronger. tribution, not the self-generated context. SCOPE keeps the ground-truth target and changes the train- C.2 Commit-Rate Schedule ing input, directly exposing the model to the con- We tested replacing the fixed commit rate ρ=0.3 texts D3IM ranks at inference. with three scheduled variants: uniform sampling A natural concern is that SCOPE simulates only ρ ∼ U (0.1, 0.8), a curriculum from 0.1 to 0.8, one D3IM step during training, yet inference runs and a narrow perturbation ρ ∼ U (0.2, 0.4). None T =64 steps whose context evolves iteratively. Our consistently improves over the fixed rate across all trajectory analysis (§4.4) suggests that single-step benchmarks: each schedule trades a small gain on training is sufficient in this setting: the correction one task for a regression on another. We therefore pattern persists across the full 64-step trajectory. keep ρ=0.3 as the default. One plausible explanation is that the relevant rank- ing behavior is local—deciding whether a visible C.3 D3IM Channel Ablation: Extended token fits its surrounding context—and generalizes Analysis across progressive stages of D3IM inference. ``` ### Table 4 in the main text reports D3IM channel ```text Table 4 in the main text reports D3IM channel ablations on the core five sampler variants. Ta- C.5 Comparison with XDLM Training ble 8 extends this comparison with inference-only XDLM (Liu et al., 2026b) introduces a stationary revokable baselines evaluated on the same GSM8K- mixed-noise corruption process that unifies masked 200 and MBPP-50 subsets at 64 steps. The addi- and uniform-noise diffusion language modeling. tional rows confirm that SCOPE training specifi- Its native sampler performs confidence-gated local cally improves D3IM: WINO (draft-only, without token replacements, providing an explicit clean- its second verification pass) is unchanged (57.0% token replacement path during inference. Unlike → 57.0%), ReMDM-conf gains only 1 point, and D3IM’s global clean-slate survival rule, XDLM’s ReMDM-cap is hurt. Tolerator reaches 70.0% on decoder replaces a token only when the model’s SCOPE, nearly matching full D3IM (70.5%), but new prediction exceeds the current token’s confi- relies on a more complex two-phase inference loop dence, without demoting all positions to [MASK]. and does not benefit from the channel decomposi- To test whether XDLM’s training can substi- tion that makes D3IM’s gains interpretable. tute for SCOPE, we pair the publicly available ``` ### Table 9: XDLM vs. SCOPE on GSM8K (full 1319, 64 stantially reducing sensitivity to output order— ```text Table 9: XDLM vs. SCOPE on GSM8K (full 1319, 64 stantially reducing sensitivity to output order— steps). consistent with the exposure bias hypothesis in §6. Model Sampler GSM8K D Additional Related Work: Remasking XDLM low-confidence 50.8 Dynamics XDLM D3IM 60.3 SCOPE (r64) D3IM 68.3 Corrective DLM (Zhang et al., 2025) finds that confidence signals lack the resolution for precise error localization; Lost in Diffusion (Guo and Tan, LLaDA-XDLM checkpoint with both its native 2026) documents failure modes including prema- low-confidence sampler and D3IM (Table 9). ture termination. STaRR (Sun et al., 2026) ana- Switching XDLM from its native sampler to lyzes spatial-temporal confidence dynamics and D3IM improves GSM8K by +9.5 points (50.8% → designs adaptive remasking thresholds; “Early De- 60.3%), indicating partial compatibility: D3IM’s cisions Matter” (Kim et al., 2026) uncovers a prox- global remasking benefits even a model not trained imity bias where initial unmasking positions shape for it. However, XDLM+D3IM (60.3%) remains the trajectory; DiffuCoder (Gong et al., 2026) an- below SCOPE+D3IM (68.3%). This supports our alyzes token generation order in code tasks. I- claim that D3IM benefits most from training that DLM (Yu et al., 2026) identifies an introspective matches its hard top-K survival dynamics, rather consistency gap—DLMs often disagree with their than from a generic mixed-noise objective. own generations—and proposes modified training and decoding to close it, a framing related to our C.6 Exposure Bias Stress Test calibration perspective. Soft-Masked DLM (Her- As discussed in §6, a wrong committed token sche et al., 2026) addresses information loss at in bidirectional models is visible to all positions the representation level by blending mask and to- and may propagate rather than get corrected. We ken embeddings. These works analyze confidence stress-test this with an answer-first prompt format patterns, denoising order, and failure modes, but where the numerical answer precedes the chain-of- none examine the revision process itself: which thought derivation. Because D3IM commits tokens tokens get semantically corrected vs. superficially by confidence rather than position, the answer is reshuffled, and how revision frequency relates to not guaranteed to appear first, but placing it be- calibration quality. Our trajectory analysis (§4.4) fore the reasoning increases the chance that answer fills this gap. tokens are committed while surrounding context is still sparse. We compare against the standard reasoning-first (CoT-first) format on 200 GSM8K examples at 64 steps. ``` ### Table 10: Exposure bias stress test on GSM8K-200 (64 ```text Table 10: Exposure bias stress test on GSM8K-200 (64 steps). CoT = reasoning-first, Ans = answer committed before reasoning. Model Sampler CoT-first Ans-first Original std unmask 55.5 56.0 Original D3IM 46.0 55.0 SCOPE (r64) std unmask 53.5 54.0 SCOPE (r64) D3IM 66.0 68.0 SCOPE+D3IM is the strongest configuration un- der both orderings, with only a 2-point gap be- tween formats. In contrast, Original+D3IM shows a 9-point drop on CoT-first (46.0%) compared to answer-first (55.0%), suggesting that without SCOPE wrong early commits propagate through surrounding context. With SCOPE, D3IM can re- vise committed tokens as context develops, sub- ``` ## reference/dmax/dMoE- dLLMs with Learnable Block Experts ### Table 1: Results of the coefficient of variation (CV) ```text Table 1: Results of the coefficient of variation (CV) centration varies substantially across denois- on various benchmarks. ing steps and blocks. To examine this phe- nomenon in more detail, we further analyze Metric GSM8K MATH500 ARC-C MMLU the distribution of the unique expert count for Mean 70.01 71.09 62.85 66.57 the generated block at Layer 10 during infer- Std 11.96 11.34 10.01 11.85 ence, where Layer 10 corresponds to the mid- CV 17.09% 15.95% 15.93% 17.80% dle layer of LLaDA2.0-mini, which contains 19 layers in total. As illustrated in Figure 3 (b), the unique expert count changes noticeably across different denoising steps, suggesting that the extent to which tokens are routed to a concentrated subset of experts is highly step-dependent. To further quantify this variability, we adopt the coefficient of variation (CV), defined as σ CV = , µ where σ and µ denote the standard deviation and mean of the unique expert count, respectively. As shown in Table 1, the unique expert count exhibits consistently large variation across various benchmarks, suggesting that this phenomenon is not tied to a specific dataset or task. These results provide further empirical evidence that expert concentration varies substantially across denoising steps and blocks. 5 4.2 dMoE To achieve our goal of reducing the number Algorithm 1: Block-Level Expert Routing of block-level unique experts, a natural idea Input: Token-level expert scores {si }i∈B is to constrain the expert pool during expert Output: Block-level expert scores Sblock , coreset C, selection. Following a coarse-to-fine strategy, routed experts {Ri }i∈B we first select a coreset from the full expert Initialize Sblock ← 0 ∈ R|E| ; pool, and then perform token-level expert se- foreach i ∈ B do lection within this coreset (Figure 4). Specif- ŝi ← TopKMask(si , k); ically, given the tokens within a block, we Sblock ← Sblock + ŝi ; first perform token-level routing to compute S̃block ← Normalize(Sblock ); token-level expert scores:  C ← TopP(S̃block , p); si = Router(ti ) = sE1 , sE2 , . . . , sE|E| , foreach i ∈ B do where ti denotes the representation of the i- sCi ← MaskOutside(si , C); th token in the block, and sEi is the routing Ri ← TopK(sCi , k); score of expert Ei for token i. Inspired by return Sblock , C, {Ri }i∈B ; observation A, which suggests that token-level expert scores are positively correlated with expert importance, we then directly aggregate the token- level expert scores to obtain block-level expert scores: Sblock = ⊕i∈B si , where B denotes the set of tokens in the current block, and ⊕ represents the aggregation operator. After obtaining the block-level expert scores, we further determine the coreset based on their normalized values. Inspired by observation B, which suggests that the degree of expert concentration varies substantially across denoising steps and blocks, we do not enforce a fixed coreset size. Instead, we first normalize the block-level expert scores and then apply a top-p criterion to select the coreset: C = Top-P({S̃block }E e=1 , p), where C denotes the selected coreset and p is the cumulative probability threshold. This design naturally yields a smaller coreset when the expert score distribution is concentrated, and a larger one when the distribution is more dispersed, thereby adapting to the routing characteristics of different blocks. As such, it is better aligned with the original inference behavior. Finally, the original token-level routing is performed within the selected coreset (see more details in Algorithm 1). During the training process, we adopt a self-distillation paradigm [67] and follow the above routing procedure in the forward pass. During the inference process, we use the same forward process to maintain alignment between training and inference. 5 Experiments 5.1 Experimental Setups Training Data. All training data are constructed via self-distillation [67]. We first collect prompts from several public datasets, including the GSM8K training set [35], PRM12K [34], a subset of Numina-Math [68], and a subset of OpenThoughts [69]. We then use LLaDA2.0-mini (i.e., our base model) to generate corresponding responses as supervision targets. During generation, we follow the official settings: the confidence threshold is set to 0.95, the block size to 32, and the maximum output length to 2048 tokens. Samples that fail to terminate within this length limit are removed. Importantly, the entire supervision signal is derived from the model’s own generations, without relying on any externally curated high-quality responses. Overall, we obtain approximately 700K training samples. Training Details. We use LLaDA2.0-mini as the base model, which is a state-of-the-art open-source ``` ### Table 2: Results on various benchmarks with LLaDA2.0-Mini. For MMLU, we use the math part for ```text Table 2: Results on various benchmarks with LLaDA2.0-Mini. For MMLU, we use the math part for evaluation. Since the official DES code is not publicly available, we implement DES for comparison. Expert count denotes the average number of unique activated experts per layer. ∗ indicates that the corresponding threshold in DES is adjusted to achieve a higher compression ratio. MATH500 GSM8K ARC-C MMLU Method Expert Count Accuracy Expert Count Accuracy Expert Count Accuracy Expert Count Accuracy LLaDA2.0-Mini 70.0 72.0% 71.5 90.8% 65.9 84.4% 70.6 88.6% Baselines Top-4 43.5 52.6% 44.3 72.4% 42.2 74.0% 43.8 61.5% DES-S 41.0 70.8% 41.9 90.7% 39.9 83.3% 41.4 87.1% DES-V 36.9 71.8% 37.0 90.5% 36.8 82.3% 37.0 85.6% DES-S∗ 12.9 53.8% 13.2 74.0% 13.0 62.9% 13.0 70.7% DES-V∗ 13.0 53.0% 12.9 73.5% 13.0 49.1% 13.0 72.6% dMoE ours (ptrain = 0.6) 16.3 71.2% 16.6 88.6% 16.9 87.1% 16.5 87.4% ours (ptrain = 0.5) 13.7 69.8% 13.8 87.1% 13.9 84.7% 13.6 86.7% × 1.61 - 76.71% - 76.78% - 76.63% - 74.36% × 1.58 × 1.36 × 2.04 (a) Memory comparison (b) Latency speedup Figure 5: (a) We report the average memory footprint of uniquely activated MoE parameters across all layers for both the baselines and our dMoE. (b) We present the average end-to-end latency and MoE latency for the original model and our dMoE. per token from 8 in the official setting to 4; (3) DES-S, where we implement DES-Seq [33], a sequence-level routing strategy that enables more adaptive expert allocation; and (4) DES-V, where we implement DES-Vote [33], which reduces the number of unique experts within each block by consolidating router-derived preferences across tokens. Additionally, for both DES-S and DES-V, the degree of expert compression can be controlled through hyperparameters. Inference Details. We follow the official inference settings to evaluate the LLaDA2.0-mini [11]. For the baselines, we report the results from our implementation, with the latter following the configurations specified in the original paper. We apply the block diffusion setting with confidence- based parallel decoding, consistent with the official implementation. We follow the official settings for both our dMoE and all the baselines: set block size to 32, set max generation length to 2048, set confidence threshold to 0.95, and enable early stopping. Evaluation Details. We conduct comprehensive experiments on multiple benchmarks spanning a broad spectrum of reasoning tasks, including mathematical reasoning, scientific reasoning, and general high-level reasoning. Specifically, we evaluate on MATH500 [34], GSM8K [35], ARC-C [36], and MMLU [37]. In addition, we follow the simple-eval framework4 for zero-shot evaluation and prompt the model to generate its reasoning trajectories step by step. 5.2 Main Results dMoE Achieves Aggressive Expert Compression While Preserving Performance. We report the main results in Table 2. Compared with the original model, our dMoE achieves substantial expert 4 https://github.com/openai/simple-evals 7 ``` ### Table 3: Ablation on the cumulative probability threshold p at training and test stages. We evaluate ```text Table 3: Ablation on the cumulative probability threshold p at training and test stages. We evaluate cumulative probability thresholds of 0.4, 0.5, 0.6, 0.7, and 0.8 while keeping other settings fixed. MATH500 GSM8K ARC-C MMLU Method Expert Count Accuracy Expert Count Accuracy Expert Count Accuracy Expert Count Accuracy LLaDA2.0-Mini 70.0 72.0% 71.5 90.8% 65.9 84.4% 70.6 88.6% train dMoE (p = 0.6) ours (ptest = 0.4) 13.7 67.0% 13.9 86.0% 13.1 83.7% 13.5 84.1% ours (ptest = 0.5) 14.0 68.8% 14.2 87.5% 13.8 85.5% 13.7 87.0% ours (ptest = 0.6) 16.3 71.2% 16.6 88.6% 16.9 87.1% 16.5 87.4% ours (ptest = 0.7) 19.8 73.0% 20.1 90.2% 20.0 88.1% 20.2 87.4% ours (ptest = 0.8) 27.0 73.2% 28.3 90.0% 28.0 89.2% 27.6 88.9% dMoE (ptrain = 0.5) test ours (p = 0.4) 13.0 67.2% 13.5 85.9% 12.9 83.8% 13.1 84.1% ours (ptest = 0.5) 13.7 69.8% 13.8 87.1% 13.9 84.7% 13.6 86.7% ours (ptest = 0.6) 16.4 70.0% 16.5 88.3% 16.1 87.0% 16.5 87.0% ours (ptest = 0.7) 19.7 73.2% 20.2 90.2% 20.1 88.4% 19.9 87.4% ours (ptest = 0.8) 27.1 72.8% 28.1 90.4% 28.0 89.0% 27.5 88.1% compression while preserving performance almost intact. Specifically, the average performance across four benchmarks decreases only slightly from 83.95% to 83.575%, retaining 99.55% of the original performance, while reducing the average unique expert count from 69.5 to 16.575, corresponding to a 76.15% reduction in the average unique expert count. Compared with the baselines, our dMoE further achieves a 55.11% to 59.62% greater reduction in unique expert count at a comparable performance level. dMoE Significantly Reduces Memory Usage. As shown in Figure 5 (a), we compare the memory footprint of the original model, the baselines, and our dMoE. Specifically, we report the average memory footprint of uniquely activated MoE parameters across all layers. The results show that our dMoE achieves the lowest memory usage, yielding a 76.64% to 79.84% reduction compared with the original model and a 59.18% to 67.64% reduction compared with the baselines. dMoE Brings End-to-End Speedup. We further report the end-to-end latency (including MoE latency), together with the corresponding speedup, in Figure 5 (b) to evaluate the practical efficiency of our method. Across four benchmarks, our method achieves 1.14× to 1.66× end-to-end latency speedup, with the gain mainly coming from the reduction in MoE latency. These results demonstrate the effectiveness of our method in delivering real inference acceleration. dMoE Performs Better at Extreme Expert Compression. We further adjust the thresholds of DES-S and DES-V to achieve compression ratios comparable to those of our dMoE (DES-S∗ and DES-V∗ in Table 2). We can observe that under similar compression levels, our dMoE remains nearly lossless, retaining 99.11% of the original performance with p = 0.6 and 97.50% with p = 0.5, whereas DES-S and DES-V suffer from substantial performance degradation. Notably, our dMoE can compress the number of unique experts to near the practical limit, i.e., the number of experts required per token in the model, which is 8 in LLaDA2.0-mini, while still maintaining strong performance. 5.3 Diagnostic Results We provide comprehensive ablation studies on the cumulative probability threshold p and block size. Furthermore, we show that our dMoE achieves a superior performance-efficiency trade-off compared with the baselines. Ablation on the Cumulative Probability Threshold p. We first conduct an ablation study on the cumulative probability threshold p. Specifically, we train two models with ptrain = 0.6 and ptrain = 0.5, respectively, and evaluate each tuned model under ptest ∈ {0.4, 0.5, 0.6, 0.7, 0.8}. As shown in Table 3, when evaluated with the same ptest , the two models trained with different ptrain achieve similar performance, while still maintaining very low unique expert counts. Moreover, as ptest increases, the unique expert count gradually rises, accompanied by improved performance. Overall, these results demonstrate not only the robustness of our method but also its tunability, allowing the number of activated experts to be flexibly adjusted to different hardware characteristics and application requirements. 8 ``` ### Table 4: Ablation on the block size. The evaluated model is trained with a block size of 32 with ```text Table 4: Ablation on the block size. The evaluated model is trained with a block size of 32 with ptrain = 0.6. We set ptest = 0.6 here. We evaluate block sizes of 8, 16, 24, and 32 while keeping all other hyperparameters fixed. MATH500 GSM8K ARC-C MMLU Method Expert Count Accuracy Expert Count Accuracy Expert Count Accuracy Expert Count Accuracy Block size = 32 LLaDA2.0-Mini 70.0 72.0% 71.5 90.8% 65.9 84.4% 70.6 88.6% ours 16.3 71.2% 16.6 88.6% 16.9 87.1% 16.5 87.4% Block size = 24 LLaDA2.0-Mini 60.7 73.6% 62.2 91.0% 57.8 82.3% 61.6 86.9% ours 16.2 68.6% 17.4 87.4% 16.1 87.9% 15.8 85.3% Block size = 16 LLaDA2.0-Mini 48.6 70.2% 49.8 90.5% 47.0 86.9% 49.6 88.5% ours 14.8 70.0% 16.7 89.9% 15.1 88.1% 14.9 87.4% Block size = 8 LLaDA2.0-Mini 31.5 71.0% 32.2 90.1% 31.1 87.7% 32.3 87.4% ours 14.1 70.2% 16.1 88.1% 14.2 87.2% 14.1 86.7% dMoE (ours) DES-V (a) GSM8K (b) MATH500 (c) ARC-C Figure 6: Comparison of the performance-efficiency trade-off between our method and baselines. We report the results of GSM8K (a), MATH500 (b), and ARC-C (c). Ablation on the Block Size. We further conduct an ablation study on the block size, with the results summarized in Table 4. Specifically, we evaluate the model under block sizes of 8, 16, 24, and 32. The results show that our dMoE consistently compresses the unique expert count while maintaining strong performance across different block sizes. Specifically, under block sizes of 32, 24, 16, and 8, our dMoE reduces the average unique expert count by 76.15%, 72.97%, 68.46%, and 53.97%, while retaining 99.55%, 98.62%, 99.79%, and 98.81% of the original performance, respectively. The consistent efficiency improvement without a performance drop across different block sizes further demonstrates the effectiveness of our method. dMoE Reaches Superior Performance-Efficiency Trade-off. We present the performance– efficiency trade-off between our dMoE and the baselines. The results of our dMoE are taken from Table 3. For DES, we compare against its stronger variant, DES-V, and obtain its trade-off points by varying the threshold. As shown in Figure 6, our dMoE achieves better performance with fewer unique experts, leading to a superior performance–efficiency trade-off. 6 Conclusion In this work, we introduce dMoE, a novel strategy for block-level expert routing in MoE dLLMs. dMoE enables aggressive expert compression while incurring almost no performance degradation. Comprehensive evaluations across diverse benchmarks demonstrate the effectiveness of our method. More broadly, this work lays a foundation for block-level routing in MoE dLLMs and opens up a promising direction for improving their efficiency. 9 References [1] Qiuhua Yi, Xiangfan Chen, Chenwei Zhang, Zehai Zhou, Linan Zhu, and Xiangjie Kong. Diffusion models in text generation: a survey. PeerJ Computer Science, 2024. [2] Lingzhe Zhang, Liancheng Fang, Chiming Duan, Minghua He, Leyi Pan, Pei Xiao, Shiyu Huang, Yunpeng Zhai, Xuming Hu, Philip S Yu, et al. A survey on parallel text generation: From parallel decoding to diffusion language models. arXiv preprint arXiv:2508.08712, 2025. [3] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025. [4] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. [5] Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding. arXiv preprint arXiv:2505.16990, 2025. [6] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [8] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [9] Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193, 2025. [10] Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, et al. Mercury: Ultra-fast language models based on ``` ## reference/efficient-dlm/Efficient-DLM- From Autoregressive to Diffusion Language Models, and Beyond in Speed ### Table 1 | Comparing different dLM training schemes on Qwen2.5 1.5B. Row (a) shows the accuracy of the ```text Table 1 | Comparing different dLM training schemes on Qwen2.5 1.5B. Row (a) shows the accuracy of the original Qwen2.5 1.5B. Row (b) presents the training scheme of Dream [4]. Row (g) shows the identified best scheme with block-wise attention, clean context, and no token shift. Row Attn Clean Token KV Human Human MBPP Minerva MBPP GSM8K Avg ID Pattern Context Shift Cache -Eval -Eval Plus Plus Math a AR - ✔ ✔ 36.59 29.88 43.6 59.52 54.74 26.40 41.79 b Bidirectional - ✔ ✘ 15.85 12.20 16.2 24.34 28.96 11.08 18.10 c Bidirectional - ✘ ✘ 19.51 15.24 17.2 24.34 28.20 11.22 19.29 d Block-wise ✘ ✔ ✔ 31.10 25.61 23.6 36.77 38.44 13.88 28.23 e Block-wise (2×) ✘ ✔ ✔ 26.22 22.56 26.0 42.33 36.69 12.56 27.73 f Block-wise ✔ ✔ ✔ 38.41 33.54 33.0 48.68 51.48 21.04 37.69 g Block-wise ✔ ✘ ✔ 39.02 34.76 34.0 48.15 52.99 21.56 38.41 Init from Qwen2.5 1.5B Init from Qwen3 4B in Tab. 1, compared to bidirectional attention in Row 4 25.1 24.0 23.2 18.4 13.6 9.7 4 42.0 41.5 39.4 37.3 34.5 30.1 60 35 (c), block-wise attention (even without clean context) 55 Training Block Size Training Block Size 8 36.3 35.4 36.0 34.8 32.7 32.1 30 8 57.8 57.6 57.6 56.3 55.1 52.0 in Row (d) can boost average accuracy by 8.94%. 16 37.5 38.2 38.4 37.6 37.7 37.9 16 59.6 59.5 59.7 59.8 59.4 57.2 50 25 When combined with other best practices, i.e., condi- 32 36.9 38.2 37.7 35.3 35.0 34.6 32 57.9 59.1 59.9 59.0 59.8 58.7 45 20 40 tioning on clean context and removing token shift in 64 35.2 35.8 35.9 35.1 34.9 34.7 64 58.4 59.7 60.2 60.6 60.3 59.4 15 35 Row (g), block-wise attention improves the average 128 33.4 33.8 33.7 33.1 33.2 32.0 128 58.7 59.3 59.8 59.1 58.9 58.6 10 accuracy over bidirectional attention by 19.12%. 4 8 16 32 64 128 4 8 16 32 64 128 Evaluation Block Size Evaluation Block Size This implies that block-wise attention better pre- Figure 3 | The average accuracy achieved by different serves block-wise causality and thus maintains the training–evaluation block size pairs. pretrained AR model’s abilities more effectively than bidirectional attention, in addition to the benefit of into the no-token-shift setting, and (2) predicting the native KV caching. Furthermore, visualizations of mask token itself (without token shift) is easier than weight changes after continuous pretraining in Fig. 2 predicting the next token of a masked position. We hy- (e) show that bidirectional attention leads to larger pothesize that the latter is harder because the model weight drifts from pretrained weights in both the at- must handle two tasks simultaneously: inferring the tention and FFN layers, ultimately causing larger mask token and predicting the following token. accuracy drops. The impact of clean context. Based on the Takeaways: When continuously pretraining comparison between Rows (d) and (f) in Tab. 1, con- from an AR model, a block-wise attention ditioning each block on clean context during training pattern with clean context and without token is critical, yielding a 9.46% accuracy improvement shift emerges as a promising training scheme over using noisy context, where both adopt block- to deliver dLMs. wise attention. We further train the noisy-context case with a doubled token budget, i.e., extend Row (d) to Row (e), to account for the increased sequence Based on this takeaway, we adopt this scheme by length caused by concatenating noisy and clean tokens default in the following study. in the setting of Row (f). However, comparing Rows (e) and (f) in Tab. 1 shows that doubling training to- 2.3. Analysis of the Optimal Block Sizes kens on corrupted context cannot effectively recover the accuracy, whereas training on fewer tokens with Building on the best training scheme in Sec. 2.2, the clean context yields substantially higher accuracy. next question is the optimal block size for training and evaluation. Intuitively, larger context sizes provide ``` ### Table 2 | Comparing token masking schemes based on the average accuracy Figure 7 | The average masking ```text Table 2 | Comparing token masking schemes based on the average accuracy Figure 7 | The average masking across six generation tasks under varying parallel decoding settings, measured probability of each token posi- in tokens per forward (TPF). tion within a block. tokens per denoising step using confidence-based sam- generation ability remains unclear. In this section, we pling [17], is presented in Tab. 2. study the training dynamics of dLMs by visualizing their task performance evolution during training. Observations. As shown in Tab. 2, we observe that (1) progressively increasing positional priors with Setting. We train Qwen2.5 1.5B for 200B tokens lower 𝜆 leads to improved average accuracy; (2) po- with the same setting as in Sec. 2.3, and evaluate sitional priors are particularly beneficial under more on both generation and likelihood-based tasks, where aggressive parallel decoding settings, with up to a accuracy is computed by estimating and selecting 4.38% average accuracy improvement; and (3) posi- the largest likelihood among multiple choices. We tional priors should not be blindly increased, as the visualize the accuracy evolution on likelihood tasks extreme case of fully right-to-left masking leads to and the accuracy–efficiency (NFE) trade-off across poor results, likely because the model is forced to different training token budgets in Fig. 8. train only on the hard cases at the block end without Observations and analysis. We observe that (1) learning to exploit the bidirectional context. with relatively low training cost (on the order of 10B These experiments indicate that positional priors tokens), dLMs converted from pretrained AR models are helpful but must be introduced properly. The can largely recover task accuracy. (2) Longer training key contribution of our method here is to introduce with more iterations consistently improves likelihood this design factor, and we hope it can inspire more estimation and yields higher accuracy on likelihood- advanced and automated schemes in the future. based tasks. The average accuracy on generation tasks, without considering parallel token generation (i.e., the rightmost points of each curve in Fig. 8 Takeaways: dLMs exhibit a left-to-right ten- (b–d)), also improves, though with fluctuations on dency during parallel generation due to the certain tasks. (3) Improved likelihood estimation al- autoregressive nature of language, and mim- lows for more aggressive parallel token generation, icking this tendency in training can boost gen- as reflected in the enhanced accuracy–NFE trade-off eration quality. with longer training. This indicates that stronger likelihood estimation produces more accurate and re- liable confidence scores, thereby improving generation 4. Training Dynamics Analysis quality under confidence-based sampling. dLM training with the objective in Eq. 1 improves the This also indicates that parallel token generation masked denoising likelihood under noisy conditions, ability is another dimension for evaluating a dLM’s but how this improved likelihood estimation translates performance: dLMs with comparable accuracy when into downstream task accuracy and parallel token denoising one token per step can exhibit notable accu- (a) Likelihood Tasks 0.55 (b) GSM8K (c) MBPP (d) HumanEval 0.75 0.40 0.70 0.50 0.35 20B 20B 0.35 20B 0.65 40B 0.30 40B 40B Accuracy Accuracy Accuracy Accuracy 0.45 60B 60B 60B 0.60 80B 0.25 80B 0.30 80B 0.40 100B 100B 100B 0.55 120B 0.20 120B 120B ARCE 140B 140B 0.25 140B 0.50 ARCC 0.35 160B 160B 160B Hellaswag 180B 0.15 180B 180B 0.45 PIQA 0.30 200B 200B 0.20 200B ``` ### Table 3 | Benchmarking against SOTA AR models and dLMs on 12 tasks spanning coding, math, factual ```text Table 3 | Benchmarking against SOTA AR models and dLMs on 12 tasks spanning coding, math, factual knowledge, and commonsense reasoning (CR), reporting average accuracy per category. TPF denotes tokens per forward, and TPS refers to throughput measured on an NVIDIA H100 GPU with a batch size of 1. Detailed per-task accuracy is provided in Appendix B. Type Model TPF TPS (tok/sec) Coding Math MMLU CR Avg. Llama3.2 1B 1.00 143.91 24.45 4.98 30.98 60.62 34.24 SmolLM2 1.7B 1.00 112.84 21.06 30.97 49.99 68.44 40.14 AR Qwen2.5 0.5B 1.00 99.93 31.90 25.97 47.65 55.31 41.98 Qwen2.5 1.5B 1.00 73.03 42.17 46.98 60.96 66.00 54.47 1.00 68.52 42.33 42.60 57.63 62.58 52.09 dLM Efficient-DLM 1.5B 2.33 158.89 42.33 42.28 57.63 62.58 52.04 2.69 184.48 41.79 41.80 57.63 62.58 51.77 Qwen3 1.7B 1.00 71.59 54.22 54.15 62.53 64.99 59.39 AR Qwen3 4B 1.00 47.13 63.85 66.27 73.19 70.91 67.97 Qwen3 8B 1.00 42.51 68.45 69.87 76.93 73.71 71.58 LLaDA 8B 1.00 25.04 38.10 49.13 65.86 68.50 54.92 dLM Dream 7B 1.00 28.11 58.92 58.39 67.00 72.83 65.30 1.00 44.13 62.08 67.98 71.80 70.87 67.54 dLM Efficient-DLM 4B 2.52 119.33 61.37 68.56 71.80 70.87 67.39 3.01 130.24 60.96 68.10 71.80 70.87 67.18 1.00 39.99 67.36 69.22 77.22 74.88 71.62 dLM Efficient-DLM 8B 2.74 109.78 65.64 68.52 77.22 74.88 70.93 3.27 130.71 64.95 68.21 77.22 74.88 70.65 8 Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed Efficient-DLM 8B Dream 7B LLaDA 8B Qwen3 1.7B Qwen3 8B GSM8K MinervaMath HumanEval HumanEvalPlus 0.5 0.6 Accuracy 0.8 0.6 0.4 0.4 0.3 0.4 0.7 50 100 150 100 200 50 100 50 100 150 Throughput Throughput Throughput Throughput Figure 9 | Visualizing the accuracy-throughput trade-off of different models across different generation tasks. spectively. More benchmarks with SOTA dLMs plus summarize their impact in Tab. 5, which performs Fast-dLLM [17] are in Appendix C. AR-to-dLM conversion on top of Qwen3 4B. We start from the baseline setting (Dream’s bidirectional mod- 5.2. One-for-All Flexibility: Adaptive Accu- eling with 25B training tokens) and progressively add racy–Efficiency Trade-offs each component, reporting accuracy across six math and coding tasks. As observed in Tab. 5, proper atten- Beyond efficiency, another advantage of dLMs is their tion patterns (with appropriate block-size selection), one-for-all flexibility: a single dLM can balance ac- removing token shift, adding position-dependent to- curacy and throughput to suit different deployment ken masking, and longer training all contribute to scenarios. This is achieved by controlling parallel to- successful AR-to-dLM conversion. ken generation via a confidence threshold [17]. Fig. 9 shows the one-for-all flexibility of our Efficient-DLM 8B across four math and coding tasks. We observe 6. Related Work that a single Efficient-DLM 8B achieves better accu- racy–throughput frontiers than the AR Qwen3 family Diffusion language models. To overcome the from 1.7B to 8B, demonstrating its promise for one- token-by-token decoding nature of AR LMs, dif- for-all deployment. Throughput results under large fusion LMs, both continuous [25, 26, 27] and dis- batch sizes are provided in Appendix C. crete [28, 1, 2, 29, 18], have been proposed to per- form non-AR decoding and thus enable parallel token 5.3. Advantages of dLMs in Text Embedding generation. Among them, masked dLMs [1, 2, 3, 4] have been successfully scaled up (e.g., LLaDA [3] and We further highlight that, thanks to their ability for Dream [4]). Follow-up work has further explored al- bidirectional modeling, dLMs are more promising ternative dLM paradigms [6, 7, 9], and scaled them than AR models for tasks requiring high-quality text to larger generalists [30] or domain-specific specialists embeddings. To demonstrate this, we evaluate our such as coding agents [31, 32, 33, 34], explored ded- Efficient-DLM against AR Qwen models on text em- icated reinforcement learning schemes [35, 36], and bedding tasks, benchmarking 15 datasets from the extended them to more modalities [37, 38]. Compared MTEB benchmark [23] across six categories, following to AR LMs, diffusion LMs have been demonstrated to ``` ### Tab. 4, we observe a clear advantage of dLMs: at the and show improved performance in planning [4] and ```text Tab. 4, we observe a clear advantage of dLMs: at the and show improved performance in planning [4] and 1.5B and 4B scales, Efficient-DLM outperforms AR text embedding [40]. Qwen models of the same sizes by 7.71% and 9.91% on average, respectively. These results also highlight the Diffusion language model acceleration. De- broader promise of dLMs for other sequence modeling spite the acceleration potential of large dLMs [3, 4], tasks that require bidirectional information. the gap between bidirectional attention and KV caching, along with the one-token-per-step denois- ``` ### Table 4 | Comparing our Efficient-DLM and AR Qwen ing process, limits their achievable speed-up. To ad- ```text Table 4 | Comparing our Efficient-DLM and AR Qwen ing process, limits their achievable speed-up. To ad- models on text embedding tasks [23]. dress these challenges, dedicated caching strategies for Pair dLMs [41, 42, 17] have been developed to reuse com- Model Retr. Ranking Clust. Class. STS Avg. Class. putations and approximate bidirectional attention. Qwen2.5 1.5B 20.69 40.01 21.42 24.59 31.22 39.33 29.54 In addition, to realize the potential of parallel token Efficient-DLM 1.5B 18.67 43.67 23.58 56.76 31.70 49.14 37.25 Qwen3 4B 19.46 39.90 21.77 33.94 29.13 40.56 30.79 generation, confidence-based sampling [17], guidance Efficient-DLM 4B 20.17 45.05 23.91 65.59 42.22 47.27 40.70 from AR models [43], and adaptive decoding with cer- tainty and positional priors [44] have been proposed. 5.4. Ablation Study of Different Components Beyond these training-free methods, [10, 4] propose initializing dLMs from AR models with token shifts to We have analyzed and demonstrated the impact of accelerate dLM training. Block Diffusion [5] combines each component of Efficient-DLM in Sec. 2–4. We also AR and diffusion by performing block-wise AR and 9 Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed ``` ### Table 5 | Ablation study of different AR-to-dLM components on Qwen3 4B by progressively adding each ```text Table 5 | Ablation study of different AR-to-dLM components on Qwen3 4B by progressively adding each component on top of the baseline setting (Dream’s bidirectional modeling with 25B training tokens). Setting HumanEval HumanEval Plus MBPP MBPP Plus GSM8K Minerva Math Avg Bidirectional (Dream’s setting) 39.02 32.32 39.60 50.00 67.40 39.17 44.59 + Block-wise w/ clean context 53.66 50.36 55.60 69.70 78.39 46.33 59.01 + Remove token shift 56.10 51.22 54.60 69.84 82.87 47.02 60.27 + Position-dependent masking 60.37 54.27 59.00 71.43 81.12 45.92 62.02 + Scale to 300B tokens 60.98 56.71 60.00 70.63 86.43 49.54 64.05 in-block diffusion to support native KV caching, and [5] Marianne Arriola, Aaron Gokaslan, Justin T Chiu, concurrent works [17, 45, 46] also convert pretrained Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar AR models or dLMs into block-wise dLMs. Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 7. Conclusion 2025. This work systematically explores how to convert pre- [6] Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, trained AR models into dLMs that achieve faster Johnna Liu, Deepansha Singh, Zhoujun Cheng, generation while retaining strong accuracy. By in- Zhengzhong Liu, Eric Xing, John Thickstun, and troducing a continuous pretraining scheme with a Arash Vahdat. Esoteric language models. arXiv block-wise attention pattern, along with a position- preprint arXiv:2506.01928, 2025. dependent token masking strategy that narrows the [7] Subham Sekhar Sahoo, Justin Deschenaux, Aaron training–test gap, we provide a principled framework Gokaslan, Guanghan Wang, Justin Chiu, and for delivering dLMs with both strong accuracy and Volodymyr Kuleshov. The diffusion duality. arXiv speed, resulting in the Efficient-DLM model family. preprint arXiv:2506.10892, 2025. Through comprehensive analyses of attention pat- terns, training dynamics, and other design choices, [8] Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian our findings offer actionable insights that we hope will Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. guide the community toward building efficient and Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024. scalable dLMs. Beyond serving as a practical recipe for AR-to-dLM conversion, our results highlight the [9] Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, broader opportunity to rethink pretraining, masking, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, and and decoding strategies for dLMs in order to realize Zhi-Ming Ma. Any-order gpt as masked diffusion their promise as alternatives to AR models. model: Decoupling formulation and architecture. arXiv preprint arXiv:2506.19935, 2025. References [10] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Ji- acheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin [1] Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuan- Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng jing Huang, and Xipeng Qiu. Diffusionbert: Improv- Kong. Scaling diffusion language models via adapta- ing generative masked language models with diffusion tion from autoregressive models. In The Thirteenth models. arXiv preprint arXiv:2211.15029, 2022. International Conference on Learning Representa- tions, 2025. [2] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron [11] Qwen Team. Qwen2.5: A party of foundation models, Gokaslan, Edgar Marroquin, Justin Chiu, Alexander September 2024. Rush, and Volodymyr Kuleshov. Simple and effec- tive masked diffusion language models. Advances in [12] NVIDIA Nemotron Nano. Efficient hybrid mamba- Neural Information Processing Systems, 37:130136– transformer reasoning model. arXiv preprint 130184, 2024. arXiv:2508.14444, 2025. [3] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, [13] Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Cheng, Liping Tang, Guowei He, Zhengzhong Liu, Wen, and Chongxuan Li. Large language diffusion and Eric P. Xing. Megamath: Pushing the limits of models. arXiv preprint arXiv:2502.09992, 2025. open math corpora. arXiv preprint arXiv:2504.02807, 2025. Preprint. [4] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. [14] Kazuki Fujii, Yukito Tajima, Sakae Mizuki, Hinari Dream 7b: Diffusion large language models. arXiv Shimada, Taihei Shiotani, Koshiro Saito, Masanari preprint arXiv:2508.15487, 2025. Ohi, Masaki Kawamura, Taishi Nakamura, Takumi 10 Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed Okamoto, Shigeki Ishida, Kakeru Hattori, Youmi Ma, [24] Parishad BehnamGhader, Vaibhav Adlakha, Marius Hiroya Takamura, Rio Yokota, and Naoaki Okazaki. Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Rewriting pre-training data boosts llm performance Siva Reddy. LLM2Vec: Large language models are in math and code, 2025. secretly powerful text encoders. In First Conference on Language Modeling, 2024. [15] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Bider- man, Sid Black, Anthony DiPofi, Charles Foster, Lau- [25] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S rence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Liang, and Tatsunori B Hashimoto. Diffusion-lm Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, improves controllable text generation. Advances in ``` ### Table 6 | Benchmarking against SOTA AR models and dLMs on 12 tasks spanning coding, math, factual ```text Table 6 | Benchmarking against SOTA AR models and dLMs on 12 tasks spanning coding, math, factual knowledge, and commonsense reasoning. TPF denotes tokens per forward, and TPS refers to throughput measured on an NVIDIA H100 GPU with a batch size of 1. This table is a complement to Tab. 3. Coding Math Factual Commonsense Reasoning Type Model TPF Throughput (t/s) Avg HumanEval HumanEval Plus MBPP MBPP Plus GSM8K Minerva Math MMLU ARC-E ARC-C Hellaswag PIQA Winogrande Llama3.2 1B 1.00 138.72 17.68 14.63 26.60 38.89 5.69 4.28 30.98 65.28 36.35 63.76 74.43 63.30 36.82 AR Qwen2.5 0.5B 1.00 95.40 27.44 25.61 29.60 44.97 37.45 14.48 47.65 64.77 31.83 52.25 70.02 57.70 41.98 Qwen2.5 1.5B 1.00 73.03 35.98 29.88 43.60 59.23 64.97 28.98 60.96 75.17 45.05 67.90 76.12 65.75 54.47 2.33 158.89 41.46 35.37 38.80 53.70 57.92 26.64 57.63 75.00 45.31 59.43 71.44 61.72 52.04 dLM (Ours) Efficient-DLM 1.5B 2.69 184.48 41.46 35.98 37.60 52.12 58.07 25.52 57.63 75.00 45.31 59.43 71.44 61.72 51.77 Qwen3 1.7B 1.00 71.59 48.17 41.46 55.80 71.43 73.54 34.76 62.53 73.32 44.62 66.43 75.63 64.96 59.39 AR Qwen3 4B 1.00 44.99 57.32 50.61 66.80 80.69 85.44 47.10 73.19 79.12 51.62 73.66 78.07 72.06 67.97 Qwen3 8B 1.00 42.51 64.63 56.71 69.40 83.07 86.73 52.94 76.93 81.90 53.16 78.59 79.22 75.69 71.58 LLaDA 8B 1.00 25.04 32.32 27.44 40.80 51.85 70.96 27.30 65.86 73.78 49.15 71.05 73.88 74.66 54.92 dLM Dream 7B 1.00 28.11 54.88 49.39 56.80 74.60 77.18 39.60 67.00 82.20 59.13 73.73 75.52 73.56 65.30 2.52 119.33 61.59 56.71 57.60 69.58 87.57 49.56 71.80 81.52 55.80 69.02 75.46 72.53 67.39 dLM (Ours) Efficient-DLM 4B 3.01 130.24 60.98 56.10 57.20 69.58 87.19 49.02 71.80 81.52 55.80 69.02 75.46 72.53 67.18 1.00 40.00 68.29 63.41 61.00 76.72 88.32 50.12 77.22 84.89 61.86 72.53 77.53 77.58 71.62 dLM (Ours) Efficient-DLM 8B 2.74 109.78 64.02 59.15 63.20 76.19 86.73 50.30 77.22 84.89 61.86 72.53 77.53 77.58 70.93 3.27 130.71 63.41 58.54 62.20 75.66 86.35 50.06 77.22 84.89 61.86 72.53 77.53 77.58 70.65 Efficient-DLM 8B Dream 7B LLaDA 8B Qwen3 8B Qwen3 1.7B Batch Size 1 Batch Size 8 Batch Size 16 Batch Size 32 0.8 Accuracy 0.7 0.6 50 100 150 0 250 500 750 1000 0 500 1000 0 500 1000 1500 2000 Throughput Throughput Throughput Throughput Figure 11 | Visualizing the accuracy-throughput trade-off under different batch sizes on the GSM8K dataset. D. The Impact of Initial LR visualize the training loss defined in Eq. 1 for diffusion Qwen2.5 1.5B trained with a block size of 16, alongside When initializing from pretrained AR models, the the AR Qwen2.5 1.5B trained with an AR loss. learning rate for continuous pretraining is a key hyper- As shown in Fig. 12, which shows the average loss of parameter, as it controls the speed of weight changes that affect both the preservation of the pretrained the first 256 tokens in training sequences, we observe models’ abilities and the adaptation to dLMs’ new that (1) in AR models, the initial tokens incur higher attention patterns. We perform an ablation study on loss due to the lack of context, while the loss of later Qwen3 4B trained for 25B tokens with different initial tokens becomes more uniform; and (2) in dLMs, the loss follows a periodic pattern aligned with block learning rates using a cosine learning rate schedule. boundaries, where later tokens within each block show Observations and analysis. As shown in Tab. 7, higher loss due to limited clean context, consistent we find that there exists a sweet-spot learning rate set- with Fig. 6 (c). In addition, similar to AR models, the ting, e.g., 1e-5 in our case, that balances both aspects initial tokens of the entire sequence also experience mentioned above. Intuitively, overly large learning higher loss from insufficient context. rates cause greater weight drifts and degrade the pre- trained models’ original abilities, while overly small AR dLM learning rates cannot effectively adapt to the new attention pattern. We also note that for any design factors in continuously training a pretrained model into dLMs, these two aspects should be carefully bal- anced to achieve decent final accuracy. Based on this Figure 12 | Visualizing the loss distributions over set of experiments, we adopt 1e-5 as the default initial token positions of AR models and dLMs. learning rate throughout the main manuscript. F. AR-to-dLM Conversion via E. Loss Distributions across Tokens Parameter-Efficient Tuning To study the difference in loss distributions across Motivated by the relatively small weight changes ob- token positions between AR models and dLMs, we served in Sec. 2.2, we investigate whether parameter- 14 ``` ### Table 7 | Results of continuous pretraining with different initial learning rates on Qwen3 4B for 25B tokens. ```text Table 7 | Results of continuous pretraining with different initial learning rates on Qwen3 4B for 25B tokens. Init LR HumanEval HumanEval Plus MBPP MBPP Plus GSM8K Minerva Math Avg 1.00E-04 49.39 43.90 44.20 56.08 72.56 39.54 50.95 3.00E-05 54.27 49.39 52.40 67.46 77.48 38.56 56.59 1.00E-05 57.93 51.22 54.40 71.96 81.73 46.54 60.63 3.00E-06 56.10 50.61 54.60 67.99 83.93 47.44 60.11 1.00E-06 45.73 42.68 47.20 66.67 81.12 43.94 54.56 ``` ### Table 8 | Comparison of different dLM training schemes on Qwen2.5 1.5B. This table extends Tab. 1, with ```text Table 8 | Comparison of different dLM training schemes on Qwen2.5 1.5B. This table extends Tab. 1, with Rows (h) and (i) added to present the LoRA tuning results. Row Attn Clean Token KV LoRA Human Human MBPP Minerva MBPP GSM8K Avg ID Pattern Context Shift Cache Rank -Eval -Eval Plus Plus Math a AR - ✔ ✔ - 36.59 29.88 43.6 59.52 54.74 26.40 41.79 b Bidirectional - ✔ ✘ - 15.85 12.20 16.2 24.34 28.96 11.08 18.10 c Bidirectional - ✘ ✘ - 19.51 15.24 17.2 24.34 28.20 11.22 19.29 d Block-wise ✘ ✔ ✔ - 31.10 25.61 23.6 36.77 38.44 13.88 28.23 e Block-wise (2×) ✘ ✔ ✔ - 26.22 22.56 26.0 42.33 36.69 12.56 27.73 f Block-wise ✔ ✔ ✔ - 38.41 33.54 33.0 48.68 51.48 21.04 37.69 g Block-wise ✔ ✘ ✔ - 39.02 34.76 34.0 48.15 52.99 21.56 38.41 h Block-wise (LoRA) ✔ ✘ ✔ 16 30.49 25.61 20.60 30.95 43.82 16.08 27.93 i Block-wise (LoRA) ✔ ✘ ✔ 64 28.66 25.61 24.40 40.21 48.14 17.64 30.78 efficient tuning can effectively convert pretrained AR models into dLMs. To this end, we apply Low-Rank Adaptation (LoRA) [47] to all linear layers in atten- tion/FFN modules, combined with the best attention pattern identified in Sec. 2.2, i.e., block-wise atten- tion conditioning on clean context without token shift. All other parameters are frozen, except for the em- bedding layer, normalization operators, and the final model head, which we find must remain trainable for effective adaptation. Observations and analysis. We extend Tab. 1 into Tab. 8 by including LoRA tuning results with two different ranks in Rows (h) and (i). We ob- serve that LoRA tuning achieves reasonably good performance for AR-to-dLM conversion. Specifically, LoRA with rank 64 (Row i) surpasses the full-model training results of fully bidirectional attention and block-wise attention without clean context, while re- maining 7.63% behind the full-model training results of the best scheme, i.e., block-wise attention with clean context. These results indicate that (1) with proper training schemes, even parameter-efficient tun- ing can yield competitive dLMs, and (2) full-model training remains necessary to obtain strong dLMs. 15 ``` ## reference/elf/ELF- Embedded Language Flows ### Table 1: Results on machine translation and summarization. We evaluate ELF-B on WMT14 ```text Table 1: Results on machine translation and summarization. We evaluate ELF-B on WMT14 German-to-English (De-En) translation and XSum summarization, comparing against baselines of similar parameter scale. † denotes results taken directly from prior work and is the default source for De-En, while ‡ denotes results we reproduced using public codebases and is the default source for XSum. For XSum, we additionally report the standard error across evaluation examples when available. ELF achieves the best performance on both settings. Unconditional Generation Gen. PPL: 20.9 H: 5.2 ....But I still haven't found a book that I don't think would be relevant to write. Maybe I could even do a very small survey and develop a book based on my experience. Is there anything to this that you would say that's relevant to your work?.....Jesse Leren is a 30-year-old journalist and student writer. She was born in Sydney and is interested in becoming a member of the writing community. She is writing around the world for students in Australia and abroad....... Translation BLEU: 71.2. Context: Es zeigt einen Ring aus schwarzen Löchern, 430 Millionen Lichtjahre von der Erde entfernt. Reference: It shows a ring of black holes, 430 million light years away from the Earth. Generated: It shows a ring of black holes, 430 million lights years from the Earth. Summarization ROUGE-1: 66.7 ROUGE-2: 38.7 ROUGE-L: 60.6 Context: Joe Cardle slotted in his eighth of the season before setting up Kallum Higginbotham to extend the Pars' lead. Dumbarton's Craig Barr hit the crossbar with a second- half header but the Sons then had Andy Dowie sent off for two bookings. Substitutes Nicky Clark and Andy Ryan added late goals to add gloss to the visitors' victory. That is four wins on the bounce for Allan Johnston's men and on this form Dunfermline appear to be more than capable of sustaining a promotion challenge. Of the six players playing in.... Reference: Dunfermline climbed to the top of the Championship with an emphatic victory at Dumbarton. Generated: Dunfermline climbed off the bottom of the Championship with a comfortable victory at Dumbarton to move within seven points. Figure 8: Qualitative examples of text generated by ELF-B. We show an unconditional sample, a German-to-English translation example, and a summarization example, along with their automatic evaluation metrics. Some text is omitted due to space limits; see Appendix E for more examples. fuSeq [79] and CDCD [13]). Some results are taken from the literature and others are reproduced from public codebases. See Appendix Tab. 8 for a summary. We use the best sampling configuration selected on the validation set: a 64-step ODE sampler with the self-conditioning CFG scale set to 1 and the input-condition CFG scale set to 2. We show the results in Tab. 1. On WMT14 De–En, ELF-B achieves a BLEU score of 26.4, outper- forming all compared baselines. On XSum, ELF-B also outperforms all compared baselines across all ROUGE metrics. These results demonstrate the effectiveness of ELF on conditional generation tasks. Qualitative examples in Fig. 8 show that ELF-B generally follows the input context and produces outputs that semantically align with the ground-truth references. 5 Conclusion We introduced Embedded Language Flows (ELF), a continuous diffusion language model that formulates language generation in continuous embedding space using continuous-time Flow Matching. In contrast to prior DLMs, ELF keeps the denoising trajectory continuous and applies discretization only at the final step, enabling straightforward adaptation of techniques from continuous diffusion models. Empirically, compared with leading discrete DLMs and existing continuous DLMs, ELF achieves a strong quality–efficiency trade-off across language generation tasks, attaining lower generative perplexity with fewer sampling steps and fewer training tokens. These results suggest that continuous DLMs remain a promising direction for diffusion-based language modeling. Acknowledgments and Disclosure of Funding We thank Mingyang Deng, Belinda Li, Itamar Pres, and Laura Ruis, for their helpful feedback and insightful discussions. We thank Google TPU Research Cloud (TRC) for granting us access to TPUs. 9 References [1] Xinyue Ai, Yutong He, Albert Gu, Ruslan Salakhutdinov, J Zico Kolter, Nicholas Matthew Boffi, and Max Simchowitz. Joint distillation for fast likelihood evaluation and sampling in flow-based models. In ICLR, 2026. 6 [2] Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. JMLR, 2025. 2, 3, 15 [3] Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In ICLR, 2023. 1, 2, 4 [4] Marianne Arriola, Yair Schiff, Hao Phung, Aaron Gokaslan, and Volodymyr Kuleshov. Encoder- decoder diffusion language models for efficient training and inference. In NeurIPS, 2025. 3, 8, 9, 27 [5] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurIPS, 2021. 1, 2, 3 ``` ### Table 2: Survey of continuous diffusion and flow-based language models. We summarize ```text Table 2: Survey of continuous diffusion and flow-based language models. We summarize representative continuous diffusion and flow-based language models along several design axes. Process denotes the diffusion or flow process, with green indicating continuous-time formulations and red indicating discrete-time formulations. State denotes the continuous state in which denoising is performed. Train per-step discr. marks methods that convert intermediate denoising states to token predictions during training and apply token-level supervision such as cross-entropy loss at intermediate steps. Infer. per-step discr. marks methods that project intermediate sampling states back to token-aligned states during generation. Sep. dec. marks methods that require a separately trained decoder to map latent representations back to text. Blank entries indicate absence. * denotes autoregressive or block-autoregressive generation. A Continuous Diffusion Language Model Survey Survey details. We provide a detailed survey in Tab. 2. The survey summarizes representative continuous diffusion and flow-based language models along several design axes, including the underlying diffusion or flow process, the continuous state in which denoising is performed, whether intermediate denoising states are discretized during training or inference, and whether a separately trained decoder is required to map latent states back to text. In particular, the Train per-step discr. and Infer. per-step discr. columns distinguish two different uses of intermediate discretization. Train per-step discr. indicates that intermediate denoising states are 15 input tokens � �� �� � �1 �2 embed corrupt self-conditon add control .... ELF �� or .... �� corrupt �� self-condition add control �� �� � time embed �� �’ add condition project CFG scale model mode Figure 9: Illustration of our training pipeline. Starting from the clean embeddings x, we apply different noise schedules in the two modes to obtain corrupted embeddings zt . We then apply self- conditioning by concatenating either 0 or the previous prediction x̂′ along the channel dimension, and project the concatenated embeddings back to the original dimension to form ẑt . Next, we prepend control tokens to the embedding sequence, including time tokens in [0, 1], CFG scale tokens in [0.5, 5], and model-mode tokens indicating either denoising or decoding. The resulting sequence is fed into ELF to produce the final prediction x̂, which is supervised using either a denoising loss LMSE or a token-wise cross-entropy loss LCE . mapped to token predictions during training and supervised with token-level objectives such as cross- entropy loss. This provides direct vocabulary-level guidance, but also couples intermediate denoising states to categorical predictions. Infer. per-step discr. indicates that intermediate sampling states are explicitly projected back to token-aligned representations during generation, such as nearest-neighbor rounding in embedding space or argmax projection on a simplex. Methods without inference-time per-step discretization keep the sampling trajectory continuous and discretize only at the final step. The Sep. dec. column indicates whether a method requires a separately trained decoder to map continuous latent representations back to discrete text. Positioning of ELF. Tab. 2 shows that existing continuous DLMs differ substantially in where the denoising process is defined and how continuous states are mapped back to text. Many embedding- space and simplex-based methods use training-time per-step discretization through token-level objectives, commonly cross-entropy, at intermediate denoising steps. These objectives provide direct token-level guidance, while making the denoising trajectory more tightly coupled to vocabulary-level prediction. Latent Diffusion LMs often avoid such per-step vocabulary supervision, but typically rely on DDPM-style or score-based formulations with DDPM noise schedules [26, 47] and require a separately trained latent-to-text decoder, such as an autoregressive decoder, non-autoregressive decoder, or latent decompressor, to recover discrete tokens. ELF occupies a different design point. It formulates language generation as continuous-time Flow Matching in a frozen contextual embedding space and keeps the sampling trajectory continuous, applying discretization only at the final decoding step. Unlike prior latent Diffusion LMs, ELF does not require a separately trained decoder: a single shared-weight network performs intermediate denoising and recovers tokens at the final step through the unembedding layer. ``` ### Table 3: ELF Model configurations across different scales. ```text Table 3: ELF Model configurations across different scales. is especially significant in the few-step regime. These results suggest that the logit-normal time schedule improves sampling efficiency and final sample quality, likely because it better aligns the inference-time trajectory with the training-time schedule and allocates more sampling steps to noisier time steps. SDE noise re-injection scale. For SDE sampling, we introduce a noise re-injection scale hyperpa- rameter γ that controls the amount of stochasticity injected at each sampling step, as discussed in Sec. B.2. Intuitively, increasing γ introduces more stochasticity, while γ = 0 reduces to deterministic ODE sampling. As shown in Fig. 15b, γ controls the generative perplexity–entropy trade-off: within a moderate range, larger γ leads to lower generative perplexity while slightly reducing entropy. We hypothesize that the noise re-injection process helps correct early denoising errors, rather than deterministically amplifying imperfect trajectories as in ODE sampling. We therefore choose γ = 1.0 as our default setting, which provides a strong balance between generative perplexity and entropy. C.7 CFG on Conditional Generation We further study the effect of CFG scale on conditional generation tasks. As shown in Fig. 16, increasing the CFG scale from 1 to 2 substantially improves performance on both WMT14 De-En and XSum, suggesting that stronger conditioning helps the model better follow the source input. However, further increasing the scale leads to a gradual decline in performance, indicating that overly strong guidance can hurt generation quality. Based on this trend, we use CFG scale 2 as the default setting for conditional generation. D Experimental Details D.1 Model Architecture Our model uses a standard Diffusion Transformer architecture [50]. We also incorporate pop- ular general-purpose improvements, including SwiGLU [61], RMSNorm [80], RoPE [67], and qk-norm [24]. We use in-context conditioning instead of adaLN-Zero [50] conditioning, which allows us to significantly reduce the number of parameters; for example, the ELF-B model size is reduced from 148M to 105M parameters. Tab. 3 summarizes the configurations of ELF across different model sizes. We report the Transformer depth, hidden size, number of attention heads, and parameter count. We also report the number of training epochs used on the OWT dataset for each variant. Larger models tend to learn faster in our setup, and therefore require fewer training epochs. D.2 Hyperparameters ELF pipeline hyperparameters. Tab. 4 summarizes the main hyperparameters used in the ELF pipeline, covering model architecture, diffusion settings, conditioning and guidance, and optimization details. Unless noted otherwise, all experiments in the paper follow this default configuration. We include these settings for completeness and to facilitate reproducibility. Inference-time settings for system-level comparison. For system-level comparison in Fig. 7, we use SDE sampling with time schedule enabled for all step budgets. We set the CFG scale to 3 for 8-, 16-, and 32-step generation. For SDE sampling, we use a stronger noise injection scale of γ = 2 in the very few-step regimes of 8 and 16 steps, and reduce it to γ = 1.5 for 32 steps, as longer denoising trajectories require less stochastic correction. For the system-level comparison in Tab. 1, we use 64-step ODE sampling with time schedule. We set the self-conditioning CFG scale to 1 and the input-condition CFG scale to 2. 24 Model Architecture Denoising and Decoding Config Model ELF-B Time schedule logit normal Model size 105M Denoiser (Pmean , Pstd ) (−1.5, 0.8) Encoder backbone T5-small Denoiser noise scale 2.0 Embedding dimension 512 Decoder (Pmean , Pstd ) (0.8, 0.8) Bottleneck dimension 128 Decoder noise scale 5.0 Model dimension 768 Denoiser vs. decoder prob. 0.8 vs. 0.2 Sequence length 1024 Conditioning and Guidance Optimization and Training Self-conditioning probability 0.5 Optimizer Muon Self-conditioning CFG range [0.5, 5] Learning rate 0.002 Num. of time tokens 4 Weight decay 0 Num. of model-mode tokens 4 Training epochs 5 Num. of CFG tokens 4 Global batch size 512 SDE γ 1.0 Learning rate schedule constant Warmup epochs 0.5 EMA decay 0.9999 Training device TPU v5p × 64 Training time 1.5 h per epoch ``` ### Table 4: Default training hyperparameters and setup for ELF-B on the OpenWebText dataset. ```text Table 4: Default training hyperparameters and setup for ELF-B on the OpenWebText dataset. Unless noted otherwise, all experiments in the paper follow this default configuration. Method Base training Distillation training Effective tokens Ratio MDLM [56] 512 × 1M × 1024 - 524.3B 11.6× Duo [57] 512 × 1M × 1024 - 524.3B 11.6× MDLM + SDTT [56] 512 × 1M × 1024 512 × 10K × 5 × 1024 550.5B 12.2× Duo + DCD [57] 512 × 1M × 1024 512 × 10K × 5 × 1024 550.5B 12.2× FLM [30] 512 × 1M × 1024 - 524.3B 11.6× FMLM [30] 512 × 1M × 1024 512 × 100K × 1024 576.7B 12.8× LangFlow [10] 512 × 1M × 1024 - 524.3B 11.6× ELF (ours) 5 × 9.04B - 45.2B 1.0× ``` ### Table 5: Estimated effective training tokens for ELF and the prior DLM baselines used in our system- ```text Table 5: Estimated effective training tokens for ELF and the prior DLM baselines used in our system- level comparison (Fig. 7c). We estimate base-training tokens as batch size × steps × sequence length; distillation / flow-map stages are added on top where applicable. Training-token budget for system-level comparison. Tab. 5 reports the estimated effective training tokens used by ELF and each baseline in Fig. 7c. We estimate base-training tokens as batch size × steps × sequence length and add distillation or flow-map stages on top where applicable. The OWT dataset contains roughly 9.04B tokens. With our default training schedule of 5 epochs, ELF therefore uses 45.2B effective training tokens. Thus, ELF requires roughly an order of magnitude fewer effective training tokens than the compared DLMs. D.3 Ablation Studies Setting We evaluate several choices of embedding representations for ELF, and report the implementation details as below. We also try two-stage training with a separate decoder. Unless specified, we keep other settings the same as the default ELF configuration. Scratch encoder. We train an encoder from scratch on OpenWebText [18] by following the original T5-small training pipeline [53]. The encoder is trained for 5 epochs with a learning rate of 1 × 10−3 , cosine learning rate schedule, 0.4 epoch warmup, and a batch size of 512. During ELF training, we apply channel-wise normalization to the encoder outputs. 25 Steps SC CFG γ Gen. PPL ↓ Entropy ↑ 8 3 2.0 67.32 ± 2.25 5.14 ± 0.085 16 3 2.0 33.66 ± 1.09 5.16 ± 0.026 32 3 1.5 24.08 ± 0.16 5.15 ± 0.002 ``` ### Table 6: System-level ELF performance reported as mean ± standard error (SE) over 6 independent ```text Table 6: System-level ELF performance reported as mean ± standard error (SE) over 6 independent evaluation runs (seeds 0–5; n = 6). ELF-B 105M ELF-M 342M ELF-L 652M Sampler SC CFG Gen. PPL Entropy Gen. PPL Entropy Gen. PPL Entropy 0.5 36.77 5.28 39.21 5.35 37.50 5.41 1.0 29.50 5.23 33.45 5.30 31.82 5.37 1.5 25.25 5.18 28.42 5.26 28.72 5.35 SDE 2.0 22.53 5.14 25.34 5.23 26.47 5.32 3.0 19.72 5.10 21.69 5.18 23.31 5.28 3.5 37.56 5.30 36.48 5.34 22.28 5.27 4.0 36.50 5.29 34.93 5.33 21.37 5.26 0.5 104.29 5.51 88.51 5.51 68.27 5.52 1.0 65.30 5.40 62.47 5.44 49.72 5.45 1.5 44.85 5.31 46.71 5.37 39.97 5.40 ODE 2.0 34.65 5.23 37.66 5.32 33.72 5.36 3.0 26.62 5.15 28.80 5.24 26.57 5.29 ``` ### Table 7: Scaling performance of generative perplexity (Gen. PPL) and unigram entropy for ELF ```text Table 7: Scaling performance of generative perplexity (Gen. PPL) and unigram entropy for ELF models of different sizes under SDE and ODE samplers with 64 sampling steps. The effect of self-conditioning (SC) CFG scaling diminishes beyond 3. Pretrained embedding layer. We use the frozen embedding table from the T5-small encoder as the token embedding layer. The embedding layer matrix is normalized, and the unembedding layer is trained separately. Gaussian embedding layer. We randomly initialize and freeze an embedding layer from a Gaussian distribution, with token-wise embedding mean 0 and standard deviation 1. The unembedding layer is trained separately using the decoder mode. Learnable embedding layer. We jointly train the embedding layer together with the denoiser and decoder modes. The unembedding layer is tied with the embedding layer: denoiser-mode updates affect the embedding layer, while decoder-mode updates affect the unembedding layer. To stabilize training, we apply normalization directly on the unembedding layer matrix at every step. Separate decoder. For the separate-decoder setting, we use a randomly initialized decoder architec- ture obtained by mirroring the T5-small encoder. We keep the encoder fixed, mask 20% of the input tokens, and add logit-normal noise to the latent representations with Pmean = 0.5 and Pstd = 1.0. The model is trained for 3 epochs with a learning rate of 3 × 10−4 and a cosine learning-rate schedule. The relative noise scale with respect to the normalized latent representations is set to 5.0. D.4 Reported Numbers System level comparison. Across 6 independent evaluation seeds, ELF shows highly consistent system-level behavior, as shown in Tab. 6. As the number of sampling steps increases from 8 to 32, the standard error (SE) decreases. The small standard errors—especially at 32 steps—suggest that these gains are robust to random seed variation and that the overall trend is reliable across runs. See ``` ### Tab. 6 for detailed numbers. ```text Tab. 6 for detailed numbers. Scaling behavior with CFG scales. The default setting for both sampling methods uses 64 sampling steps with time schedule. For the SDE sampler, we set γ = 1.0. The exact numbers are reported in 26 Config AR MDLM E2D2 Duo Architecture Codebase E2D2 E2D2 E2D2 Duo Duo Tokenizer Qwen3-0.6B Qwen3-0.6B Qwen3-0.6B T5-small T5-small Hidden Size 256 256 256 768 768 Intermediate Size 768 768 768 – – #Layers / Blocks 28 28 enc=20, dec=8 12 12 Sequence Length 64 64 64 64 64 Max Cond Length 1024 1024 1024 1024 64 Cond Embed – – – T5-small T5-small Training Dataset XSum XSum XSum XSum De-En Learning Rate 3e-4 3e-4 3e-4 3e-4 3e-4 LR Scheduler const const const const const Warmup Steps 1000 1000 1000 2500 2500 Global Batch Size 128 128 128 512 512 Optimizer DecoupledAdamW DecoupledAdamW DecoupledAdamW AdamW AdamW Loss Type NLL MDLM ELBO E2D2 ELBO Duo ELBO Duo ELBO Train Steps 500K 500K 500K 1M 1M Evaluation Sampling Strategy greedy predict_and_noise predict_and_noise Duo sampler Duo sampler Sampling Steps L = 64 (AR) ≈ L (first-hit) ≈ L (first-hit) 1000 1000 Block size 1 32 8 - - CFG Scale – – – 1.0 1.5 Checkpoint best best best best best EMA true true true true true ``` ### Table 8: Detailed training and evaluation configurations for conditional generation tasks of ```text Table 8: Detailed training and evaluation configurations for conditional generation tasks of our reproduced AR, MDLM, E2D2, and Duo baselines. AR, MDLM, and E2D2 are reproduced on XSum using the E2D2 [4] codebase and follow the configurations reported in the E2D2 paper. For Duo, we build on the original Duo [57] repository, add cross-attention conditioning and CFG, adapt the T5-small encoder to match our setting, and tune the hyperparameters to obtain the strongest reproduced results. ``` ### Tab. 7. Larger CFG scales improve generation quality by reducing Gen. PPL within a certain range. ```text Tab. 7. Larger CFG scales improve generation quality by reducing Gen. PPL within a certain range. The effect of CFG scaling reverses beyond 3. Only ELF-L benefits from increasing the CFG scale from 3 to 4. Thus, in most default ablation studies, we only consider CFG scales from 0.5 to 3. D.5 Conditional Generation Specifically, the WMT14 results for AR, MDLM, and E2D2 are taken from the E2D2 [4] paper, the SeqDiffuSeq result is taken from the LD4LG [41] paper, and the CDCD result is taken from the original CDCD [13] paper. For reproduced results, Duo [57] is implemented using the Duo codebase4 , while AR, MDLM, and E2D2 are reproduced using the E2D2 codebase5 . For a fair comparison, we reproduce all baselines using settings that are as close as possible to their original implementations, as summarized in Tab. 8. For AR, MDLM, and E2D2, we use the E2D2 codebase and follow the training and evaluation configurations reported in the E2D2 paper on XSum. Note that although E2D2 is primarily designed for semi-autoregressive generation, we find that MDLM also achieves its best performance under a semi-autoregressive setting (i.e., block size 32 with two-block generation); using single-block diffusion without semi-autoregressive generation degrades performance. For Duo, we start from the official Duo repository and adapt it to our conditional generation setting by adding cross-attention conditioning and classifier-free guidance, and by using a T5-small encoder for the conditioning input. During inference, we generate without 4 https://github.com/s-sahoo/duo 5 https://github.com/kuleshov-group/e2d2 27 t=0 strength will building building building building back played playedband bit choiceband bitband played playedband played bit bit The results was ab disturbing . EFuture after watching various games , I was pretty fierce withLI The results were flat striking . Immediately after watched the games , I was pretty determined with t=1 The results were particularly striking . Immediately after watching the games , I was very concerned about Figure 17: Denoising trajectory of ELF-B. As t increases from 0 to 1, ungrammatical sentences are progressively refined into fluent and grammatical text. semi-autoregressive decoding. We tune the main sampling and guidance hyperparameters and report the best reproduced results we obtain. E Qualitative Examples E.1 Denoising Trajectory Fig. 17 visualizes the intermediate predictions along ELF’s denoising process. Starting from repetitive tokens at t = 0, the model gradually forms semantically meaningful phrases, improves grammar, and refines word choices as t approaches 1. This trajectory illustrates how continuous diffusion generation progressively transforms noisy embeddings that decode to gibberish text into clean embeddings that decode to grammatical sentences. E.2 Unconditional Generation Examples on OpenWebText We provide three unconditional samples generated by ELF-B on OpenWebText, reported with their entropy and generative perplexity (Gen. PPL). The examples illustrate that ELF produces fluent, syntactically coherent, and topically consistent long-form text across diverse domains. ELF-B OWT entropy: 5.36 Gen. PPL: 21.04 The company has been developing a virtual sleep mode for its iPhone and iPad for years. This means that users can improve their quality of life without turningping off their fingers thanks to Google’s new virtual sleep technology. To make the experience a reality, virtual sleep mode was developed for Google, using a new built-in technology that includes real-time photography and shadow monitoring. This technology enables users to have a safe, comfortable look at where they sleep, even if you place the keyboard or a button under your fingers. Some sources point to the iPhone 6 and iPhone 6 as yet another example of the importance of virtual sleep mode in our everyday lives. This technology has been shown to be useful when staying busy on tight days, during difficult times or lying asleep on a hot night. This technology could also be used to improve sleep quality and help users improve quality of life. Editor’s note: This post has been updated to answer to relevant questions. Google says it will add virtual sleep mode to its iPhone and iPad in coming week. Google ``` ## reference/flash-dllm/Fast-dLLM- Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding ### Table 1 | Comprehensive benchmark results on the LLaDA-Instruct suite. Each cell presents the accuracy and the ```text Table 1 | Comprehensive benchmark results on the LLaDA-Instruct suite. Each cell presents the accuracy and the decoding throughput in tokens per second with relative speedup to the LLaDA baseline (bottom row, blue: tokens per second/orange: relative speedup). The highest throughput and speedup for each configuration are highlighted. Benchmark Gen Length LLaDA +Cache +Parallel +Cache+Parallel (Fast-dLLM) 256 79.3 79.5 79.2 78.5 GSM8K (5-shot) 6.7 (1×) 21.2 (3.2×) 16.5 (2.5×) 54.4 (8.1×) 512 77.5 77.0 77.6 77.2 3.2 (1×) 10.4 (3.3×) 18.6 (5.8×) 35.3 (11.0×) 256 33.5 33.3 33.4 33.2 MATH (4-shot) 9.1 (1×) 23.7 (2.6×) 24.8 (2.7×) 51.7 (5.7×) 512 37.2 36.2 36.8 36.0 8.0 (1×) 19.7 (2.5×) 23.8 (3.0×) 47.1 (5.9×) 256 41.5 42.7 43.9 43.3 HumanEval (0-shot) 30.5 (1×) 40.7 (1.3×) 101.5 (3.3×) 114.1 (3.7×) 512 43.9 45.7 43.3 44.5 18.4 (1×) 29.3 (1.6×) 57.1 (3.1×) 73.7 (4.0×) 256 29.4 29.6 28.4 28.2 MBPP (3-shot) 6.0 (1×) 17.0 (2.8×) 24.8 (4.1×) 44.8 (7.5×) 512 14.8 13.4 15.0 13.8 4.3 (1×) 10.1 (2.3×) 22.3 (5.1×) 39.5 (9.2×) 4. Experiments 4.1. Experimental Setup All experiments are conducted on an NVIDIA A100 80GB GPU. The proposed approach, Fast-dLLM, comprises two components: a Key-Value Cache mechanism and a Confidence-Aware Parallel Decoding strategy. The KV Cache component introduces a hyperparameter, the cache block size, varied between 4 and 32. The parallel decoding strategy uses a confidence threshold hyperparameter, explored in the range of 0.5 to 1.0. Unless otherwise specified, we use PrefixCache with block size of 32 and the threshold to 0.9. We evaluate Fast-dLLM on two recent diffusion-based language models: LLaDA [21], LLaDA-1.5 [44] and Dream [36]. Bench- Selected 80 22 marks include four widely-used datasets—GSM8K, MATH, Hu- No cache manEval, and MBPP, to assess performance across diverse rea- 20 GSM8K (5-shot) Accuracy 75 Throughput (tokens/s) soning and code generation tasks. We also test under varying 18 generation lengths to evaluate scalability and robustness. 70 16 3.3x Speedup In addition, we extend our evaluation to LLaDA-V [38], a multi- 14 65 modal variant of LLaDA tailored for vision-language reasoning 12 tasks. For this, we use two challenging multimodal benchmarks: 60 10 MathVista and MathVerse, which require solving math problems 8 grounded in complex visual scenes. 55 No cache 6 4 8 16 32 64 128 256 Inference throughput is measured as the average number of output Cache Block Size tokens generated per second, calculated over the full sequence until the end-of-sequence () token is reached. This met- Figure 4 | Impact of Cache Block Size on Accu- ric reflects true end-to-end decoding speed. All evaluations are racy and Throughput. The orange line illustrates conducted using the standardized lm-eval library to ensure con- the effect of varying cache block size on through- sistency and reproducibility. put, while the blue line depicts accuracy. 4.2. Main Results: Performance and Speed We report decoding performance and efficiency gains for Fast-dLLM on both the LLaDA-Instruct and Dream-Base models across the four benchmarks in Tables 1 and 2. 7 Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding ``` ### Table 2 | Comprehensive benchmark results on Dream-Base variants over four tasks with different generation lengths ```text Table 2 | Comprehensive benchmark results on Dream-Base variants over four tasks with different generation lengths (256 and 512). Each cell shows accuracy (top row) and decoding throughput in tokens per second with relative speedup to Dream-Base baseline (bottom row, blue: tokens per second/orange: relative speedup). Numbers in yellow indicate the highest throughput and speedup per configuration. Benchmark Gen Length Dream +Cache +Parallel +Cache+Parallel (Fast-dLLM) 256 75.0 74.3 74.2 74.8 GSM8K (5-shot) 9.1 (1×) 32.5 (3.6×) 14.2 (1.6×) 48.2 (5.3×) 512 76.0 74.3 73.4 74.0 7.7 (1×) 25.6 (3.3×) 14.6 (1.9×) 42.9 (5.6×) 256 38.4 36.8 37.9 37.6 MATH (4-shot) 11.4 (1×) 34.3 (3.0×) 27.3 (2.4×) 66.8 (5.9×) 512 39.8 38.0 39.5 39.3 9.6 (1×) 26.8 (2.8×) 31.6 (3.2×) 63.3 (6.5×) 256 49.4 53.7 49.4 54.3 HumanEval (0-shot) 23.3 (1×) 35.2 (1.5×) 45.6 (2.0×) 62.0 (2.8×) 512 54.3 54.9 51.8 54.3 16.3 (1×) 27.8 (1.7×) 29.8 (1.8×) 52.8 (3.2×) 256 56.6 53.2 53.8 56.4 MBPP (3-shot) 11.2 (1×) 34.5 (3.1×) 31.8 (2.8×) 76.0 (6.8×) 512 55.6 53.8 55.4 55.2 9.4 (1×) 26.7 (2.8×) 37.6 (4.0×) 73.6 (7.8×) Overall, introducing the KV Cache mechanism yields significant speed improvements for all tasks and sequence lengths, typically achieving a 2× to 3.6× speedup compared to the vanilla backbone. When the parallel decoding strategy is applied individually, we see additional acceleration, often pushing speedups to 4×–6× for the evaluated settings, particularly as the generation length increases. When both techniques are combined, the improvements become even more pronounced. On LLaDA, for example, combined KV Cache and parallel decoding methods boost throughput by up to 11× (GSM8K, length 512) and 9.2× (MBPP, length 512) over the standard baseline. Similarly, on Dream-Base, the largest throughput gains are observed on MBPP (7.8× at length 512) and GSM8K (5.6× at length 512). These results indicate that not only are our methods effective individually, but they are also highly complementary, resulting in the combined acceleration. 3.25 300 4.24 1.00 Ours 3.25 4.24 80 5.12 Selected 2 tokens per step 1.00 80 5.12 GSM8K (5-shot) Accuracy GSM8K (5-shot) Accuracy 6.17 250 4 tokens per step Selected 6.17 70 7.01 8 tokens per step 70 2 7.01 Inference Steps 200 4 60 Ours 60 2 tokens per step 4 tokens per step 150 50 8 tokens per step 50 100 3.25 Selected 40 5.12 4.24 40 50 7.01 6.17 Ours Fixed-Step Baseline (2/4/8 tokens) 30 30 Non-Parallel Baseline (1 token/step) 0 8 0.5 0.6 0.7 0.8 0.9 1.0 0.5 0.6 0.7 0.8 0.9 1.0 0 2 4 6 8 Threshold Threshold Average #Tokens per Step (a) (b) (c) Figure 5 | (a) The red line shows the GSM8K (5-shot) accuracy across different confidence thresholds. Numbers along the red line indicate the average number of tokens decoded at each step. The three dashed lines represent the accuracy of the baseline method when selecting the top 2, 4, or 8 tokens per step. (b) The number of inference steps required under varying confidence thresholds. (c) A comparison between our method and the baseline on GSM8K (5-shot) accuracy, plotted against the average number of tokens per step. Our method consistently outperforms the baseline. Importantly, these efficiency gains are achieved with negligible impact on accuracy. Across all benchmarks and settings, the accuracy of our accelerated methods remains within 1–2 points of the backbone, and in several cases, accuracy is even slightly improved. This demonstrates that the speedup comes at almost no cost to task performance, ensuring reliability for practical deployment. We also observe that longer sequences, which are common in few-shot and code generation scenarios, benefit proportionally more from our caching and parallelization techniques due to greater opportunities for ``` ### Table 3 | Performance and Speedup Comparison of LLaDA-V on MathVista and MathVerse. Each benchmark ```text Table 3 | Performance and Speedup Comparison of LLaDA-V on MathVista and MathVerse. Each benchmark includes results from Full Steps, Half Steps, and Fast-dLLM. Fast-dLLM significantly improves throughput (highlighted), with minimal accuracy loss. MathVista MathVerse Metric Full Steps Half Steps Fast-dLLM Full Steps Half Steps Fast-dLLM Accuracy (%) 59.2 59.7 56.6 28.5 28.3 28.6 Throughput (Speedup) 2.84 (1×) 5.56 (1.96×) 28.2 (9.9×) 2.75 (1×) 5.17 (1.88×) 23.3 (8.5×) In addition to text-only models, we evaluate Fast-dLLM on the multimodal LLaDA-V using the MathVista and MathVerse datasets, which require complex vision-language reasoning. As shown in Table 9, LLaDA-V shows a strong sensitivity to block size, with accuracy dropping by over 8% when reducing from 96 to 8 on MathVista. To address this, we retain a full block length and apply refresh-based updates instead of small-block caching. This yields up to 9.9× speedup with minimal accuracy degradation (Table 3). On MathVerse, accuracy is even slightly improved under Fast-dLLM, demonstrating the broad applicability of our method to multimodal reasoning tasks. Furthermore, the improvements generalize across model architectures (LLaDA and Dream), task types (math reasoning, program synthesis), and modalities (text and vision), confirming that Fast-dLLM is a practical and broadly applicable framework for accelerating masked diffusion-based language models. 4.3. Ablations and Analysis ``` ### Table 4 | Performance and Speedup Comparison on Table 5 | Impact of Generation Length on Accuracy ```text Table 4 | Performance and Speedup Comparison on Table 5 | Impact of Generation Length on Accuracy LLaDA Between 5-Shot and 8-Shot Settings at Gen- and Speedup Under 8-Shot for LLaDA. This table il- eration Length 1024. This table compares the accuracy lustrates the effect of varying generation lengths (256, and throughput speedups of different decoding strategies 512, and 1024) on decoding performance and efficiency under 5-shot and 8-shot configurations using a generation for different caching strategies under the 8-shot setting. length of 1024. The results demonstrate how increased Longer generation lengths lead to higher throughput gains, prefill length enhances the effectiveness of caching strate- especially for DualCache, validating the scalability of our gies, particularly for DualCache. approach. Parallel Decoding Parallel Decoding Setting. LLaDA Len. LLaDA No Cache PrefixCache DualCache No Cache PrefixCache DualCache 77.0 77.4 75.2 74.7 77.6 77.9 77.3 76.9 5-shot 256 1.1 (1×) 11.7 (10.6×) 14.4 (13.1×) 21.6 (19.6×) 4.9 (1×) 16.4 (3.3×) 49.2 (10.0×) 46.3 (9.4×) 77.3 78.0 75.7 76.0 78.9 78.9 74.8 75.4 8-shot 512 0.7 (1×) 9.3 (13.3×) 13.0 (18.6×) 19.3 (27.6×) 2.3 (1×) 14.0 (6.1×) 32.0 (13.9×) 36.4 (15.8×) 77.3 78.0 75.7 76.0 1024 0.7 (1×) 9.3 (13.3×) 13.0 (18.6×) 19.3 (27.6×) We conduct extensive ablation studies to understand how different components of Fast-dLLM contribute to performance, focusing on factors such as prefill length, generation length, cache mechanism variants, cache block size, and confidence thresholds. Influence of Prefill and Generation Length on Acceleration Table 4 and Table 5 indicate that both prefill length (n-shot) and generation length markedly impact overall speedup. Specifically, as the prefill length increases from 5-shot to 8-shot, the speedup obtained by both versions of KV Cache rises significantly (e.g., speedup for DualCache increases from 19.6× in 5-shot to 27.6× in 8-shot for generation length 1024). Similarly, extending the generation length amplifies the potential for cache reuse, leading to higher speedup. Notably, for 8-shot, speedup with DualCache grows from 9.4× (gen len 256) up to 27.6× (gen len 1024). This aligns with the theoretical expectation that amortizing computation over longer sequences yields more pronounced efficiency gains. Comparison of prefix KV Cache vs. DualCache We further compare our prefix KV Cache and DualCache versions in multiple settings. As shown in Table 5, DualCache generally achieves higher speedup than the prefix KV Cache, especially for longer generation lengths. For gen len 512 and 1024, DualCache demonstrates up to 27.6× speedup, 9 Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding outperforming the prefix KV Cache’s 18.6× in the same scenario. Importantly, DualCache maintains competitive accuracy, with only minor trade-offs relative to the cache-only variant. This highlights DualCache’s effectiveness in exploiting parallelism and cache locality for both efficiency and accuracy. Effect of Cache Block Size Figure 4 analyzes the influence of the cache block size hyperparameter. We observe that smaller block sizes tend to maximize accuracy but incur overhead due to frequent cache updates. In contrast, larger block sizes may diminish accuracy owing to increased context mismatch. Block size of 32 achieves the best trade-off, substantially improving throughput while largely preserving accuracy. This hyperparameter thus offers a practical knob for balancing latency and precision in real deployments. Dynamic Threshold vs. Fixed Token-per-Step Strategies We evaluate our Confidence-Aware Parallel Decoding method against fixed token-per-step baselines on GSM8K (Figure 5). Our adaptive strategy consistently outperforms fixed baselines across key metrics: it delivers higher accuracy at comparable or reduced number of function evaluations (NFE) and generates more tokens per step on average while closely tracking accuracy. In the rightmost panel, the dynamic method approaches or exceeds the accuracy of the 1-token (non-parallel) baseline, but with much greater throughput. The result demonstrates the effectiveness of Confidence-Aware Parallel Decoding, offering practical advantages. Factor Decoding vs. Fixed Token-per-Step Strategies We further compare our factor-based parallel decoding approach with fixed token-per-step baselines on GSM8K (Figure 8) and with the threshold-based strategy (Table 11). Across a range of factor values, our method consistently achieves competitive or higher accuracy with fewer inference steps. As the factor increases, the number of tokens decoded per step grows steadily, reducing iteration count while maintaining performance. Compared to the threshold strategy, factor decoding achieves similar accuracy but significantly higher throughput by adaptively controlling decoding granularity. We also analyze parallel token counts across decoding step at Appendix C.4. Decoding Efficiency Analysis and Limitations As discussed in Section C.5, PrefixCache significantly accelerates diffusion-based LLMs like LLaDA with up to 5× throughput improvement in compute-bound scenarios compared to ``` ### Table 6 | Qualitative comparison of responses across methods. ```text Table 6 | Qualitative comparison of responses across methods. Prompt: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? Original PrefixCache DualCache The robe takes 2 bolts of blue fiber. The robe takes 2 bolts of blue fiber. The robe takes 2 bolts of blue fiber. It also takes half that much white fiber, so It also takes half that much white fiber, so It also takes half that much white fiber, so it takes 2/2 = 1 bolt of white fiber. it takes 2/2 = 1 bolt of white fiber. it takes 2/2 = 1 bolt of white fiber. In total, the robe takes 2 + 1 = 3 bolts of In total, the robe takes 2 + 1 = 3 bolts of In total, it takes 2 bolts + 1 bolt = 3 bolts fiber. So, the value is 3 fiber. So, the value is 3 of fiber. The final result is 3 ``` ### Table 7 | Qualitative comparison of responses with varying block size for DualCache. ```text Table 7 | Qualitative comparison of responses with varying block size for DualCache. Prompt: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? Block Size 8 Block Size 16 Block Size 32 The robe takes 2 bolts of blue fiber. The robe takes 2 bolts of blue fiber. The robe takes 2 bolts of blue fiber. It also takes half that much white fiber, so It also takes half that much white fiber, so It also takes half that much white fiber, so it takes 2/2 = 1 bolt of white fiber. it takes 2/2 = 1 bolt of white fiber. it takes 2/2 = 1 bolt of white fiber. In total, the robe takes 2 + 1 = 3 bolts of In total, the robe takes 2 + 1 = 3 bolts of In total, the robe takes 2 + 1 = 3 bolts of fiber. So, the value is 3 fiber. So, the value is 3 fiber. So, the value is 3 ``` ### Table 8 | Qualitative comparison of responses under different threshold settings. ```text Table 8 | Qualitative comparison of responses under different threshold settings. Prompt: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? Threshold 0.7 Threshold 0.8 Threshold 0.9 The robe takes 2 bolts of blue fiber. The robe takes 2 bolts of blue fiber. The robe takes 2 bolts of blue fiber. It also takes half that much white fiber, so It also takes half that much white fiber, so It also takes half that much white fiber, so it takes 2/2 = 1 bolt of white fiber. it takes 2/2 = 1 bolt of white fiber. it takes 2/2 = 1 bolt of white fiber. In total, it takes takes 2 + 1 = 3 bolts of In total, the robe takes 2 + 1 = 3 bolts of In total, the robe takes 2 + 1 = 3 bolts of fiber. So, the value is 3 (NFE: 9) fiber. So, the value is 3 (NFE: 12) fiber. So, the value is 3 (NFE: 20) B.1. Effect of Caching Strategies on Response Quality ``` ### Table 6 qualitatively compares answers from the Original, PrefixCache, and DualCache methods for the arithmetic ```text Table 6 qualitatively compares answers from the Original, PrefixCache, and DualCache methods for the arithmetic prompt. All correctly compute the answer (3 bolts), following similar step-by-step reasoning, with only minor differences in phrasing. This shows cache strategies maintain answer accuracy and logical clarity while improving efficiency; semantic fidelity and interpretability are unaffected. B.2. Effect of Block Size in DualCache ``` ### Table 7 examines different block sizes (8, 16, 32) in DualCache. For this arithmetic prompt, all settings yield correct, ```text Table 7 examines different block sizes (8, 16, 32) in DualCache. For this arithmetic prompt, all settings yield correct, clearly explained answers with no meaningful output differences. Thus, DualCache is robust to block size for such problems, allowing efficiency improvements without compromising quality. B.3. Impact of Dynamic Threshold Settings ``` ### Table 8 investigates dynamic threshold values (0.7, 0.8, 0.9). The model consistently produces the correct answer ```text Table 8 investigates dynamic threshold values (0.7, 0.8, 0.9). The model consistently produces the correct answer and clear explanations, regardless of threshold. While higher thresholds increase computational effort (NFE from 9 to 20), answer quality remains stable, indicating threshold adjustment mainly affects efficiency, not correctness, for straightforward arithmetic questions. 15 Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding B.4. Multimodal Generation with LLAda-V To qualitatively analyze the effectiveness of our Fast-dLLM framework in multimodal scenarios, we conduct a visual case study where the model is tasked with generating a highly detailed image description. As illustrated in Figure 6, both the baseline model and our Fast-dLLM are given the same visual input and user prompt: “Please describe the image in detail.” User: Please describe the image in detail. Baseline (63.0 secs): The image captures a serene rural landscape. Dominating the scene is a wooden barn, its weathered exterior a testament to the passage of time. The barn is nestled in a field of green grass, which stretches out to meet the horizon. In the distance, a range of mountains rises, their peaks shrouded in a blanket of clouds. The sky above is a soft pink, suggesting either dawn or dusk. The perspective of the image is from a low angle, making the barn appear grand and imposing. The overall composition of the image creates a sense of depth and tranquility, inviting the viewer to imagine themselves in this peaceful rural setting. Fast-dLLM (6.8 secs): The image captures a serene rural landscape. Dominating the scene is a wooden barn, its rustic charm enhanced by the gray shingles on its roof. The barn is nestled in a field of tall grass, which stretches out towards the horizon. The field is framed by a range of mountains, their peaks dusted with snow, adding a touch of tranquility to the scene. The sky above is a soft pink, suggesting either early morning or late afternoon. The perspective of the image is from a low angle, making the barn appear grand and imposing. The overall composition of the image creates a harmonious blend of man-made structures and the natural world. Figure 6 | Comparison between the baseline and Fast-dLLM on a visual description task. Fast-dLLM produces a comparable and faithful image caption in a fraction of the decoding time. The baseline model requires 63.0 seconds to complete the generation, producing a detailed and poetic description of the rural landscape. It highlights elements such as the weathered wooden barn, the soft pink sky, and the tranquil atmosphere. In contrast, our Fast-dLLM completes the task in just 6.8 seconds—a nearly 10× speedup—while maintaining rich visual detail. It further enhances the description with additional grounding (e.g., “gray shingles on its roof”, “touch of tranquility”), reflecting a strong alignment with both appearance and mood cues from the image. Notably, the generated caption retains compositional depth and stylistic fluency, illustrating the model’s ability to balance fluency and factuality even under diffusion-based parallel decoding. This case highlights how LLAda-V with Fast-dLLM decoding enables high-quality vision-language generation at significantly improved efficiency, paving the way for faster and more interactive multimodal applications. C. Experiment Details C.1. Further Experiments with LLaDA-V ``` ### Table 9 | Effect of block length on performance (MathVista, 48 Steps) ```text Table 9 | Effect of block length on performance (MathVista, 48 Steps) Block Length 4 8 16 32 96 Accuracy (%) 51.2 50.7 51.8 52.3 59.7 Throughput (tok./s) 6.1 6.2 5.5 5.5 5.6 16 Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding ``` ### Table 10 | MathVista Performance with Fast-dLLM at different refresh intervals (block length = 96) ```text Table 10 | MathVista Performance with Fast-dLLM at different refresh intervals (block length = 96) Refresh Interval 2 4 8 16 32 Accuracy (%) 59.2 59.2 58.2 57.1 56.6 Throughput (tok./s) 15.9 19.5 21.1 25.2 28.2 In Table 9, we investigate how the choice of block length affects the performance of LLaDA-V on MathVista under a fixed decoding length of 48 steps. The results show that the model achieves the highest accuracy with a block length of 96. However, when reducing the block size to 8 or 4, the accuracy drops significantly by over 8%. Given this sensitivity to block length, we choose not to break the output into small blocks for updating caches individually. Instead, we keep the block length fixed at 96 and adopt a refresh-based strategy: the cache is updated only every 𝑟 decoding steps using the most recent full block. As shown in Table 10, increasing the refresh interval leads to consistent gains in throughput—from 15.9 tokens/s at interval 2 to 28.2 tokens/s at interval 32. While accuracy drops slightly with larger intervals, it remains above 56.6%, suggesting that aggressive refresh scheduling can yield substantial speedups with only minor performance degradation. C.2. Performance Comparison between Threshold and Factor Strategy ``` ### Table 11 | Performance comparison between Threshold and Factor confidence-aware decoding on GSM8K and MATH ```text Table 11 | Performance comparison between Threshold and Factor confidence-aware decoding on GSM8K and MATH benchmarks with generation lengths of 256 and 512. Each block shows accuracy (top row) and throughput with speedup (bottom row). Factor decoding provides favorable trade-offs in most settings. Benchmark Gen. Len Threshold Factor 256 78.5 77.5 GSM8K (5-shot) 54.4 (8.1×) 78.5 (11.7x) 512 77.2 74.8 35.3 (11.0×) 47.1 (14.7x) 256 33.2 32.0 MATH (4-shot) 51.7 (5.7×) 78.3 (8.6x) 512 36.0 35.2 47.1 (5.9×) 64.6 (8.1x) We compare the performance of our threshold-based and factor-based confidence-aware parallel decoding strategies on GSM8K and MATH benchmarks (Table 11). While the threshold strategy achieves marginally better accuracy in most settings (e.g., 78.5% vs. 77.5% on GSM8K with 256 tokens), the factor strategy demonstrates substantially superior throughput performance. Specifically, factor decoding achieves 1.4-1.5× higher throughput than threshold decoding across all settings. On GSM8K with 256 tokens, factor decoding reaches 78.5 tokens/sec (11.7× speedup) compared to 54.4 tokens/sec (8.1× speedup) for threshold decoding. This throughput advantage becomes even more pronounced on longer generation tasks—for GSM8K with 512 tokens, factor decoding attains 47.1 tokens/sec while threshold only achieves 35.3 tokens/sec. The results demonstrate that factor decoding offers a compelling trade-off: it sacrifices minimal accuracy (typically 1-3%) in exchange for significant throughput improvements (40-50% higher). This makes factor decoding particularly attractive for latency-sensitive applications where the slight accuracy reduction is acceptable. The consistent pattern across both benchmarks and generation lengths validates the robustness of the factor strategy’s theoretical foundation, which adaptively controls parallelism based on the confidence bound (𝑛 + 1)𝜖 < 𝑓 . 17 Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding Decoding Steps vs Parallel Token Count 95% Confidence Interval 12 Avg Parallel Token Count 10 Average Parallel Token Count 8 6 4 2 0 0 20 40 60 80 100 Step Index Figure 7 | Average number of tokens generated at each decoding step. Blue line shows the mean token count, and the shaded area denotes the 95% confidence interval. C.3. Comparison between LLaDA and LLaDA-1.5 We compare the performance of LLaDA and its enhanced version LLaDA-1.5 across both GSM8K (5-shot) and MATH (4-shot) benchmarks under two generation length settings (256 and 512 tokens), as shown in Table 12. Each cell reports accuracy and decoding throughput (in tokens per second), along with the relative speedup over the greedy baseline. Across GSM8K settings, LLaDA-1.5 consistently improves accuracy over the original LLaDA, achieving a notable +2.2% absolute gain at 256-token generation and +3.2% at 512-token generation. Furthermore, it maintains strong decoding efficiency, with throughput reaching 59.4 tokens/sec at 256 tokens, improving upon LLaDA’s 54.1 tokens/sec under the same setting. On the MATH benchmark, accuracy between the two versions remains comparable. However, LLaDA-1.5 slightly ``` ### Table 12 | Performance comparison between LLaDA and LLaDA-1.5. Each cell presents the accuracy and the ```text Table 12 | Performance comparison between LLaDA and LLaDA-1.5. Each cell presents the accuracy and the decoding throughput in tokens per second with relative speedup to the LLaDA baseline (bottom row, blue: tokens per second/orange: relative speedup). Benchmark Gen Length LLaDA (Fast-dLLM) LLaDA 1.5 (Fast-dLLM) 256 78.5 80.7 GSM8K (5-shot) 54.1 (8.1×) 59.4 (8.9×) 512 77.2 80.4 35.3 (11.0×) 33.0 (10.3×) 256 33.2 32.6 MATH (4-shot) 51.7 (5.7×) 53.7 (5.9×) 512 36.0 35.1 47.1 (5.9×) 41.1 (5.1×) 85 3.79 Selected 250 80 4.92 5.68 GSM8K (5-shot) Accuracy GSM8K (5-shot) Accuracy 80 3.79 4.41 4.41 4.92 Ours 2 5.26 75 Selected 5.26 5.68 200 2 token/step 70 Inference Steps 4 token/step 4 70 Non-Parallel Baseline (1 token/step) 60 150 65 50 100 60 Ours 3.79 4.41 2 token/step 4.92 5.26 5.68 40 Ours 55 50 Selected 4 token/step Fixed-Step Baseline (2/4 tokens) Non-Parallel Baseline (1 token/step) 30 Non-Parallel Baseline (1 token/step) 50 0 0.7 1.0 1.3 1.6 1.9 0.7 1.0 1.3 1.6 1.9 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 Factor Factor Average #Tokens per Step (a) (b) (c) Figure 8 | (a) GSM8K (5-shot) accuracy across different factor values using our factor-based decoding strategy. Numbers above each point indicate the average number of tokens decoded per step. The dashed lines show the accuracy of the baseline method with 2 or 4 tokens per step, and the non-parallel (1 token/step) baseline. (b) The corresponding number of inference steps needed under each factor setting. Our method generally requires significantly fewer steps than fixed-step baselines. (c) Accuracy versus average number of tokens decoded per step on GSM8K (5-shot). Our factor-based decoding achieves better accuracy-efficiency trade-offs compared to baselines. The red “Selected” point represents the setting chosen in our main results. tokens required to complete the output, and the number of remaining tokens could differ widely among different samples (e.g., due to early completion or padding). These observations are important for understanding how decoding efficiency can be optimized: increasing parallelism during high-confidence phases (middle steps) offers computational savings, while conservative behavior near boundaries maintains quality. C.5. Throughput Comparison under Varying Batch Sizes All experiments are conducted on an NVIDIA A100 GPU, with the prefill length fixed to 256 tokens. The generation length is varied among 16, 32, and 64 tokens, and batch sizes range from 1 to 32. This setup reflects realistic deployment scenarios, allowing the evaluation of decoding efficiency under diverse conditions. It should be noted that parallel decoding allows multiple tokens to be generated simultaneously affected by dummy input tokens. To ensure fairness, we focus solely on the acceleration provided by caching techniques. PrefixCache is designed as an acceleration mechanism for LLaDA, a diffusion-based LLM, and successfully boosts the throughput significantly. Figure 9 shows that PrefixCache achieves consistent improvements across all batch sizes and generation lengths, making it particularly suited for scenarios with smaller generation lengths and larger batch sizes. For instance, with a generation length of 16 and batch size of 32, PrefixCache achieves a throughput of over 211 tokens/s, significantly outperforming the native LLaDA which reaches only 43 tokens/s, demonstrating nearly 5× improvement. While LLaDA exhibits limited scalability with increasing batch sizes—its throughput plateaus after batch size 8—this 19 ``` ## reference/nemotron/Nemotron-Labs-Diffusion-Image- Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis ### Table 1 | Text-to-Image Generation Performance on Geneval Benchmark. ```text Table 1 | Text-to-Image Generation Performance on Geneval Benchmark. Model Params Single Obj.↑ Two Obj.↑ Counting↑ Colors↑ Position↑ Color Attri.↑ Overall↑ Unified MLLM Emu3 [9] 8B - - - - - - 0.66 Janus-Pro [32] 7B 0.99 0.89 0.59 0.90 0.79 0.66 0.80 MMaDA [12] 8B 0.99 0.76 0.61 0.84 0.20 0.37 0.63 Show-o [18] 1.3B 0.98 0.80 0.66 0.84 0.31 0.50 0.68 BAGEL [38] 14B 0.98 0.95 0.84 0.95 0.78 0.77 0.88 LaViDa-O [10] 10B 0.99 0.85 0.71 0.86 0.65 0.58 0.77 Show-o2 [47] 7B 1.00 0.87 0.58 0.92 0.52 0.62 0.76 Gen. Only PixArt-𝛼 [48] 0.6B 0.98 0.50 0.44 0.80 0.08 0.07 0.48 DALL-E 3 [46] - 0.96 0.87 0.47 0.83 0.43 0.45 0.67 SD3-Medium [45] 2B 0.99 0.94 0.72 0.89 0.33 0.60 0.74 FLUX.1-dev [2] 12B 0.98 0.81 0.74 0.79 0.22 0.45 0.66 Meissonic [7] 1B 0.99 0.66 0.42 0.86 0.10 0.22 0.54 Qwen-Image-2507[3] 20B 0.99 0.92 0.89 0.88 0.76 0.77 0.87 Nemotron-Labs-Diffusion-Image 8B 0.98 0.93 0.83 0.94 0.88 0.82 0.90 ``` ### Table 2 | Text-to-Image Generation Performance on DPG Benchmark and MJHQ-30k Dataset. ```text Table 2 | Text-to-Image Generation Performance on DPG Benchmark and MJHQ-30k Dataset. *Finetuned on 6M synthetic data for better image quality. MJHQ-30k Model Params Codebook DPG↑ FID↓ HPSv3↑ SD3[45] 8B - 83.5 11.92 9.42 GPT-4o [45] - - 85.3 - - Flux-Dev[2] 12B - - 10.15 - Janus-Pro[32] 7B 16,384 84.1 10.10 8.81 Emu3 [9] 8B 32,678 80.6 - - Show-o [18] 1B 8,192 - 15.18 7.20 MMaDa[12] 8B 8,192 53.4 32.85 5.43 LaViDa-O [10] 10B 8,192 81.8 6.68 8.81 Nemotron-Labs-Diffusion-Image 8B 131,072 85.2 6.46 9.61 Nemotron-Labs-Diffusion-Image* 8B 131,072 86.9 12.23 10.76 4.2. Ablations for Token Editing and Self-Correction To validate the effectiveness of our proposed token editing pipeline and investigate whether self-correction improves image fidelity, we conduct both qualitative and quantitative evaluations on the MJHQ dataset. In Figure 5, we fix the random seed and visually compare images generated with and without the token editing pipeline. Token editing consistently improves image fidelity by correcting artifacts and refining texture details in the generated images. In Figure 6b, we report HPSv3 scores on the MJHQ dataset with and without token editing. We evaluate Nemotron-Labs-Diffusion-Image under different numbers of sampling steps, also known as the Number of Function Evaluations (NFEs). We draw two main conclusions from these results. First, token editing consistently improves image quality across all NFEs. Second, although reducing NFEs decreases image quality in both settings, the degradation is substantially smoother when token editing is enabled. Notably, generations with token editing at 32 NFEs achieve performance comparable to generations without token editing at 64 NFEs, representing an effective 2× reduction in forward calls given a fixed target quality. 4.3. Ablations for the Grouped Cross-Entropy Objective To fairly evaluate the effectiveness of our proposed GCE objective and demonstrate its advantages over alternatives such as SNCE, we conduct extensive ablation studies in a controlled setting. Following the setup of SNCE [22], we perform experiments on class-conditional image generation using the ImageNet [49] dataset at 256 × 256 resolution. We use the exact same model architecture, tokenizer, and training schedule as SNCE [22], with the optimization objective being the only varying factor. Results are reported in Table 3. GCE consistently outperforms both SNCE and the vanilla cross-entropy baseline, demonstrating its effectiveness and advantages. 8 Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis w/o Edit w/ Edit w/o Edit w/ Edit Figure 5 | Qualitative comparison of generated samples with and without token editing. ``` ### Table 3 | Class-conditioned Image Synthesis on ImageNet256 dataset. *Models have identical-sized ```text Table 3 | Class-conditioned Image Synthesis on ImageNet256 dataset. *Models have identical-sized transformer layers. Parameter count increased due to larger token embedding and final linear head. Objective Params Tokenizer Tokenizer Pretraining Codebook Epoch FID ↓ CE 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 100 7.53 SNCE 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 100 3.62 GCE 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 100 3.40 CE 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 300 5.44 SNCE 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 300 3.42 GCE 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 300 3.00 CE 846M* FVQ [19] ImageNet256 262,144 300 4.11 SNCE 846M* FVQ [19] ImageNet256 262,144 300 3.20 GCE 846M* FVQ [19] ImageNet256 262,144 300 2.69 ``` ### Table 4 | Performance Comparison of Loss Function Implementations. ```text Table 4 | Performance Comparison of Loss Function Implementations. Operation Latency ↓ Input VRAM↓ Active VRAM↓ Max VRAM↓ F.cross_entropy (hard label) 12.71 ms 8.1 GB 8.1 GB 16.1 GB F.cross_entropy (soft label) 25.00 ms 16.1 GB 8.1 GB 24.2 GB GCE (eager) 44.14 ms 8.1 GB 17.1 GB 25.2 GB GCE (optimized forward) 17.86 ms 8.1 GB 16.1 GB 24.2 GB GCE (optimized fwd & bwd) 20.04 ms 8.1 GB 8.1 GB 16.1 GB 4.4. Compute Efficiency Optimized Operator. To validate the effectiveness of our optimized operator, we benchmark both VRAM usage and latency when processing 16,384 tokens on a H100 GPU, corresponding to either 4 images at 1024 × 1024 resolution or 64 images at 256 × 256 resolution. We report latency, input tensor VRAM usage, and maximum VRAM consumption during both forward and backward computation. We define active VRAM as the difference between maximum VRAM and input VRAM, which measures the additional memory overhead introduced by loss computation. For fairness, we enable torch.compile in all experiments. We report results 9 Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis 0.90 42.4x Faster w/o edit 9.60 w/ edit 0.88 0.86 9.55 GenEval Score HPSv3 0.84 9.50 0.82 Nemotron-Labs-Diffusion-Image (MDM) 9.45 Infinity-8B (VAR) 0.80 Qwen-Image (Flow-Matching) Emu3.5 (AR) 9.40 2.5 5 10 20 40 80 160 320 24 32 48 64 Latency (s) / Image @ 1024x1024 NFE (a) Comparision of GenEval score and inference latency between (b) HPSv3 scores under different num- Nemotron-Labs-Diffusion-Image and other state-of-the-art models. bers of sampling steps with and without Results are measured with a batch size of 1 on a H100 GPU. token editing. Figure 6 | Speed-Quality Tradeoff of Nemotron-Labs-Diffusion-Image. 1 step 2 steps 3 steps 4 steps 5 steps Ours (8B) Qwen-Image-2512 (28.5B) NaN Error Figure 7 | Few-Step Generation Results. We visualize the sampled images with 1,2,3,4,5 total steps and compare with a continuous model Qwen-Image [3]. in Table 4 Compared with the eager implementation, our optimized operator reduces latency from 44 ms to 20 ms while decreasing maximum VRAM usage from 25 GB to 16 GB. Compared with the standard cross-entropy baseline using one-hot labels, our method introduces no additional memory overhead and only an 8 ms increase in latency. We note that this overhead is negligible given that the overall training time is approximately 3.2 s per step. All measurements are conducted on H100 GPUs. Generation Latency. We also compare the inference latency of Nemotron-Labs-Diffusion-Image against ``` ### Table 5 | Training configurations across two stages. ```text Table 5 | Training configurations across two stages. Stage 1 Stage 2 Learning Rate 1 × 10−4 1 × 10−5 Steps 200k 100k 𝛽1 0.99 0.99 𝛽2 0.999 0.999 optimizer AdamW AdamW Learning Rate Schedule Cosine Cosine Final Learning Rate 1 × 10−5 1 × 10−6 Model Size 8B 8B Image Resolution 256 512 → 1024 Global Batch Size 1,024 256 Token Editing Disabled Enabled B. Additional Experiment Details and Results In this section, we provide additional training details and experimental results, including qualitative samples and ablation studies. B.1. Training Data Our training dataset consists of 137M text-image pairs sourced from public datasets. The data pipeline largely follows the prior work LaViDa-O [10]. Specifically, we source raw images from LAION-2B [54], COYO-700M [55], BLIP3o-60k [43], and ShareGPT4o-Image [56]. These datasets are heavily filtered to remove NSFW prompts, low CLIP-score samples [57], low aesthetic-score samples [58], and low-resolution images. For all images from LAION-2B and COYO-700M, we use Qwen3-VL [59] to re-caption the images instead of relying on the original alt-text annotations, which are often noisy. However, we retain raw captions with high CLIP scores and randomly choose between VLM-generated captions and raw captions for these samples. We make this choice primarily to support keyword-based prompting such as “high quality” and “4k” during inference, since such keywords do not naturally emerge in VLM-generated captions. B.2. Training Setup and Hyperparameters We adopt the Emu-3.5 tokenizer [17], which has a vocabulary size of 131,072. We initialize Nemotron-Labs- Diffusion-Image from a pretrained diffusion language model [39]. Training consists of two stages. In the first stage, we pretrain the model on 256 × 256 images for 200k steps with a global batch size of 1024. In the second stage, we scale to 512 × 512 resolution for 20k steps and then to 1024 × 1024 resolution for 80k additional steps with a global batch size of 256. Training is conducted on 64 H100 GPUs. Additional details are provided in Table 5. B.3. Ablation Studies of Editing Thresholds We study the effect of varying the editing threshold 𝜏 under different numbers of inference steps. We visualize the resulting HPSv3 scores in Figure 9. Overall, enabling token editing consistently outperforms the no-editing baseline. Among all evaluated settings, 𝜏 = 0.6 achieves the best image quality for most NFEs. B.4. Ablation Studies of Cluster Sizes We investigate the effect of varying the number of clusters used in the GCE objective and report FID scores on ImageNet-256 in Table 6. We find that combining both 16,384-cluster and 8,192-cluster supervision performs better than using either clustering level alone. When using only a single clustering level, the 16,384-cluster 18 Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis w/o edit 9.6 =0.6 =0.4 9.5 =0.8 9.4 HPSv3 9.3 9.2 9.1 16 24 32 48 64 NFE Figure 9 | Effect of different token-editing thresholds on HPSv3 scores. ``` ### Table 6 | Ablation Experiments on Cluster size. *Model have identical-sized transformer layer. Parameter ```text Table 6 | Ablation Experiments on Cluster size. *Model have identical-sized transformer layer. Parameter count increased due to larger token embedding and final linear head. Cluster Sizes Params Tokenizer Tokenizer Pretraining Codebook Epoch FID ↓ 8,192 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 100 3.67 16,384 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 100 3.44 Both 577M* Emu3.5-IBQ [17] Large-Scale T2I 131,072 100 3.40 setting performs better, presumably because the 8,192-cluster setting is coarser and provides less refined supervision signals. B.5. Ablation Studies of Corruption Type and Scale We explore different corruption strategies for the token-editing objective by evaluating HPSv3 scores after 10K steps 1024 resolution training in Stage-2. Specifically, we evaluate random noise corruption, corruption using neighboring tokens in embedding space, and resampling tokens from the same input image. Results are reported in Table 8. Using a combination of neighboring tokens and resampled tokens achieves the best performance. Additionally, we experiment with different corruption ratios 𝛼 and observe no significant differences in image quality when 𝛼 lies within a reasonable range. However, image quality degrades when 𝛼 becomes too large. This is because high corruption levels (e.g., 𝛼 = 0.5) make it substantially more difficult to distinguish clean tokens and corrupted tokens due to the lower signal-to-noise ratio, increasing optimization difficulty. In our final experiments, we use 𝛼 = 0.1. ``` ### Table 7 | Noise Type Comparison Table 8 | Threshold 𝛼 Impact ```text Table 7 | Noise Type Comparison Table 8 | Threshold 𝛼 Impact Noise Type HPSv3 (10k step) 𝛼 Value HPSv3 (10k step) Random 8.53 0.1 8.99 Adjacent Tokens 8.81 0.3 8.97 Adj. Tokens + Resamp. 8.99 0.5 7.52 B.6. Additional Qualitative Results In this section, we provide additional qualitative samples to further demonstrate the effectiveness of Nemotron- Labs-Diffusion-Image. Figure 10 presents additional text-to-image generation results. Figures 11 and 12 19 Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis compare generations produced with and without token editing under the same random seed. We observe that token editing consistently improves image fidelity by refining details and correcting artifacts. C. Compute Resources We train the model on 64 H100 GPUs across 8 nodes. Training takes approximately 16 days in total. D. Limitations Despite the effectiveness of Nemotron-Labs-Diffusion-Image, several limitations remain. First, although we demonstrate that the token-editing mechanism improves image quality, it does not eliminate all artifacts, and the model may still generate erroneous outputs. Second, although we achieve substantial performance gains by optimizing the GCE objective with a carefully designed fused operator, additional improvements may still be possible through customized low-level CUDA kernels. We leave this direction for future work. E. Broader Impact Nemotron-Labs-Diffusion-Image has strong text-to-image generation capabilities, which may be misused to generate harmful or offensive content. We strongly caution against such use cases. Additionally, our model may inherit biases present in the base language model as well as biases contained in the training data. Our model is intended primarily for research purposes to facilitate future exploration of foundational discrete image generators. We do not recommend its use for other purposes. F. Licenses We make use of the following assets: Models: Emu-3.5-Tokenizer [17] (Apache-2.0), Qwen3-VL [59] (Apache-2.0), Nemotron-Labs-Diffusion [39] (Nvidia Open Model License) Datasets: LAION [54] (MIT), COYO [55] (CC-BY-4.0), MJHQ [25] (CC-BY-4.0), BLIP3o-60k [43] (Apache-2.0), and ShareGPT4o-Image [56] (CC-BY-4.0). 20 Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis a beautiful sunset, bright and fluffy white bunny as a Jedi, blue Panna Cotta with Mixed Berry underground fantasy city with colourful, ultra realistic, UHD, 8k lightsaber, fierce expression Compote in a country kitchen, magic warm light landscape Minecraft artstyle, Medieval fantasy landscape landscape of aurora borealis DD art, Tall thin, male human photo contest, high contrast, magical wizard sunset vertigo sky and snowy ground, bright sorcerer, handsome face, gold bright and clear photo, sharp, flux of light golden light colors, realistic, wide shot from and green magic sparkling detailed, delicate fantasy landscape beautiful far away, breathtaking around him photorealistic calm illustration sunset composition hecate walking in a temple closeup portrait, in the style of pirate character portrait gray hair dental assistant, female, in her forest, anime, magical realism, hand painted anime by ilya weathered face thick beard 20s, 4k, realistic young female in hd, photorealistic, anime kuvshinov, vibrant colors, weathered face colorful her 20s, dental assistant, realistic attention to detail, highlydetailed headband patched clothing and photograph ``` ## reference/nemotron/PRESTO- Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding ### Table 1. Accept length comparison between BFS and Beam Search. ```text Table 1. Accept length comparison between BFS and Beam Search. Qwen3-4B-DFlash Qwen3-8B-DFlash 5. Empirical Validation Budget BFS Beam BFS Beam We show that PRESTO significantly improves the decoding B = 128 9.21 9.27 9.25 9.28 B = 256 9.55 9.54 9.60 9.62 efficiency of both standard and SS diffusion SDs. B = 512 9.81 9.79 9.90 9.88 B = 1024 9.98 9.85 10.10 9.93 5.1. Experiment Setup BFS vs. Beam Search. Table 1 reports acceptance length Models, Tasks, and Metrics. We evaluate PRESTO under the two policies across budget sizes. At small to mod- on: (i) standard SD with a separate diffusion drafter and erate budgets (B ≤ 512), best-first and beam search yield AR target: dFlash with Qwen3-4B/8B and Qwen3-Coder- similar acceptance length, as beam search’s per-depth bud- 30B-A3B; and (ii) SS dLLM: NLD-8B at both linear and get suffices to cover the shallow trees that best-first would quadratic SS decoding modes. We test three task categories: expand under global priority. At B = 1024, best-first out- math (GSM8K (Cobbe et al., 2021), MATH (Lightman performs beam search by a small margin: with a larger bud- et al., 2023), AIME24 (Art of Problem Solving, 2024), get, the optimal expansion pattern can become less uniform AIME25 (MAA)), code (HumanEval (Chen et al., 2021), across depth, where a global priority handles more flexibly MBPP (Austin et al., 2021), LiveCodeBench (Jain et al., than a fixed per-depth beam. In practice, verification shifts 2024)), and conversation (MT-Bench (Zheng et al., 2023), from memory-bound to compute-bound once B exceeds Alpaca (Taori et al., 2023)). We report average acceptance a hardware-dependent threshold, after which forward-pass length (τ ) and throughput (token/s) speedup over the AR cost grows linearly with B and erodes the speedup. We baseline on NVIDIA B200 GPUs. therefore operate at B ≤ 512, where the two policies are Implementation. For dFlash and NLD under quadratic SS, comparable, and adopt beam search as the default. we use a PyTorch implementation with FlexAttention (Dong et al., 2024); under linear SS, we integrate PRESTO into 4.3. Extending Tree Construction to hybrid dLLMs SGLang with FlashInfer (Ye et al., 2025b). The prefix- The above tree-construction algorithm can be naturally aligned signal is an n-gram model (n = 3)3 , with λd = 0.2, extended to support quadratic SS decoding in hybrid batch size 1, and beam search with global retention (width dLLMs (Liu et al., 2025; Fu et al., 2026a)2 , as shown in 10, top-k = 10 per position). We evaluate PRESTO under Figure 3. Similar to the literature (Liu et al., 2025), we par- deterministic (T = 0) and stochastic (T = 1) decoding, allelize the autoregressive-based verification and diffusion- sweeping tree budgets T ∈ {128, 256, 512} for dFlash (re- porting the best end-to-end throughput) and fixing T = 32 2 NLD primarily highlights a linear SS decoding mode. Apply- for Nemotron-Labs-Diffusion per its speculative block size. ing PRESTO to this setting is identical to the dedicated diffusion drafter setting discussed earlier. Note that the NLD paper also 3 Following the n-gram construction in (Liu et al., 2026). introduces a quadratic SS variant which is explored in this work. 5 PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding ``` ### Table 2. Decoding speedup and average acceptance length (τ ) on Qwen3 models with thinking mode disabled and a maximum of 2048 ```text Table 2. Decoding speedup and average acceptance length (τ ) on Qwen3 models with thinking mode disabled and a maximum of 2048 generated tokens. M ATH C ODE C HAT Avg. Model Method GSM8K MATH-500 AIME24 AIME25 HumanEval MBPP LCB SWE-Bench MT-Bench Alpaca Avg. Temperature = 0 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ dFlash 5.1× 6.5 6.5× 7.9 6.5× 7.4 6.2× 7.4 5.6× 6.7 5.0× 6.1 5.9× 6.9 3.2× 3.6 2.9× 4.4 2.4× 3.1 4.9× 6.0 4B PRESTO 7.9× 9.4 9.2× 10.9 8.7× 10.5 8.8× 10.3 8.1× 9.9 7.9× 9.3 8.4× 10.0 5.3× 5.9 4.9× 6.7 3.9× 5.0 7.3× 8.8 × Gain 1.55× 1.42× 1.34× 1.42× 1.45× 1.58× 1.42× 1.66× 1.69× 1.62× 1.48× dFlash 5.0× 6.6 6.3× 7.9 6.2× 7.5 6.0× 7.0 5.4× 6.5 4.8× 6.0 5.7× 7.1 3.3× 3.6 2.8× 4.3 2.4× 3.1 4.8× 6.0 8B PRESTO 8.0× 9.6 9.4× 11.1 8.5× 10.7 8.7× 10.2 8.5× 9.9 7.7× 9.4 8.5× 10.3 5.0× 6.0 4.8× 6.6 3.9× 5.0 7.3× 8.9 × Gain 1.60× 1.49× 1.37× 1.45× 1.57× 1.60× 1.49× 1.52× 1.71× 1.62× 1.52× dFlash 3.1× 5.2 4.1× 5.6 2.8× 5.3 2.7× 5.1 5.6× 8.0 5.6× 7.2 3.6× 6.2 3.2× 3.6 2.2× 3.5 1.8× 2.2 3.5× 5.2 30B PRESTO 4.1× 7.9 5.4× 8.0 3.6× 8.0 3.6× 7.9 7.0× 10.9 7.5× 9.9 4.7× 8.5 4.6× 5.6 3.0× 5.2 2.5× 3.4 4.6× 7.5 × Gain 1.32× 1.32× 1.29× 1.33× 1.25× 1.34× 1.31× 1.44× 1.36× 1.39× 1.33× ``` ### Table 3. Decoding speedup and average τ on NLD-8B at both linear and quadratic SS decoding mode and a maximum of 2048 generated ```text Table 3. Decoding speedup and average τ on NLD-8B at both linear and quadratic SS decoding mode and a maximum of 2048 generated tokens. M ATH C ODE C HAT Avg. Decoding Mode Method GSM8K MATH-500 AIME24 AIME25 HumanEval MBPP LCB SWE-Bench MT-Bench Alpaca Avg. Temperature = 0 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ vanilla 5.7× 10.8 6.4× 12.2 5.7× 11.0 5.5× 10.5 5.0× 9.3 4.4× 8.2 4.7× 9.0 3.7× 6.8 2.8× 5.2 2.5× 4.6 4.6× 8.7 Linear PRESTO 6.0× 11.9 6.7× 13.2 6.1× 12.2 5.8× 11.5 5.2× 10.3 4.6× 9.1 4.9× 10.0 4.0× 7.9 3.2× 6.5 2.9× 5.8 4.9× 9.9 × Gain 1.05× 1.04× 1.07× 1.06× 1.04× 1.04× 1.04× 1.08× 1.14× 1.16× 1.07× vanilla 2.6× 4.7 2.7× 6.8 3.2× 5.3 3.3× 5.7 3.2× 4.8 3.1× 4.5 2.4× 4.7 1.8× 3.5 2.1× 3.0 1.8× 2.7 2.6× 4.7 Quadratic PRESTO 5.3× 6.5 4.4× 7.5 3.5× 6.1 3.7× 6.4 3.7× 5.8 3.7× 5.4 3.1× 5.6 2.7× 4.7 2.8× 4.2 2.3× 3.6 3.5× 5.6 × Gain 2.04× 1.63× 1.09× 1.12× 1.16× 1.19× 1.29× 1.50× 1.33× 1.28× 1.35× 5.2. Experiment Results ing our full scoring function against using only the diffusion marginal qd . As shown in Figure 4 (top) in Appendix F, Main Results. As shown in Tables 2 and 3, PRESTO consis- incorporating ρd consistently improves average acceptance tently improves average acceptance length τ and throughput length across tasks, confirming its effectiveness. across both dFlash and Nemotron-Labs-Diffusion (NLD), across all models and tasks. On Qwen3-4B-DFlash with Sensitivity to λd . As shown in Figure 4 (middle) in Ap- T = 0, τ rises from 7.9 to 10.9 on MATH-500, 7.4 to 10.3 pendix F, moderate λd achieves the best acceptance length on AIME25, 6.9 to 10.0 on LCB, and 6.7 to 9.9 on Hu- across datasets and tree budgets, with λd ≈ 0.2 yielding manEval, with the largest gains (+2 to +3 tokens) on math the most stable gains. Small values underutilize the prefix- and code; stronger drafters (Qwen3-8B) yield comparable conditioned signal, while overly large values suppress qd relative gains, indicating that PRESTO pushes dFlash’s ac- and substantially degrade performance. ceptance length closer to its upper bound. The same trends More detailed discussions and ablations including tree oper- hold on NLD. Under linear SS decoding, PRESTO improves ations overhead and distribution of acceptance length are in average speedup from 4.6× to 4.9× and average acceptance Appendix F. length from 8.7 to 9.9, with consistent gains across all bench- marks. The improvements are also large under quadratic SS decoding, where average speedup increases from 2.6× 6. Conclusion to 3.5× and average acceptance length from 4.7 to 5.6. Re- We propose PRESTO, a tree-based SD framework for dif- sults under stochastic decoding (T = 1), where the gains fusion LLMs. We identify that diffusion probabilities are become substantially larger due to PRESTO ’s ability to inherently prefix-blind, causing unreliable path ranking un- preserve multiple plausible continuations under uncertainty, der prefix-based AR verification. PRESTO addresses this are reported in Appendix E. via prefix-aligned scoring and priority-based tree search, en- abling effective exploration of high-quality candidate paths. 5.3. Discussions and Ablations Extending PRESTO to production serving frameworks such Effectiveness of prefix-conditioned signal. We ablate as vLLM, together with characterizing its behavior under the prefix-conditioned compatibility term ρd by compar- larger batch sizes remains meaningful future work. 6 PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding Impact Statement D., Plappert, M., Chantzis, F., Barnes, E., Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, This work aims to improve the efficiency and accessibility of N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saun- large language model inference by accelerating speculative ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, decoding through tree-based diffusion drafting. By increas- J., Misra, V., Morikawa, E., Radford, A., Knight, M., ing throughput while preserving generation quality loss- Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- lessly, PRESTO reduces the computational cost and latency Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and of serving high-capacity language models, potentially broad- Zaremba, W. Evaluating large language models trained ening access to advanced AI capabilities for researchers and on code, 2021. URL https://arxiv.org/abs/ practitioners with limited hardware resources. 2107.03374. References Chen, Z., May, A., Svirschevski, R., Huang, Y., Ryabinin, M., Jia, Z., and Chen, B. Sequoia: Scalable, robust, Art of Problem Solving. Aime problems and solu- and hardware-aware speculative decoding, 2025. URL tions. https://artofproblemsolving.com/ https://arxiv.org/abs/2402.12374. wiki/index.php/AIME, 2024. Accessed: 2025-04- 20. Cheng, S., Bian, Y., Liu, D., Zhang, L., Yao, Q., Tian, Z., Wang, W., Guo, Q., Chen, K., Qi, B., and Zhou, B. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, Sdar: A synergistic diffusion-autoregression paradigm H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and for scalable sequence generation, 2025. URL https: Sutton, C. Program synthesis with large language mod- //arxiv.org/abs/2510.06303. els, 2021. URL https://arxiv.org/abs/2108. ``` ### Table 4. Decoding speedup over baseline and average acceptance length (τ ) on Qwen3 models with thinking mode disabled and a ```text Table 4. Decoding speedup over baseline and average acceptance length (τ ) on Qwen3 models with thinking mode disabled and a maximum of 2048 generated tokens. M ATH C ODE C HAT Avg. Model Method GSM8K MATH-500 AIME24 AIME25 HumanEval MBPP LCB SWE-Bench MT-Bench Alpaca Avg. Temperature = 0 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ dFlash 5.1× 6.5 6.5× 7.9 6.5× 7.4 6.2× 7.4 5.6× 6.7 5.0× 6.1 5.9× 6.9 3.2× 3.6 2.9× 4.4 2.4× 3.1 4.9× 6.0 4B PRESTO 7.9× 9.4 9.2× 10.9 8.7× 10.5 8.8× 10.3 8.1× 9.9 7.9× 9.3 8.4× 10.0 5.3× 5.9 4.9× 6.7 3.9× 5.0 7.3× 8.8 × Gain 1.55× 1.42× 1.34× 1.42× 1.45× 1.58× 1.42× 1.66× 1.69× 1.62× 1.48× dFlash 5.0× 6.6 6.3× 7.9 6.2× 7.5 6.0× 7.0 5.4× 6.5 4.8× 6.0 5.7× 7.1 3.3× 3.6 2.8× 4.3 2.4× 3.1 4.8× 6.0 8B PRESTO 8.0× 9.6 9.4× 11.1 8.5× 10.7 8.7× 10.2 8.5× 9.9 7.7× 9.4 8.5× 10.3 5.0× 6.0 4.8× 6.6 3.9× 5.0 7.3× 8.9 × Gain 1.60× 1.49× 1.37× 1.45× 1.57× 1.60× 1.49× 1.52× 1.71× 1.62× 1.52× dFlash 3.1× 5.2 4.1× 5.6 2.8× 5.3 2.7× 5.1 5.6× 8.0 5.6× 7.2 3.6× 6.2 3.2× 3.6 2.2× 3.5 1.8× 2.2 3.5× 5.2 30B PRESTO 4.1× 7.9 5.4× 8.0 3.6× 8.0 3.6× 7.9 7.0× 10.9 7.5× 9.9 4.7× 8.5 4.6× 5.6 3.0× 5.2 2.5× 3.4 4.6× 7.5 × Gain 1.32× 1.32× 1.29× 1.33× 1.25× 1.34× 1.31× 1.44× 1.36× 1.39× 1.33× Temperature = 1 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ dFlash 4.7× 6.0 5.2× 6.6 3.8× 5.0 3.9× 4.9 4.8× 6.0 4.4× 5.6 5.0× 6.6 2.5× 3.1 2.7× 4.1 2.2× 3.0 3.9× 5.1 4B PRESTO 7.2× 9.1 7.6× 9.8 6.0× 7.7 6.1× 7.7 7.6× 9.3 7.0× 9.0 6.7× 9.4 4.2× 5.0 4.3× 6.2 3.7× 4.7 6.0× 7.8 × Gain 1.53× 1.46× 1.58× 1.56× 1.58× 1.59× 1.34× 1.68× 1.59× 1.68× 1.54× dFlash 4.8× 6.0 5.0× 6.6 3.9× 5.1 3.8× 5.0 4.4× 5.4 4.1× 5.2 5.2× 6.8 2.3× 2.8 2.6× 3.8 2.1× 2.9 3.8× 5.0 8B PRESTO 7.4× 9.0 7.4× 9.6 6.3× 7.8 6.5× 7.7 7.2× 8.6 6.9× 8.6 7.0× 9.8 4.0× 4.7 4.2× 5.9 3.6× 4.7 6.0× 7.6 × Gain 1.54× 1.48× 1.62× 1.71× 1.64× 1.68× 1.35× 1.74× 1.62× 1.71× 1.58× dFlash 4.1× 5.1 4.2× 5.3 3.3× 4.3 3.3× 4.2 5.8× 7.7 5.6× 7.1 3.8× 5.6 2.7× 3.2 2.1× 3.2 1.8× 2.1 3.7× 4.8 30B PRESTO 4.5× 7.8 5.5× 7.7 3.7× 6.7 3.4× 6.5 6.8× 10.2 7.9× 9.6 4.3× 8.0 4.4× 4.9 3.0× 5.0 2.6× 3.3 4.6× 7.0 × Gain 1.10× 1.31× 1.12× 1.03× 1.17× 1.41× 1.13× 1.63× 1.43× 1.44× 1.26× ``` ### Table 5. Decoding speedup over baseline and average acceptance length (τ ) on Nemotron-Labs-Diffusion-8B at linear self speculation ```text Table 5. Decoding speedup over baseline and average acceptance length (τ ) on Nemotron-Labs-Diffusion-8B at linear self speculation decoding mode with a maximum of 2048 generated tokens. M ATH C ODE C HAT Avg. Model Method GSM8K MATH-500 AIME24 AIME25 HumanEval MBPP LCB SWE-Bench MT-Bench Alpaca Avg. Temperature = 0 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ vanilla 5.7× 10.8 6.4× 12.2 5.7× 11.0 5.5× 10.5 5.0× 9.3 4.4× 8.2 4.7× 9.0 3.7× 6.8 2.8× 5.2 2.5× 4.6 4.6× 8.7 8B PRESTO 6.0× 11.9 6.7× 13.2 6.1× 12.2 5.8× 11.5 5.2× 10.3 4.6× 9.1 4.9× 10.0 4.0× 7.9 3.2× 6.5 2.9× 5.8 4.9× 9.9 × Gain 1.05× 1.04× 1.07× 1.06× 1.04× 1.04× 1.04× 1.08× 1.14× 1.16× 1.07× Temperature = 1 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ vanilla 2.9× 5.6 2.9× 5.7 2.5× 5.0 2.6× 4.9 2.3× 4.6 2.1× 4.2 2.0× 4.0 1.4× 2.9 1.4× 2.7 1.2× 2.3 2.1× 4.2 8B PRESTO 3.1× 6.1 3.3× 7.1 2.9× 5.7 2.9× 6.4 2.7× 5.5 2.6× 5.0 2.3× 4.8 1.8× 3.6 1.8× 3.5 1.5× 3.2 2.5× 5.1 × Gain 1.07× 1.14× 1.16× 1.12× 1.17× 1.24× 1.15× 1.29× 1.29× 1.25× 1.17× PRESTO on dFlash. As illustrated in Table 4, PRESTO consistently achieves substantial improvements in the average acceptance length τ across all models and tasks, which directly translates into higher end-to-end throughput speedup. For example, on Qwen3-4B with T = 0, τ increases from 7.9 to 10.9 on MATH-500 and from 7.4 to 10.3 on AIME25, while on code benchmarks it improves from 6.9 to 10.0 on LCB and from 6.7 to 9.9 on HumanEval, leading to corresponding speedup gains across all settings. The largest improvements are observed on math and code benchmarks, where τ typically increases by +2 to +3 tokens. These trends remain stable across model scales and decoding regimes: even for stronger drafters such as Qwen3-8B and under stochastic decoding (T = 1), PRESTO continues to deliver comparable relative gains. These results indicate that PRESTO effectively pushes the acceptance length of dFlash closer to its upper bound, delivering consistent and substantial end-to-end throughput speedup gains. PRESTO on Nemotron-Labs-Diffusion (NLD). As shown in Table 5, PRESTO consistently improves both throughput and average acceptance length over the official linear self-speculative decoding implementation of Nemotron-Labs-Diffusion across all benchmarks and decoding temperatures. Notably, these experiments are conducted using the highly optimized SGLang implementation released by NLD, where substantial additional gains are already difficult to obtain. Nevertheless, 14 PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding ``` ### Table 6. Decoding speedup over baseline and average acceptance length (τ ) on Nemotron-Labs-Diffusion-8B at quadratic self speculation ```text Table 6. Decoding speedup over baseline and average acceptance length (τ ) on Nemotron-Labs-Diffusion-8B at quadratic self speculation decoding mode with a maximum of 2048 generated tokens. M ATH C ODE C HAT Avg. Model Method GSM8K MATH-500 AIME24 AIME25 HumanEval MBPP LCB SWE-Bench MT-Bench Alpaca Avg. Temperature = 0 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ vanilla 2.6× 4.7 2.7× 6.8 3.2× 5.3 3.3× 5.7 3.2× 4.8 3.1× 4.5 2.4× 4.7 1.8× 3.5 2.1× 3.0 1.8× 2.7 2.6× 4.7 8B PRESTO 5.3× 6.5 4.4× 7.5 3.5× 6.1 3.7× 6.4 3.7× 5.8 3.7× 5.4 3.1× 5.6 2.7× 4.7 2.8× 4.2 2.3× 3.6 3.5× 5.6 × Gain 2.04× 1.63× 1.09× 1.12× 1.16× 1.19× 1.29× 1.50× 1.33× 1.28× 1.35× Temperature = 1 Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ Sp. τ vanilla 1.4× 3.0 1.8× 3.4 1.5× 2.6 1.7× 3.0 1.7× 2.6 1.6× 2.4 1.2× 2.3 1.0× 1.8 1.2× 1.7 1.0× 1.5 1.4× 2.4 8B PRESTO 4.1× 5.7 4.0× 6.6 2.9× 5.1 3.3× 5.7 3.2× 4.9 3.0× 4.4 2.8× 4.8 2.2× 3.7 2.3× 3.5 2.0× 3.0 3.0× 4.7 × Gain 2.93× 2.22× 1.93× 1.94× 1.88× 1.88× 2.33× 2.20× 1.92× 2.00× 2.14× by replacing linear drafting with tree-based exploration while preserving the original verification pipeline, PRESTO remains effective across a wide range of tasks. Under deterministic decoding (T = 0), PRESTO increases average end-to-end throughput from 4.6× to 4.9× while improving average acceptance length from 8.7 to 9.9, with consistent gains observed on all benchmarks. Under stochastic decoding (T = 1), where a single drafted trajectory is more likely to diverge from the target model, the benefits become more pronounced: average end-to-end throughput improves from 2.1× to 2.5×, while average acceptance length increases from 4.2 to 5.1. Across individual benchmarks, PRESTO achieves up to 1.29× end-to-end throughput improvement and increases acceptance length by as much as 39% on Alpaca under stochastic decoding (T=1). These results demonstrate that the benefits of tree-based drafting persist even within a production-grade, highly optimized inference engine. PRESTO on Nemotron-Labs-Diffusion (NLD) at quadratic self speculation mode. We further evaluate PRESTO under the quadratic self-speculative decoding setting introduced in Nemotron-Labs-Diffusion. As Table 6 shows, across all benchmarks and decoding settings, PRESTO also consistently outperforms vanilla quadratic self-speculative decoding in both end-to-end throughput and average acceptance length. Under deterministic decoding (T = 0), PRESTO provides stable gains across math, code, and chat tasks, improving average speedup from 2.6× to 3.5× on NLD-8B, while also consistently increasing acceptance length from 4.7 to 5.6. The largest gains are observed on math benchmarks, where PRESTO achieves up to 2.04× on GSM8k for NLD-8B. Gains are largest under stochastic decoding (T = 1), where vanilla single-path drafting degrades sharply from trajectory mismatch and accumulated uncertainty, while PRESTO ’s tree-structured exploration retains multiple plausible continuations. On NLD-8B, this raises average end-to-end speedup from 1.4× to 3.0× and nearly doubles acceptance length from 2.4 to 4.7, reaching up to 2.9× throughput and over 2× acceptance length over vanilla self-speculative decoding. These positive results indicate the significant benefits of applying PRESTO to quadratic self-speculation. We note that the throughput for dFlash and NLD quadratic self speculation can be further improved through system-level optimizations, including custom kernels, optimized KV-cache management, and efficient request scheduling. The goal here is not to fully optimize absolute throughput, but rather to provide a first-hand comparison between linear and tree-based drafting for diffusion drafters under the same native PyTorch implementation. 15 PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding F. More Discussions and Ablations Figure 4. Top: prefix-conditioned signals consistently improve acceptance length. Middle: effect of λd under different datasets and tree token budgets. Bottom left: Per-step latency breakdown across different model scales. Bottom right: Comparison of accepted lengths distribution on GSM8K. Note that all results are obtained from Qwen3-4B-DFlash with diffusion block size being 16. Effectiveness of prefix-conditioned signal. We first ablate the contribution of the prefix-conditioned compatibility term ρd by comparing our full scoring function with only using the diffusion marginal confidence qd . As shown in Figure 4 top, incorporating ρd consistently improves the average acceptance length across tasks, demonstrating the effectiveness of prefix-conditioned compatibility. Sensitivity of λd . Figure 4 middle shows that moderate λd consistently achieves the best acceptance length across different datasets and tree token budgets, with λd ≈ 0.2 yielding the most stable gains. Small λd underutilizes the prefix-conditioned signal, while overly large values overly suppress the diffusion confidence qd , leading to substantial performance degradation. Entropy-Adaptive λd vs. Fixed λd . Beyond the fixed interpolation coefficient λd used in our main experiments, we explore an entropy-adaptive variant that scales the prefix correction by thePdiffusion drafter’s uncertainty: confident predictions rely more on qd , while uncertain ones up-weight ρd . Letting Hd = − w∈V qd (w) log qd (w) denote the entropy of the diffusion marginal at depth d, we define Hd λd (Hd ) = β · , Hd + C where β > 0 controls the maximum correction strength and C > 0 stabilises the ratio. Substituting into the token-level score from Section 4 gives sd,t (cd ) = log qd (t) + λd (Hd ) log ρd (t | cd ), which recovers the fixed-λd variant when λd (Hd ) is replaced by a constant. Empirically, this entropy-adaptive schedule yields only marginal gains over the fixed variant (Figure 4, top), suggesting that a simple fixed λd already captures most of ``` ## reference/nemotron/Teaching Diffusion to Speculate Left-to-Right ### Table 1: Draft-token utilisation of a position-uniform DFlash ```text Table 1: Draft-token utilisation of a position-uniform DFlash Token Position drafter trained against Llama-3-8B-Instruct with block size K = 16, evaluated across five benchmarks. Each block con- Figure 2: Per-position breakdown of the drafter’s correct tributes 16 candidate slots: 15 drafted positions plus the tar- predictions on HumanEval, for a position-uniform DFlash get’s bonus correction sampled at the rejection point (always baseline trained against Llama-3-8B-Instruct with block size retained, always matches the target’s greedy output). Accept K = 16. Bars are stacked: the green segment is the fraction of is the fraction of the 16 slots whose token is kept by the draft tokens accepted under the rejection-sampling contract rejection-sampling contract (1), i.e. accepted drafts plus the (1), and the red segment is the fraction that actually match bonus; Correct is the fraction whose draft (or bonus) matches the target, but is nevertheless discarded as a consequence of the target’s greedy output; Waste is the fraction of those cor- an upstream rejection within the same block. Per-position rect tokens that the verification contract is nevertheless forced correctness (total bar height) decays only mildly with offset to discard due to an upstream rejection within the same block, k, whereas per-position acceptance (green) decays at the ge- i.e. Wasted/(Accepted + Bonus + Wasted). ometric rate implied by (3), so the discarded fraction grows monotonically and dominates beyond a short prefix. attention over the K draft positions: 3.3 Block-Diffusion Drafters (x̃t , . . . , x̃t+K−1 ) ∼ qψ (· | H