Multi-Block Diffusion Language Models Yijie Jin1 , Jiajun Xu2 , Yuxuan Liu1 , Chenkai Xu1 , Yi Tu3 , Jiajun Li3 , Dandan Tu3 , Xiaohui Yan3 , Kai Yu1 , Pengfei Liu1 , Zhijie Deng1,† 1 Shanghai Jiao Tong University, 2 Xi’an Jiao Tong University, 3 Huawei Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation. A natural next step is to extend them from Single-Block Diffusion (SingleBD) to Multi-Block Diffusion (MultiBD), where a running-set of consecutive blocks is decoded concurrently for inter-block parallelism. However, existing BD- LMs are mostly trained under teacher forcing, where the model observes only one noisy block conditioned on a clean prefix. While the recent diffusion forcing strategy introduces visibility among multiple noisy blocks, its training states still arXiv:2606.29215v2 [cs.LG] 30 Jun 2026 differ from MultiBD inference, where decoding operates on a bounded running-set with heterogeneous slot-wise noise patterns. To bridge this gap, we propose Multi-Block Diffusion Language Models (MBD-LMs), obtained by post-training BD-LMs with Multi-block Teacher Forcing (MultiTF). MultiTF integrates teacher forcing and diffusion forcing by training on bounded noise-groups conditioned on clean prefixes, with randomized noise-schedulers that better match MultiBD inference states. To make MultiBD practically executable, we further introduce an optimized decoding algorithm based on the Block Buffer mechanism that preserves prefix-cache reuse, keeps input shapes static, and translates increased decoding parallelism into wall-clock acceleration. Empirically, MBD-LLaDA2-Mini increases average Tokens Per Forward pass (TPF) from 3.47 to 6.19 and improves average accuracy from 79.95% to 81.03%; when combined with DMax, MBD- LLaDA2-Mini-DMax reaches an average TPF of 9.34 with only a 1.02% accuracy drop on math and code benchmarks. Project Page: https://sjtu-deng-lab.github.io/mbd-lms Correspondence: Zhijie Deng: zhijied@sjtu.edu.cn Contributions: † Corresponding author. Date: July 1, 2026 1 Introduction Diffusion Language Models (DLMs) have emerged as a promising alternative to autoregressive language models by enabling native parallel decoding (Sahoo et al., 2024; Nie et al., 2025). However, fully bidirectional DLMs struggle to serve efficiently because they lack support for KV caching and dynamic-length generation. Recent Block Diffusion Language Models (BD-LMs) have become a representative DLM paradigm for efficient generation, addressing the above limitations through block-causal generation (Arriola et al., 2025; Bie et al., 2025; Cheng et al., 2025). Most BD-LMs trained under Teacher Forcing (TF) naturally support Single-Block Diffusion (SingleBD): at each forward pass, the model decodes one noisy block while preceding blocks are already clean and cached, enabling KV caching and intra-block parallelism. However, blocks themselves are still processed sequentially. As shown in Figure 1, SingleBD must finish decoding a block and storing its KV cache before later blocks can proceed, creating storing bubbles and locking inter-block parallelism. The Discrete Diffusion Forcing (D2F) (Wang et al., 2025) strategy introduces the visibility of multiple noisy blocks to BD-LMs. Conditioned on a clean prefix, it corrupts suffix blocks with monotonic increasing noise ratios during training. Consequently, D2F obtains Multi-Block Diffusion (MultiBD) capability, as shown in Figure 1, enabling decode-store overlap and inter-block parallelism. However, a train–inference mismatch problem remains. Specifically, it is not possible to process the entire noisy suffix as one running-set in a single forward pass, from both the perspectives of efficiency and empirical efficacy (Lu et al., 2026). For the naive MultiBD introduced by D2F, the expected running-set size is often around two, and adjacent slots exhibit large noise-ratio gaps. This suggests that reliable MultiBD requires training states that match both the bounded running-set size and the heterogeneous slot-wise noise patterns observed during inference. 1 KV Cache Storing Bubble: No token generated, no decode-store overlap 2 Single-Block Diffusion: 3 Native BD-LMs Inference 4 Block Index Multi-Block Diffusion: Ours 2-Level Pipelining: Store KV Cache and Goal: Generate 4 blocks of tokens generate tokens concurrently 1 Assuming a MultiBD process with buffer-size of 2, 2 which construct a 2-level decoding pipeline, overlapping decoding and storing. 3 Parallel Decoding: 2 active 4 Active Step: darker color -> higher noise blocks are decoded in parallel Padding Storing Step Forward-pass Steps Figure 1 SingleBD decodes blocks sequentially and creates KV cache storing bubbles. In contrast, MultiBD overlaps future-block refinement with KV cache storing of completed blocks, and enables inter-block parallelism. To this end, we formulate Multi-Block Diffusion Language Models (MBD-LMs), a unified view of existing BD-LMs. This view covers both TF-trained BD-LMs and D2F-trained BD-LMs as extreme cases, while identifying practical MultiBD as the bounded intermediate regime for reliable and efficient inference. We introduce Multi-block Teacher Forcing (MultiTF), a post-training method that turns BD-LMs into MBD- LMs. MultiTF extends TF by concatenating the clean prefix with a bounded group of consecutive noisy blocks, where noisy blocks can attend to each other under a Group-Aware Dual-Stream Mask. It applies a more aggressive and randomized noise-scheduler within each noise-group to simulate the heterogeneous slot-wise noise patterns observed during inference. During training, blocks are partitioned into groups with varying sizes to cover possible running-set sizes and group-relative positions. We further propose an optimized inference pipeline for MultiBD. MultiBD relies on a dynamic running-set for decoding, which is unfriendly to CUDA Graph capture and replay. To address this, we introduce the Block Buffer mechanism, which maintains a fixed number of block slots. Future blocks enter the Block Buffer by activating existing idle slots rather than extending the physical input, while completed front blocks leave after being committed to the KV cache. This design keeps the input shape static, preserves KV caching and prefix caching, and translates the increased TPF into practical wall-clock speedup. Experiments on math and code benchmarks show that MBD-LMs improve decoding parallelism while pre- serving generation quality. Compared with LLaDA2-Mini (Bie et al., 2025), MBD-LLaDA2-Mini increases the average TPF from 3.47 to 6.19 (+78.4%) and improves the average accuracy from 79.95% to 81.03%. When combined with DMax (Chen et al., 2026), MBD-LLaDA2-Mini-DMax further reaches an average TPF of 9.34 (+47.1% over LLaDA2-Mini-DMax under SingleBD) with only a 1.02 percentage-point accuracy drop. Using our inference engine, MBD-LLaDA2-Mini-DMax achieves 951.41 TPS on average, compared with 781.50 TPS for LLaDA2-Mini-DMax.  Main Contributions ¥ Unified MBD-LM formulation. We formulate Multi-Block Diffusion Language Models (MBD- LMs) as a unified DLM framework parameterized by a running-set of consecutive blocks. This view covers both TF-trained BD-LMs and D2F-trained BD-LMs, while identifying practical MultiBD as the bounded intermediate regime for reliable and efficient inference. ¥ MultiTF post-training for MBD-LMs. We propose Multi-block Teacher Forcing (MultiTF), a post-training method that turns BD-LMs into MBD-LMs. MultiTF improves train–inference align- ment by training BD-LMs on states that resemble practical MultiBD inference. ¥ Optimized MultiBD inference engine. We design and implement an optimized MultiBD infer- ence pipeline based on the Block Buffer mechanism. The pipeline overlaps decoding and KV cache storing, preserves prefix caching, and keeps input shapes static for CUDA Graph capture and replay, translating increased TPF into practical TPS gains. (A) Training Noise — D2F (B) Training Noise — Chain Uniform (Ours) (C) Inference Mask Ratio (D) Mean ± 1σ Active Blocks (E) Sampled Trajectories 4.0 slot 1 slot 6 slot 1 slot 2 trained slot 1 base slot 1 ±1σ 4.0 slot 2 slot 8 trained slot 2 base slot 2 2.50 Mean 3.5 slot 4 3.5 8 4 2.25 3.0 3.0 2.00 Active Blocks Active Blocks 2.5 6 3 2.5 Density 1.75 2.0 2.0 4 2 1.5 1.50 1.5 1.0 1.25 1.0 2 1 0.5 0.5 1.00 sample 1 sample 3 sample 5 0.0 0 0 0.0 sample 2 sample 4 sample 6 0.75 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0 50 100 150 200 250 0 20 40 60 80 100 120 Noise / Mask Ratio Noise / Mask Ratio Noise / Mask Ratio Decode Step Decode Step Figure 2 Train–inference statistics for MultiBD. (A) Slot-wise mask-ratio distributions induced by the D2F-style monotonic scheduler. (B) Slot-wise mask-ratio distributions induced by our chain-uniform scheduler. (C) Inference- time mask-ratio distributions before and after MultiTF post-training. (D) Mean and one-standard-deviation range of the active-block count during MultiBD inference. (E) Sampled active-block trajectories during decoding. Panels (A–C) compare scheduler-induced training noise patterns with inference-time mask-ratio patterns for train–inference alignment analysis. Panels (D–E) report the active part of the MultiBD running-set under a buffer size of four; the active-block count can therefore occasionally exceed two. 2 Preliminaries 2.1 Diffusion Language Models Diffusion Language Models (DLMs) (Sahoo et al., 2024; Nie et al., 2025; Ye et al., 2025) formulate text generation as iterative denoising. Let V denote the vocabulary, [M] denote a special mask token, and L denote the sequence length. Given a clean sequence x0 = (x10 , . . . , xL 0 ) ∈ V , the forward process gradually L masks tokens independently. For t ∈ [0, 1], the noisy sequence xt ∈ (V ∪ {[M]})L masks each token with probability t:  i i 1 − t, xt = x0 ,  i i qt (xt | x0 ) = t, i xt = [M], (2.1) otherwise.  0,  Let M(xt ) = {i : xit = [M]} denote the masked positions. A DLM parameterized by θ predicts clean tokens at masked positions: L Y pθ (x0 | xt ) = pθ (xi0 | xt ). (2.2) i=1 The standard training objective is a weighted masked-token cross-entropy (Nie et al., 2025):  X L  1 LDLM (θ) = −Et,x0 ,xt i i 1[x = [M]] · log pθ (x0 | xt ) , (2.3) t i=1 t where t ∼ U(0, 1), xt ∼ qt (· | x0 ), and 1[·] denotes the indicator function, ensuring that the loss is computed only on masked tokens. The inference starts from an all-[M] sequence and iteratively fills high-confidence masked positions. 2.2 Block Diffusion Language Models Block Diffusion Language Models (BD-LMs) (Arriola et al., 2025; Bie et al., 2025) partition the sequence into blocks, i.e., x0 = [b1 , . . . , bK ], bk ∈ V B , (2.4) where B is the block size and K = L/B is the number of blocks. BD-LMs model the sequence autoregressively at the block level: K ( 1, and the forward pass processes all physical buffer slots, including active blocks, completed resident blocks, and dummy slots used to preserve static input shapes. Thus, Qs measures the computational workload of a forward pass, whereas TPF measures the number of useful tokens committed by that forward pass. This distinction defines a token-efficiency factor: TPFs ηtok (s) = . Qs Equivalently, TPF ηtok Qs TPS = = . Tstep Tstep Therefore, increasing the block-buffer size can improve throughput only when the useful-token gain outweighs the additional per-step cost. MultiBD increases Qs and enables more tokens to be committed per forward pass, but its token efficiency can be reduced by inactive dummy slots and resident blocks that are processed for static-shape execution but do not immediately contribute to committed tokens. Each decoding forward can be viewed as an extend-attention step with Qs query tokens and a cached prefix of length Ps . For a transformer with Nlayer layers, hidden size d, FFN hidden size dff , and vocabulary V, the per-step FLOPs can be approximated as Fstep (Qs , Ps ) = Θ Nlayer Qs (d2 + ddff ) + d(Qs Ps + Q2s ) + Qs d|V| .    The first term comes from QKV/O projections and FFN layers, the second term comes from attention between the buffer and the cached prefix as well as attention inside the buffer, and the last term comes from the LM head when logits are computed. Thus, increasing Nbuf from 1 to a larger value improves inter-block decoding parallelism, but also increases the amount of computation performed by each forward pass. The memory cost follows the same extend-attention structure. Let sdtype be the number of bytes per activation element. The per-step weight traffic scales as MW = Θ Nlayer sdtype (d2 + ddff ) ,  while the KV-cache traffic of extend attention can be approximated as MKV (Qs , Ps ) = Θ (Nlayer sdtype [ρ(Qs )(Ps + Qs )d + Qs d]) , where ρ(Qs ) captures repeated KV reads caused by query tiling. The first term corresponds to reading KV cache for the prefix and current buffer, while the second term corresponds to KV cache storing. This gives a roofline-style view of the step latency:   Fstep (Qs , Ps ) MW + MKV (Qs , Ps ) Tstep (Qs , Ps ) ≈ max , + Tcomm (Qs ) + Tlaunch , Πeff BHBM Table 3 Throughput and single-step latency comparison. Results are measured for single-sample decoding on two H100 GPUs with tensor parallelism degree 2 (TP=2). Step latency denotes the average wall-clock latency of one decoding forward pass. TPF and TPS gains are computed relative to LLaDA2-Mini, while latency cost reports the relative increase in per-step latency. Forward-step statistics Realized throughput Model Avg. TPF ↑ TPF Gain ↑ Step Lat. (ms) ↓ Lat. Cost ↓ GSM8K TPS ↑ MATH500 TPS ↑ MBPP+ TPS ↑ HumanEval+ TPS ↑ Avg. TPS ↑ TPS Gain ↑ LLaDA2-Mini 3.47 – 7.07 1.00× 344.05 403.45 496.19 824.94 517.16 – MBD-LLaDA2-Mini 6.19 +78.39% 8.78 1.24× 687.87 707.89 646.73 941.18 745.92 +44.24% LLaDA2-Mini-DMax 6.35 +83.00% 9.02 1.28× 700.82 730.60 754.97 931.55 779.49 +50.73% MBD-LLaDA2-Mini-DMax 9.34 +169.16% 11.20 1.58× 834.52 851.07 896.65 1124.43 926.67 +79.19% where Πeff is the effective compute throughput, BHBM is the effective HBM bandwidth, Tcomm includes fixed-configuration tensor-parallel communication, and Tlaunch denotes launch and runtime overhead. This expression shows that the realized throughput depends on both the useful-token numerator and the roofline- limited per-step cost denominator. The attention arithmetic intensity further explains why MultiBD can still be efficient despite processing more tokens per step. Ignoring lower-order terms, the attention arithmetic intensity is approximately   dQs Ps Qs AIattn ≈ =Θ sdtype ρ(Qs )Ps d sdtype ρ(Qs ) when Ps ≫ Qs . Therefore, increasing Qs through a larger Block Buffer makes the extend-attention step more compute intensive. Prefix KV reads, weight reads, and kernel-launch overheads are amortized over more query tokens. However, the gain is useful only to the extent that these processed tokens lead to committed tokens, as captured by ηtok . The measurements in Table 3 match this analysis. For LLaDA2-Mini, MBD increases the average TPF from 3.47 to 6.19, a 1.78× improvement, while the step latency increases from 7.07 ms to 8.78 ms, a 1.24× cost increase. The expected throughput scaling is therefore approximately 1.78/1.24 = 1.44×, closely matching the measured Avg. TPS improvement from 517.16 to 745.92, i.e., 1.44×. Similarly, for LLaDA2-Mini-DMax, MBD increases the average TPF from 6.35 to 9.34, a 1.47× improvement, while the step latency increases from 9.02 ms to 11.20 ms, a 1.24× cost increase. This predicts a throughput scaling of 1.47/1.24 = 1.18×, which closely matches the measured Avg. TPS improvement from 779.49 to 926.67, i.e., 1.19×. Thus, the observed gap between TPF gain and TPS gain is primarily explained by the increased per-forward cost of processing the larger static Block Buffer. Overall, MultiBD improves wall-clock throughput by increasing the number of useful tokens committed per forward pass and by making each extend-attention step more compute intensive. At the same time, static- shape execution introduces extra processed tokens through resident blocks and dummy slots, reducing token efficiency relative to the ideal case. The final TPS gain is therefore determined by the balance among TPF improvement, token efficiency, and roofline-limited step latency. 5 Related Work 5.1 Diffusion Language Models Diffusion Language Models (DLMs) generate text through iterative denoising and enable parallel token refinement as an alternative to autoregressive generation. Representative models include LLaDA (Nie et al., 2025), Dream (Ye et al., 2025), and LLaDA2.x (Bie et al., 2025, 2026), which improve scaling, initialization, and editable refinement. However, fully bidirectional DLMs are difficult to serve efficiently because they do not naturally support KV caching or flexible-length generation. Block Diffusion Language Models (BD-LMs) (Arriola et al., 2025; Bie et al., 2025; Cheng et al., 2025) address these limitations by introducing block-causal generation. Their native Single-Block Diffusion (SingleBD) inference decodes one noisy block conditioned on a clean cached prefix, enabling KV caching and intra-block parallel decoding. Nevertheless, SingleBD still processes blocks sequentially, leaving inter-block parallelism underused. Our work studies Multi-Block Diffusion (MultiBD) as a broader inference regime for BD-LMs, where a bounded running-set of consecutive blocks can be refined concurrently. 5.2 Efficient DLM Inference and Training Efficient DLMs have been studied through distillation, scheduling, caching, and parallel decoding. D2F (Wang et al., 2025) introduces noisy-block visibility during training and demonstrates the potential of MultiBD-style pipelined decoding. DMax (Chen et al., 2026), d3LLM (Qian et al., 2026), LightningRL (Hu et al., 2026), and dParallel (Chen et al., 2025) improve the accuracy–parallelism trade-off through training objectives or decoding schedules. Fast-dLLM (Wu et al., 2025) and LoPA (Xu et al., 2025) accelerate inference through caching and lookahead parallelism. Our work is complementary to these efforts but focuses on a different level of parallelism. Instead of only increasing token-level parallelism or applying MultiBD as an inference-time heuristic, we treat MultiBD as a target inference regime for BD-LMs. We identify the bounded running-set structure and heterogeneous slot-wise noise patterns as key train–inference alignment factors, and propose MultiTF to post-train BD-LMs into MBD-LMs with inference-like multi-block states. We further provide Block Buffer inference support so that MultiBD preserves prefix-cache reuse and static-shape execution. 6 Conclusion We proposed Multi-Block Diffusion Language Models (MBD-LMs), a unified formulation of BD-LMs for reliable MultiBD inference. Starting from the sequential bottleneck of SingleBD, we showed that MultiBD can expose inter-block parallelism but requires training states aligned with its bounded running-set structure and heterogeneous slot-wise noise patterns. To bridge this gap, we introduced Multi-block Teacher Forcing (MultiTF), which post-trains BD-LMs with bounded noise-groups, the Group-Aware Dual-Stream Mask, and randomized block-level noise-schedulers. We further developed an optimized MultiBD inference engine with the Block Buffer mechanism, enabling static-shape execution while preserving KV caching and prefix-cache reuse. Experiments on math and code benchmarks show that MBD-LMs improve decoding parallelism and realized throughput while maintaining generation quality, demonstrating that reliable MultiBD requires both training-time state alignment and inference-time system support. References Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2503.09573. Oral Presentation. Tiwei Bie, Zenan Huang, Chongxuan Li, et al. Llada2.0: Scaling up diffusion language models to 100b. arXiv preprint arXiv:2512.15745, 2025. URL https://arxiv.org/abs/2512.15745. Tiwei Bie et al. Llada2.1: Speeding up text diffusion via token editing. arXiv preprint arXiv:2602.08676, 2026. URL https://arxiv.org/abs/2602.08676. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El-Haddad, Céline Hudelot, and Pierre Colombo. When does reasoning matter? a controlled study of reasoning’s contribution to model performance. arXiv preprint arXiv:2509.22193, 2025. URL https://arxiv.org/abs/2509.22193. Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. dparallel: Learnable parallel decoding for dllms. arXiv preprint arXiv:2509.26488, 2025. URL https://arxiv.org/abs/2509.26488. Zigeng Chen, Gongfan Fang, Xinyin Ma, Ruonan Yu, and Xinchao Wang. Dmax: Aggressive parallel decoding for dllms. arXiv preprint arXiv:2604.08302, 2026. URL https://arxiv.org/abs/2604.08302. Shuang Cheng, Yihan Bian, Dawei Liu, Linfeng Zhang, Qian Yao, Zhongbo Tian, Wenhai Wang, Qipeng Guo, Kai Chen, Biqing Qi, and Bowen Zhou. Sdar: A synergistic diffusion-autoregression paradigm for scalable sequence generation. arXiv preprint arXiv:2510.06303, 2025. URL https://arxiv.org/abs/2510.06303. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. Yanzhe Hu, Yijie Jin, Pengfei Liu, Kai Yu, and Zhijie Deng. Lightningrl: Breaking the accuracy–parallelism trade-off of block-wise dllms via reinforcement learning. arXiv preprint arXiv:2603.13319, 2026. URL https://arxiv.org/abs/2603. 13319. jtatman. Python code dataset 500k. Hugging Face dataset, 2025. URL https://huggingface.co/datasets/jtatman/ python-code-dataset-500k. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572, 2023. Guanxi Lu, Hao Mark Chen, Yuto Karashima, Zhican Wang, Daichi Fujiki, and Hongxiang Fan. Adablock-dllm: Semantic-aware diffusion llm inference via adaptive block size. arXiv preprint arXiv:2509.26432, 2026. URL https://arxiv.org/abs/2509.26432. Qianli Ma, Yaowei Zheng, Zhelun Shi, Zhongkai Zhao, Bin Jia, Ziyue Huang, Zhiqi Lin, Youjie Li, Jiacheng Yang, Yanghua Peng, Zhi Zhang, and Xin Liu. Veomni: Scaling any modality model training with model-centric distributed recipe zoo. arXiv preprint arXiv:2508.02317, 2025. URL https://arxiv.org/abs/2508.02317. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025. URL https://arxiv.org/abs/ 2502.09992. Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, and Hao Zhang. d3llm: Ultra-fast diffusion llm using pseudo-trajectory distillation. arXiv preprint arXiv:2601.07568, 2026. URL https://arxiv.org/abs/ 2601.07568. Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexan- der Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07524, 2024. URL https://arxiv.org/abs/2406.07524. Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09192, 2025. URL https://arxiv.org/abs/2508.09192. Chengyue Wu et al. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025. URL https://arxiv.org/abs/2505.22618. Chenkai Xu, Yijie Jin, Jiajun Li, Yi Tu, Guoping Long, Dandan Tu, Mingcong Song, Hongjie Si, Tianqi Hou, Junchi Yan, and Zhijie Deng. Lopa: Scaling dllm inference via lookahead parallel decoding. arXiv preprint arXiv:2512.16229, 2025. URL https://arxiv.org/abs/2512.16229. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487, 2025. URL https://arxiv.org/abs/2508.15487. A Theoretical View of MultiTF This appendix provides a simple theoretical view of Multi-block Teacher Forcing (MultiTF). The goal is not to prove that MultiTF directly improves downstream accuracy. Instead, we show that MultiTF can be interpreted as a coverage-based surrogate for the ideal MultiBD training objective, and that its approximation gap is controlled by the mismatch in running-set coverage and noise-ratio distributions. Ideal MultiBD objective. Let R = {a, . . . , c} denote a consecutive MultiBD running-set with size |R| ≤ Gmax , where Gmax is the maximum noise-group size used in MultiTF. For a clean sequence x0 , noise ratios t, and model θ, define the state loss 1 X ( τadd and EOS has not appeared then 5: Append a fully masked future block to Y. 6: end if // Decode all blocks in the current running-set 7: Run θ on Y with prefix cache K. 8: for each active block b ∈ Y do 9: Accept masked positions with confidence > τM2T . 10: if the previous active block is semi-complete and no token is accepted then 11: Accept the highest-confidence masked position. 12: end if 13: if b is fully decoded then 14: Mark b as to-cache. 15: end if 16: end for // Commit completed prefix blocks 17: while the front block of Y is to-cache do 18: Write the front block into K and remove it from Y. 19: end while 20: end while 21: return generated tokens. C.1 A dynamic running-set prevents static-shape execution. A direct implementation of MultiBD maintains a dynamic running-set in addition to the committed prefix cache. When the latest active block reaches an add-block threshold, the decoder appends a fully masked future block to the running-set. When the front active block is completed, the decoder writes it into the KV cache and removes it from the running-set. This dynamic procedure exposes inter-block parallelism, but the number of active tokens changes across decoding steps and across requests. As shown in Figure 6a(2), such shape variation is unfriendly to CUDA Graph capture and replay. C.2 A fixed Block Buffer implements MultiBD states. Optimized MultiBD replaces dynamic appending with a fixed-size Block Buffer. The Block Buffer contains Nbuf physical block slots. At each decoding step, active slots represent the logical running-set Rs , while dummy slots reserve capacity for future blocks. Adding a future block therefore activates an existing dummy slot instead of extending the physical input sequence. When the front active block is completed, it is committed to the KV cache, removed from the running-set, and the Block Buffer slides forward by replacing the consumed slot with a new dummy slot at the tail. This realizes MultiBD while keeping the number of processed buffer tokens fixed at Nbuf · B. 1. Traditional Block-wise dLLM: Block Diffusion Req1 Req2 Req3 Req4 Req5 D2F training mask: prefix-full vs fully-block Static running window 80 77.60 78 Static Input Slot 2. Our Design: Multi-Block Diffusion 76 Accuracy (%) Req1 Req2 Req3 Req4 Dynamic running window 74 Static Input Slot 72 69.60 3. Using Dummy Block Padding for Static CUDA Graph 70 Req1 Req2 Req3 Req4 Static running window 68 Static Input Slot 66 Prefix-full Block Causal Fully Block Causal (a) CUDA Graph compatibility across decoding designs. (b) Making D2F fully block-causal hurts accuracy. Prefix- (1) SingleBD uses a fixed single active block but ex- full attention gives D2F stronger noisy-prefix visibility poses no inter-block parallelism. (2) Naive MultiBD ap- but is not naturally compatible with prefix KV caching. pends future blocks dynamically, making the running-set Directly replacing it with a fully block-causal mask length change over time. (3) Optimized MultiBD maps improves cache compatibility but drops accuracy from the logical running-set into a fixed-size Block Buffer with 77.60% to 69.60%. dummy slots, keeping tensor shapes static for CUDA Graph capture and replay. Figure 6 Static-shape execution and prefix-cache compatibility analyses. Left: optimized MultiBD keeps tensor shapes static through a fixed-size Block Buffer, enabling CUDA Graph capture and replay. Right: making D2F fully block- causal improves cache compatibility but substantially hurts accuracy. C.3 Block states advance the fixed Block Buffer. Each physical slot in the Block Buffer follows the transition dummy → active → to-cache → in-cache. A dummy slot is an idle placeholder that preserves the static buffer shape. An active slot participates in the current MultiBD forward pass. A to-cache block has completed decoding and is ready to be committed. An in-cache block has been written into the prefix KV cache and no longer belongs to the active part of the running-set. These state transitions implement the logical evolution of Rs without changing the physical input shape. C.4 Thresholds control activation and token updates. MultiBD uses separate thresholds for block activation, fallback progress, and token updates. The add-block threshold τadd controls when a future block can enter the fixed Block Buffer. The stability threshold τstable prevents premature activation when the current latest active block is still unstable. The semi-completion threshold τsemi allows later active blocks to use the top-1 context of a preceding block once it has made sufficient progress, even before it is fully cached. The M2T threshold τM2T controls mask-to-token acceptance, and the optional T2T threshold τT2T controls token-to-token revision for models that support T2T updates. This separation is important because M2T and T2T updates have different reliability profiles. M2T introduces new content into an active block, while T2T overwrites tentative content before commitment. Using separate thresholds stabilizes concurrent block refinement and reduces error propagation across the running-set. Algorithm 5 Optimized MultiBD with a Fixed Block Buffer Require: Model θ; block size B; buffer size Nbuf ; thresholds τadd , τsemi , τstable , τM2T , and optional τT2T . // Initialize fixed Block Buffer 1: Initialize prefix KV cache K and a fixed Block Buffer W with Nbuf slots. 2: Set W[0] to a fully masked active block and all remaining slots to dummy. 3: while generation is not complete do // Activate future blocks without changing shape 4: Let R be the non-dummy resident blocks in W. 5: Let blast be the last active block in R. 6: if blast satisfies progress > τadd and stability > τstable then 7: Activate the first trailing dummy slot if one exists. 8: end if // Decode the static Block Buffer 9: Run θ on the static Nbuf · B Block Buffer tokens with prefix cache K. 10: for each active block b ∈ W do 11: Accept masked positions with confidence > τM2T . 12: if no masked position is accepted and the preceding active block is semi-complete then 13: Accept the highest-confidence masked position in b. 14: end if 15: if T2T revision is enabled then 16: Revise eligible filled but uncommitted positions with confidence > τT2T . 17: end if 18: if b is complete and all preceding resident blocks are cached or ready-to-cache then 19: Mark b as to-cache. 20: end if 21: end for // Commit prefix blocks and slide the buffer 22: while the front slot of W is to-cache do 23: Write the front block into K; its state becomes in-cache. 24: Pop the front slot and append a new dummy slot at the tail. 25: end while 26: end while 27: return generated tokens. C.5 Prefix Caching and Fully Block-Causal D2F Prefix caching is a key advantage of BD-LMs. In SingleBD, Native D2F is not directly prefix-cache compatible. completed blocks form an immutable clean prefix, so their KV states can be stored and directly reused in later steps. As shown in Figure 7(1), only the current noisy block requires repeated computation. By contrast, native D2F uses prefix-full attention. Future noisy blocks condition on a prefix-full context, where prefix states are not organized as immutable block-causal prefix pages in the standard BD-LM cache. As illustrated in Figure 7(2), their KV states cannot be reused in the same way as SingleBD prefix blocks. Fully block-causal D2F variant.To isolate the prefix-caching issue, we construct a fully block-causal D2F vari- ant. Let the full clean sequence be partitioned into BD-LM blocks: x0 = [b1 , . . . , bK ], bk ∈ V B . Suppose native D2F uses a token-level clean prefix xpre 1 P 0 = (x0 , . . . , x0 ), where P can be arbitrary and need not be divisible by B. Let   P a= + 1, r = P − (a − 1)B B denote the first block that contains suffix tokens and the number of prefix tokens inside this boundary block, respectively. Then b1 , . . . , ba−1 are complete clean prefix blocks, while ba may contain both prefix tokens and suffix tokens. We use ba as the first noisy block of the D2F-style suffix, rather than inserting padding tokens. For the boundary block, only its suffix positions are noised and included in the loss: Ia = {r + 1, . . . , B}. 1. SingleBD enables Prefix Caching 2. D2F-style prefix-full attention 3. How BD-LM combines breaks Prefix Caching PagedAttention + Prefix Caching You are a helpful assistant. Requests (Block Size = PageSize) Page Table / Physical Page Pool User: Hello! (Without Prefix Caching) Assistant: Hi, what can I help you? Req1 0 1 2 9 0 1 2 3 4 5 Req1 0 1 2 3 4 5 0 1 2 3 Req2 Req2 3 4 10 11 15 16 Prefix (System Prompt + 6 7 8 9 10 11 User: What is Diffusion LLM? Assistant: Diffusion LLM (dLLM) ... User Prompt) Req3 5 6 12 14 17 12 13 14 15 16 17 Req4 7 8 13 0 1 2 Same logical prefix pages map to the same physical pages Add Prefix Caching User: What is PagedAttention? Assistant: PagedAttention is the core ... Add Prefix Caching Req1 0 1 2 3 0 1 2 3 4 5 Prefix Full Req2 0 1 4 5 6 9 User: Then how Prefix Caching work with 6 7 8 9 10 11 Prefix blocks attend Paged Attention? ✕ Req3 0 1 7 8 11 Assistant: Prefix Caching is a ... bidirectionally 12 13 14 15 16 17 Req4 0 1 10 Cached KV / pages are ✓ BlockSize = PageSize ✕ not reusable Block Causal as a stable causal prefix ✓ Paged attention manages page-level KV storage ✓ Shared prefix pages can be directly reused Prefix caching is not ✓ Prefix caching reuses shared prefix pages ✕ ✓ Only new suffix blocks need computation naturally supported ✓ New pages are appended only for uncached suffix blocks ✓ Prefix cache hit on shared pages ✕ Cannot directly reuse prefix cache ✓ Efficient reuse for BD-LMs Figure 7 Prefix caching in block-causal BD-LMs. (1) SingleBD keeps completed blocks as an immutable clean prefix, enabling direct KV-cache reuse. (2) D2F-style prefix-full attention breaks this cache semantics because noisy prefix blocks are not reusable as stable causal prefix pages. (3) Block Buffer MultiBD separates cached prefix blocks from active Block Buffer slots, enabling prefix KV reuse while refining multiple active blocks. For later blocks j > a, all positions belong to the suffix: Ij = {1, . . . , B}. We then apply a monotonic D2F-style noise-scheduler to the valid suffix positions of blocks a, . . . , K: 0 ≤ ta < ta+1 < · · · < tK ≤ 1. Let b̄j,tj denote the partially corrupted block, where positions in Ij are corrupted by qtj (· | bj ) and positions outside Ij are kept clean. For the boundary block, this means that the prefix part of ba remains clean, while the suffix part is noised. The fully block-causal D2F variant factorizes the suffix as K (