One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

We are excited to share our ICML 2026 paper on making Asynchronous Pipeline Parallel LLM pretraining work in practice: by benchmarking modern optimizers under one-step delay, we show that staleness is not a fundamental barrier at scale.

Modern large-scale LLM pretraining often relies on Pipeline Parallelism (PP) to distribute computation across GPUs; however, synchronous implementations leave devices idle during pipeline bubbles, wasting computational resources. Async PP eliminates these bubbles and improves throughput, but does so at the cost of gradient staleness. Even schedules such as PipeDream-2BW, which reduce this staleness to a constant one-step delay, have seen limited adoption due to the common belief that optimization under delayed gradients is fundamentally unstable.

We challenge this assumption. We show that gradient delay is not an intrinsic limitation, but strongly depends on optimizer choice. While AdamW is highly sensitive to staleness, modern optimizers like Muon remain highly robust. To further reduce the remaining gap, we derive a lightweight Error-Feedback-inspired correction mechanism, operating on the optimizer step level. We also provide a theoretical analysis showing convergence for delayed Muon both with and without this correction. Finally, we validated this approach at scale: a 10B MoE model trained asynchronously with Error-Feedback correction matches the synchronous baseline exactly.