Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models
Abstract
A training-free, lossless speculative decoding framework exploits idle compute in chunk-interleaved full-duplex speech language models to draft and verify upcoming audio chunks, substantially reducing response and tool-call delays.
Chunk-interleaved full-duplex speech language models listen and speak in fixed time chunks, and each chunk must be processed before the next one arrives. Their workload is asymmetric across chunks: listen chunks leave compute idle while awaiting audio, and speak and tool-call chunks require serial generation, which delays responses and tool calls. Using the idle compute for the next chunk requires decoding it before its audio arrives, under an assumed input. Taking silence as the assumed input and comparing the output with the output under real audio on four models, we observe two properties. First, the next audio chunk does not change the output in 95.3% of chunks. Second, 88% of changes begin at the first output token. Building on these properties, we propose Speculative Listening, a training-free, lossless framework for input-time self-speculative decoding. During idle time, the model drafts the next chunk under silence and, for changes at the first token, a second branch from the runner-up first token. It keeps all drafts in a cross-chunk hypothesis pool and verifies them when the audio arrives. On our eight-task benchmark TickTock-Bench, Speculative Listening speeds up speak chunks by 2.6-2.8x and tool-call chunks by up to 23.4x. On average, it reduces the decoding delay before a response starts by 73%, to 74-241 ms, close to the 200 ms gap between human turns, and the maximum lag behind real time by 94.4%.
Community
Decode Before You Hear: Speculative Listening for Full-Duplex Speech Language Models
Most existing full-duplex speech models alternate between listening and speaking in fixed time chunks, which leaves the load across chunks highly uneven: compute sits idle during "listening", while "speaking" and tool calls miss their deadlines and fall behind because they generate many tokens serially. We observe that with silence as the hypothesized input, the output of 95.3% of chunks is identical to the output produced from the real audio, and 88% of divergences appear at the very first token. Building on this, Speculative Listening uses the idle compute of listening to draft the next chunk before the real audio arrives, using silence and the runner-up first-token candidate. All drafts are then verified together once the audio arrives, which requires no additional training and guarantees lossless output.
Speed: On TickTock-Bench, the eight-task evaluation suite we built, speaking chunks are accelerated by 2.6–2.8×, and tool-call chunks by up to 23.4×.
Latency: Decoding latency before a response begins drops by 73% on average, to 74–241 ms, close to the 200 ms turn-taking gap in human conversation. The maximum lag behind real time is reduced by 94.4%.
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper