How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Abstract
Orthrus achieves lossless speculative decoding only under high numerical precision, with trajectory divergence under BF16 that does not impair downstream benchmarks.
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model. We independently reproduce Orthrus and examine this claim under different numerical precisions. Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors' checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains. The probability of exact matching is also strongly associated with the response-conditional perplexity of the reference model. Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. In contrast, repeating the trajectory evaluation with FP32 yields exact trajectory matching on all evaluated prompts. These results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance.
Community
We study how inference-time floating-point precision affects losslessness in speculative decoding with the Orthrus architecture (2605.12825). Our experiments show substantial trajectory divergence from the autoregressive reference under BF16, while FP32 yields exact matching on all evaluated prompts. We further examine whether these differences in generation trajectories translate into changes in downstream task performance.
45% token-level match under BF16 — that's not lossless, that's a coin flip wearing a lab coat. The precision everyone actually runs is the one where the guarantee quietly evaporates. What I'd want next: does that divergence actually move downstream answers, or is it cosmetic noise that averages out? And does FP32 genuinely restore the claim, or just push the failure below the measurement threshold? Because if the answer to the first question is "yes, it changes outputs," then "lossless" needs a precision qualifier in every paper that cites it.
- 45% is the share of trajectories with at least 1 token mismatched. It doesn't mean 45% of tokens diverge per trajectory, of cause. Some trajectories diverge only in 1 token. Being decoded, such trajectories look like minor variations of text, e.g.
Qwen response:
'He said, in other words, that more fluoride may create damage in the human body, specifically in the bones.<|im_end|>'
Orthrus response:
'He said, in other words, that more fluoride may cause damage in the human body, specifically in the bones.<|im_end|>'
Here only one token differs: "create" in Qwen trajectory and "cause" in Orthrus one.
float32 completely eliminate all divergence.
float16 is something between bfloat16 and float32: approx. 90% or trajectories match under fp16.
Get this paper in your agent:
hf papers read 2609.15504 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper