Papers
arxiv:2611.21925

DNeo: Your Parallel Draft Model Needs Better Target Feature Encoding

Published on Nov 16
· Submitted by
Jameson Sandler
on Nov 18
Authors:
,
,
,
,
,
,

Abstract

A lightweight feature-encoding method with rejection memory and architectural enhancements enables faster large language model inference by improving speculative-diffusion draft verification.

Speculative-diffusion decoding uses a small parallel drafter model to generate b draft tokens in one forward pass, which then undergo parallel verification via a larger target model. Faster inference can be achieved when multiple draft tokens pass verification simultaneously. Recent parallel draft architectures (DFlash, DSpark) improve draft quality by conditioning on rich features computed by the target, through a mechanism known as feature encoding. However, standard feature encoding mechanisms have three limitations: (1) they are lossy, applying encoding projections that are destructive to the underlying target feature-space via a discovered effect known as residual mixing, (2) they are memory-less, with no record of previously processed proposals, and (3) they utilize features sampled uniformly from across the target layers, omitting important hidden states. To address these issues, we propose Mixture-of-Features Attention (MoFA), a lightweight feature-encoding approach equipped with a working memory that encodes previously rejected proposals. Leveraging MoFA, rejection memory, and a set of further enhancements, including 2-tap causal convolutions, prefix-only cross-attention, and non-uniform target layer selection, the final architecture, DNeo, achieves 1.3x faster generation speed than SoTA (DFlash-2) without impacting drafter footprint or generation quality. DNeo marks a new state of the art in high quality, fast LLM inference.

Community

Paper author Paper submitter

Diffusion drafters need better feature encoding.

We discover that existing feature encoders destroy the feature space of the target model (residual mixing). Next, we show that feature encoders should possess working memory of previous mistakes (rejection memory). We also find that feature encoders should focus on only the most useful target model layers (the latter half).

The proposed architecture DNeo fixes these issues, using Mixture-of-Features Attention (MoFA) as a superior encoding mechanism, alongside other enhancements . DNeo exhibits up to 30% faster inference speeds than SoTA.

Screenshot 2026-11-18 at 12.01.20 PM

Screenshot 2026-11-18 at 12.02.37 PM

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.21925
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 1

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.21925 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.21925 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.