Yet to see any gains from DFlash
Both DFlash (3.6) and DFlash 2 (3.8) are slower than MTP on the 2 machines that I've tested.
Can't say I've seen anyone get these improvements either. DFlash gains seem very not true.
on Coding?
I have tested 27B NVFP4 + MTP (fp8 kv), FP8 + MTP (=5 and =7, fp8 kv and full precision), FP8 + DFlash 2 (=7, fp8) on my DGX Spark, in hermes and pi, the best acceptance length at 100k tokens generated was 2.8 using dflash 2, followed by 2.2 using fp8+mtp=5 at full precision kv and last was NVFP4 + MTP at 1.9. Tasks were mostly agentic (set up dockers, manage services through ssh) and some coding (python, typescript).
In my setup, the gains are there. My only issue is that it uses more vram than MTP, however at gpu util 0.85 i get 5.15x max ctx with the dflash 2 =7 fp8 kv config, which is enough for my needs (as even with 8 concurrent, i've yet to get over 1M total KV in use).
on Coding?
Mixed tasks.
I have tested 27B NVFP4 + MTP (fp8 kv), FP8 + MTP (=5 and =7, fp8 kv and full precision), FP8 + DFlash 2 (=7, fp8) on my DGX Spark, in hermes and pi, the best acceptance length at 100k tokens generated was 2.8 using dflash 2, followed by 2.2 using fp8+mtp=5 at full precision kv and last was NVFP4 + MTP at 1.9. Tasks were mostly agentic (set up dockers, manage services through ssh) and some coding (python, typescript).
In my setup, the gains are there. My only issue is that it uses more vram than MTP, however at gpu util 0.85 i get 5.15x max ctx with the dflash 2 =7 fp8 kv config, which is enough for my needs (as even with 8 concurrent, i've yet to get over 1M total KV in use).
I genuinely wanted someone confirm if the performance benefit is only seen on NVIDIA hardware. Since that's the only hardware I see people reproducing the results from the authors. And if so, they really should add that information to the Model Card.
I also saw reports of people with NV GPUs that still have slow performance and memory usage explosions with DFlash as well. So this is truly puzzling to me. If anyone can provide this information in an auditable way, I'd really appreciate it.
There should be improvements on M5 Pro/Max and nothing else by Apple, as, to my understanding, to a large degree the benefit of speculative decoding is utilizing the high prefill (compute-bound) to accept/drop cheaply decoded tokens (bandwidth-bound) from a draft model. For all the amazing things in M chips, especially their bandwidth, until the M5 generation there were no dedicated tensor cores, making compute lag behind dedicated GPUs, even if the bandwidth is amazing. It's my main gripe with my M4 Max Studio.
I can't speak for AMD and Intel cards, I think some of them should also have a speedup, but given that I'm only seeing it on my Spark (NVIDIA) and in the blog they use a M5 Max, those are the 2 platforms I'm reasonably certain an improvement should be seen. I'd like for someone to provide numbers from AMD/Intel cards too as I'm curious what the situation there is.
To provide a sample image from my Spark's dashboard, for context, at 4 agents it evens out around 27-30 t/s aggregate during mixed loads. The main issue is when one is prefilling, the other 3's generation tanks until that is done. While everyone is decoding, very high numbers can be seen like in the image, which was a snapshot of when along the 4 agents, i ran 4 concurrent benchmarks (which were just decode) and in moments when all 8 were decoding, peaks like this were seen.
Both DFlash (3.6) and DFlash 2 (3.8) are slower than MTP on the 2 machines that I've tested.
Can't say I've seen anyone get these improvements either. DFlash gains seem very not true.
this is a v2 Dflash model, did you use it with sglang ? there were commits added on 8-18-26 that are required. I can see a increase vs the dspark I was running in sglang most notably in the thinking where it was the slowest.
Both DFlash (3.6) and DFlash 2 (3.8) are slower than MTP on the 2 machines that I've tested.
Can't say I've seen anyone get these improvements either. DFlash gains seem very not true.this is a v2 Dflash model, did you use it with sglang ? there were commits added on 8-18-26 that are required. I can see a increase vs the dspark I was running in sglang most notably in the thinking where it was the slowest.
this was on tp2 ada6000s running the official fp8 release
Im not sure what the community complains about mtp would never touch this on 8bit qwen 27b 3.8 dense
im happy with dflash 2 great job guys
Decode batch, #running-req: 1, #full token: 6406, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 5.55, accept rate: 0.33, cuda graph: True, gen throughput (token/s): 203.94, #queue-req: 0
[2026-08-20 00:42:58] Decode batch, #running-req: 1, #full token: 6741, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.57, accept rate: 0.54, cuda graph: True, gen throughput (token/s): 316.80, #queue-req: 0
[2026-08-20 00:42:59] Decode batch, #running-req: 1, #full token: 7024, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 6.72, accept rate: 0.41, cuda graph: True, gen throughput (token/s): 246.51, #queue-req: 0
[2026-08-20 00:43:00] Decode batch, #running-req: 1, #full token: 7306, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.15, accept rate: 0.44, cuda graph: True, gen throughput (token/s): 263.82, #queue-req: 0
[2026-08-20 00:43:01] Decode batch, #running-req: 1, #full token: 7536, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 5.70, accept rate: 0.34, cuda graph: True, gen throughput (token/s): 209.13, #queue-req: 0
[2026-08-20 00:43:02] Decode batch, #running-req: 1, #full token: 7871, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.68, accept rate: 0.55, cuda graph: True, gen throughput (token/s): 318.05, #queue-req: 0
[2026-08-20 00:43:03] Decode batch, #running-req: 1, #full token: 8171, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.25, accept rate: 0.45, cuda graph: True, gen throughput (token/s): 265.86, #queue-req: 0
[2026-08-20 00:43:04] Decode batch, #running-req: 1, #full token: 8471, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.53, accept rate: 0.47, cuda graph: True, gen throughput (token/s): 276.06, #queue-req: 0
[2026-08-20 00:43:05] Decode batch, #running-req: 1, #full token: 8733, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 6.78, accept rate: 0.41, cuda graph: True, gen throughput (token/s): 249.33, #queue-req: 0
[2026-08-20 00:43:06] Decode batch, #running-req: 1, #full token: 9024, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 6.92, accept rate: 0.42, cuda graph: True, gen throughput (token/s): 253.79, #queue-req: 0
[2026-08-20 00:43:07] Decode batch, #running-req: 1, #full token: 9334, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.07, accept rate: 0.51, cuda graph: True, gen throughput (token/s): 296.00, #queue-req: 0
[2026-08-20 00:43:08] Decode batch, #running-req: 1, #full token: 9627, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.03, accept rate: 0.43, cuda graph: True, gen throughput (token/s): 259.28, #queue-req: 0
[2026-08-20 00:43:10] Decode batch, #running-req: 1, #full token: 9950, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.40, accept rate: 0.53, cuda graph: True, gen throughput (token/s): 311.18, #queue-req: 0
[2026-08-20 00:43:11] Decode batch, #running-req: 1, #full token: 10247, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.42, accept rate: 0.46, cuda graph: True, gen throughput (token/s): 275.15, #queue-req: 0
[2026-08-20 00:43:12] Decode batch, #running-req: 1, #full token: 10630, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 9.50, accept rate: 0.61, cuda graph: True, gen throughput (token/s): 352.51, #queue-req: 0
[2026-08-20 00:43:13] Decode batch, #running-req: 1, #full token: 11085, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 11.35, accept rate: 0.74, cuda graph: True, gen throughput (token/s): 418.37, #queue-req: 0
[2026-08-20 00:43:14] Decode batch, #running-req: 1, #full token: 11397, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.58, accept rate: 0.47, cuda graph: True, gen throughput (token/s): 279.07, #queue-req: 0
[2026-08-20 00:43:15] Decode batch, #running-req: 1, #full token: 11688, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.50, accept rate: 0.46, cuda graph: True, gen throughput (token/s): 276.18, #queue-req: 0
[2026-08-20 00:43:16] Decode batch, #running-req: 1, #full token: 11989, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.30, accept rate: 0.45, cuda graph: True, gen throughput (token/s): 267.96, #queue-req: 0
[2026-08-20 00:43:17] Decode batch, #running-req: 1, #full token: 12288, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.53, accept rate: 0.47, cuda graph: True, gen throughput (token/s): 274.22, #queue-req: 0
[2026-08-20 00:43:18] Decode batch, #running-req: 1, #full token: 12614, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.12, accept rate: 0.51, cuda graph: True, gen throughput (token/s): 296.19, #queue-req: 0
[2026-08-20 00:43:19] Decode batch, #running-req: 1, #full token: 12913, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.45, accept rate: 0.46, cuda graph: True, gen throughput (token/s): 271.83, #queue-req: 0
[2026-08-20 00:43:20] Decode batch, #running-req: 1, #full token: 13297, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 9.85, accept rate: 0.63, cuda graph: True, gen throughput (token/s): 359.31, #queue-req: 0
[2026-08-20 00:43:22] Decode batch, #running-req: 1, #full token: 13707, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 10.18, accept rate: 0.66, cuda graph: True, gen throughput (token/s): 372.71, #queue-req: 0
[2026-08-20 00:43:23] Decode batch, #running-req: 1, #full token: 14062, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 8.68, accept rate: 0.55, cuda graph: True, gen throughput (token/s): 315.14, #queue-req: 0
[2026-08-20 00:43:24] Decode batch, #running-req: 1, #full token: 14376, full token usage: 0.01, mamba num: 4, mamba usage: 0.05, accept len: 7.88, accept rate: 0.49, cuda graph: True, gen throughput (token/s): 286.31, #queue-req: 0
accept len: 8.12,
what have you set the draft length to? have you changed any of the default sampling parameters? I'm running it at 7 and see 4s and 5s relatively often, but outside very short coding bursts, I haven't seen them in a row like this
TPS maxing with super long draft length on decoding simple predictable stuff is not useful contribution to the conversation. I too can artificially select the peak performance of running DFlash to make it look incredible. In reality though performance uplift is very uneven. The biggest benefit is, as people pointed out, high prefill and certain code and routine agentic turns where there is heavy predictable json and other kinds of boilerplate code. For all that you sacrifice vision, and take a bit of a VRAM hit compared to MTP.
I still choose to use it because 1) I use Nvidia, and 2) it offers me real time saving on the kinds of tasks I run. I suggest people avoid making black and white conclusions and test things out and honestly try to make tools work for yourselves. After all if you miss out on performance, it's your own time that's wasted.
I doubled the token throughput here. I was using two 16GB 5060 Tis and one 4060 for image generation.
I offloaded some of the experts to the 4060. The speed went from 35 tk/s to 72 tk/s.
--ctx-size 491520 --parallel 3 = 163.840/slot
Text Dflash ~72 t/s vs MTP ~35
4060 7,5 / 8,2 GB
5060 15,7 / 16,3 GB
5060 13,4 / 16,3 GB
I might be able to get more context, but these contexts worked well and safely.