Inferact

/

Kimi-K3-DSpark

like 55

Follow

Inferact Inc. 100

Hi, thank you for releasing Kimi-K3-DSpark.

We tested the public Inferact/Kimi-K3-DSpark checkpoint at revision
cf6b824 on a 16xH200 deployment. We consistently observe a sharp speculative
acceptance cliff between approximately 107K and 196K prompt tokens.

This is not intended as a direct reproduction of the model-card benchmark. We
would like to understand whether this is an expected limitation of the current
checkpoint, a runtime configuration issue, or a gap that may be addressed by
future long-context draft-model training.

Configuration

• Hardware: 16x NVIDIA H200 across two nodes
• Parallelism: PP2 x TP8
• Base model: Kimi K3
• Draft checkpoint: Inferact/Kimi-K3-DSpark, revision cf6b824
• vLLM: based on the K3 PP/DSpark implementation from vLLM PR #50514, with
  local H200 compatibility patches
• DSpark: seven speculative tokens, probabilistic draft sampling, block
  rejection, and draft top-k 512
• Attention backend: FlashMLA
• KV cache: FP8
• Sampling: temperature=1.0, top_p=1.0
• Output: exactly 512 tokens, with EOS ignored

The prompts contain low-entropy coding and code-review text derived from
SPEED-Bench. Four coding examples are concatenated and the repository context
is repeated to reach each target length. Each request has a unique prefix to
avoid reusing an existing prefix-cache entry.

This is an artificial long-context construction, but the repeated,
low-entropy input should normally be favorable to speculative decoding rather
than adversarial to it.

Acceptance results

Draft acceptance is accepted draft tokens divided by proposed draft tokens.
Accepted tokens per draft step is accepted draft tokens divided by the
number of draft iterations. The table below uses concurrency one.

Actual prompt tokens	Draft acceptance	Accepted tokens per draft step	Decode throughput
7,161	39.42%	2.759	154.74 tok/s
27,067	36.90%	2.583	143.81 tok/s
107,336	30.31%	2.122	122.22 tok/s
195,985	2.27%	0.159	44.17 tok/s
228,708	0.86%	0.060	40.43 tok/s
260,958	1.35%	0.094	41.42 tok/s
392,214	0.77%	0.054	37.97 tok/s

The clearest transition is:

107K tokens: 30.31% acceptance
196K tokens:  2.27% acceptance
229K tokens:  0.86% acceptance

The same behavior appears at concurrency 2, 4, and 8. At approximately 196K
tokens, acceptance was 2.27%, 2.17%, 2.08%, and 2.15% for C1, C2, C4, and C8,
respectively. This suggests that the cliff is primarily related to context
length rather than concurrency.

We also reproduced the discontinuity with deterministic temperature=0
sampling. Aggregate acceptance across the tested concurrency groups was
36.30% at 8K, 37.66% at 32K, 26.42% at 128K, 0.94% at 256K, 0.47% at 384K,
and 0.39% at 512K. Therefore, the use of top_p=1.0 in our production-sampling
matrix does not appear to be the sole cause.

Throughput impact

In a separate matched temperature=0 A/B, DSpark was the only intended
model-path difference:

Context	DSpark C1 decode	Target-only C1 decode	DSpark difference
128K	117.07 tok/s	58.61 tok/s	+99.7%
256K	41.18 tok/s	56.27 tok/s	-26.8%
384K	36.12 tok/s	54.35 tok/s	-33.5%
512K	35.66 tok/s	52.71 tok/s	-32.3%

DSpark is therefore highly beneficial at 128K, but the acceptance cliff has a
material end-to-end cost at 256K and longer contexts in this setup.

Questions

1. Is this sharp acceptance drop beyond approximately 128K a known or expected
   limitation of the current checkpoint, and do you have acceptance results
   for prompts in the 128K-256K range?
2. Is there a required long-context setting, such as RoPE/YaRN configuration,
   hidden-state capture layers, draft max_model_len, or a recommended vLLM
   revision, that we should verify?
3. Are there plans to release a future Kimi K3 draft checkpoint trained or
   fine-tuned specifically for 128K-256K contexts? Alternatively, do you plan
   to publish guidance for long-context fine-tuning with TorchSpec?

We would be happy to run a focused reproduction matrix using any recommended
settings and share the resulting acceptance and throughput metrics.

Thanks for the detailed report! We are aware of the issue and will release an updated version of the model weights soon

Hi @dhbloo , off topic but do you mind sharing what you mean by "local H200 compatibility patches"?

Hi @dhbloo , off topic but do you mind sharing what you mean by "local H200 compatibility patches"?

I asked Codex to adapt and tune the deployment for a whole day, which resulted in several local patches. For our two-node, 16×H200 PP2×TP8 setup, the necessary changes were Hopper FlashMLA compatibility for BF16 queries with an FP8 KV cache and, when using FlashMLA for the draft model, the non-causal attention support required by DSpark. The remaining patches (covering the DSpark Markov sampling layout, asynchronous scheduling boundaries, hybrid KV-cache grouping, and SM90 kernel tuning) were correctness, capacity, or performance improvements rather than basic H200 requirements.

@dhbloo Interesting. Do you mind sharing your fork?