RedHatAI
/

Kimi-K3-speculator.dspark

speculative-decoding

Model card Files Files and versions

shanjiaz commited on 11 days ago

Commit

48beb88

·

verified ·

1 Parent(s): 46264ce

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -143,7 +143,7 @@ Kimi-K3 does not fit on a single GB300 node, so training used a multi-node setup
 ## Deployment
 ```bash
   # Deploy with speculative decoding on vLLM latest main
-  vllm serve moonshotai/Kimi-K3 --spec-method dspark --spec-model RedHatAI/Kimi-K3-speculator.dspark --spec-tokens 8 --trust-remote-code --gpu-memory-utilization 0.95 --tensor-parallel-size 8 --load-format fastsafetensors --no-enable-flashinfer-autotune --max-model-len 131072 --kv-cache-dtype fp8 --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' --no-enable-prefix-caching --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3
 ```
 ## Acceptance Rates
 Per-position token acceptance rates across datasets, at 8 draft tokens per step:

 ## Deployment
 ```bash
   # Deploy with speculative decoding on vLLM latest main
+  vllm serve moonshotai/Kimi-K3 --spec-method dspark --spec-model RedHatAI/Kimi-K3-speculator.dspark --spec-tokens 8 --trust-remote-code --gpu-memory-utilization 0.95 --tensor-parallel-size 8 --load-format fastsafetensors --no-enable-flashinfer-autotune --max-model-len 131072 --kv-cache-dtype fp8 --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' --enable-auto-tool-choice --tool-call-parser kimi_k3 --reasoning-parser kimi_k3
 ```
 ## Acceptance Rates
 Per-position token acceptance rates across datasets, at 8 draft tokens per step: