cloudyu/DeepSeek-V4-Flash-0731-4Experts

DeepSeek-V4-Flash-0731

---

Technical Report👁️

Note (this fork). This is not a new or re-trained model. The weights are the official DeepSeek-V4-Flash-0731, byte-for-byte unchanged. The only modification is setting num_experts_per_tok from 6 to 4 in config.json (plus a one-line vLLM weight-loading shim to make it load). Everything else added here is analysis: a top_k=4 vs top_k=6 accuracy/speed comparison and the statistical evidence behind recommending top_k=4. See Expert Routing: top_k=4 vs top_k=6.

Introduction

DeepSeek-V4-Flash-0731 is the official release of DeepSeek-V4-Flash, superseding the preview version, with substantially enhanced agentic capabilities. It has the same model structure as DeepSeek-V4-Flash-DSpark, i.e. it comes with a speculative decoding module attached.

DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on benchmarks listed below despite its far smaller activated parameter count, and is broadly competitive with the strongest proprietary models available.

Benchmark	DeepSeek-V4-Flash-0731	DeepSeek-V4-Flash (Preview)	DeepSeek-V4-Pro (Preview)	GLM-5.2	Opus-4.8
Terminal Bench 2.1	82.7	61.8	72.1	81.0	85.0
NL2Repo	54.2	39.4	38.5	48.9	69.7
Cybergym	76.7	38.7	52.7	-	83.1
DeepSWE	54.4	7.3	12.8	46.2	58.0
Toolathlon-Verified	70.3	49.7	55.9	59.9	76.2
Agents' Last Exam	25.2	15.8	16.5	23.8	25.7
AutomationBench Public	25.1	10.8	12.8	12.9	27.2
DSBench-FullStack †	68.7	37.0	41.8	61.8	71.6
DSBench-Hard †	59.6	25.8	31.1	54.5	71.7

Notes:

1. For the Code Agent tasks among the public benchmarks above, DeepSeek-V4-Flash-0731 is evaluated with the minimal mode of DeepSeek Harness (to be released) as the agent framework, using the max reasoning effort level with temperature = 1.0, top_p = 0.95.
2. † DSBench-FullStack is an internal full-stack development test set; DSBench-Hard is an internal test set of difficult coding-agent problems.

Chat Template

This release does not include a Jinja-format chat template. Instead, we provide a dedicated encoding folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the encoding folder for full documentation.

The reasoning_effort parameter now supports three levels — low, high, and max — which control how much deliberation the model spends before answering.

A brief example:

from encoding_dsv4 import encode_messages, parse_message_from_completion_text

messages = [
    {"role": "user", "content": "hello"},
    {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
    {"role": "user", "content": "1+1=?"}
]

# messages -> string
prompt = encode_messages(messages, thinking_mode="thinking", reasoning_effort="max")

# string -> tokens
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
tokens = tokenizer.encode(prompt)

How to Run with vLLM

DSpark speculative decoding is enabled with a single flag — add --speculative-config with method: dspark to your vLLM launch command:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

For example, the command below serves the model with vLLM on a single 4×GB300 node. See the vLLM recipe for detailed instructions and other hardware configurations.

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

Expert Routing: top_k=4 vs top_k=6 (Recommended: top_k=4)

DeepSeek-V4-Flash-0731 ships with num_experts_per_tok=6 (6 of 256 routed experts activated per token, 13B active params). We additionally evaluated num_experts_per_tok=4 (11B active params) and recommend top_k=4 as the default: it is measurably faster at statistically indistinguishable accuracy.

Why top_k=4

1. ~15% fewer activated parameters, for free. Routing to 4 experts instead of 6 drops per-token active params from ~13B to ~11B. Shared experts and attention are unchanged, so quality is preserved (see measurements below).
2. Faster inference (~13–18%). Fewer expert FFN computations plus less gather/scatter and softmax overhead in the router. Measured end-to-end: HumanEval wall-time ~15% lower, per-token generation ~13% faster.
3. Power-of-2 dispatch alignment. 6 is not a power of two; MoE dispatch, warp scheduling, and memory alignment on the GPU are more efficient when the expert count aligns to a power of two (4), improving tensor-core utilization for the dispatch/combine shapes.
4. No accuracy regression. On our internal SWE-bench-Lite and HumanEval runs the difference between top_k=4 and top_k=6 is within run-to-run noise (details below).

Single-GPU test environment (used for the numbers below)

The measurements were produced on a single NVIDIA B300 (SXM6, ~275 GiB) — no data/expert parallelism and DSpark speculative decoding disabled (the official multi-GPU launch in How to Run with vLLM enables DSpark; that is orthogonal to the routing comparison here). Exact launch command:

CUDA_HOME=$HOME/.local/lib/python3.11/site-packages/nvidia/cu13 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
CUDA_VISIBLE_DEVICES=0 \
vllm serve /path/to/DeepSeek-V4-Flash-0731 \
  --served-model-name dsv4 --port 18002 \
  --trust-remote-code --kv-cache-dtype fp8 \
  --max-model-len 32768 --gpu-memory-utilization 0.85 \
  --reasoning-parser deepseek_v4 --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice

Notes for single-GPU:

• No --speculative-config — DSpark is left off, so throughput numbers reflect the base model. (DSpark is a decode-speed optimization and does not change which tasks pass; it can be re-enabled independently.)
• --max-model-len 32768 fits the KV cache comfortably in 275 GiB alongside the fp8+MXFP4 weights; raise it if you have headroom.
• The tid2eid shim from How to enable top_k=4 must be applied before serving with num_experts_per_tok=4.
• Weight load takes ~6–9 min (fp8/fp4 MoE autotuning on first start).

Test harnesses: HumanEval via /v1/chat/completions (code-fence extraction, temperature=0.1); SWE-bench-Lite via the mini-swe-agent minimal text-based agent (no Docker; each repo in an isolated uv venv), with the official code-agent sampling temperature=1.0, top_p=0.95.

Measured accuracy — the difference is within noise

All numbers below are from the single-B300 setup above on the native fp8+MXFP4 weights.

HumanEval (pass@1, 164 problems, thinking mode, temperature=0.1):

Config	Pass@1	Note
top_k=4	92.1% / 91.5% / 92.1% (3 reps: 151 / 150 / 151)	run-to-run spread ±1 problem
top_k=6	90.9% (149)	within the ±1-problem noise band

SWE-bench-Lite (n=86 subset, mini-swe-agent harness, no Docker, official code-agent sampling temperature=1.0, top_p=0.95 unless noted):

Config	Resolved	Rate	Sampling
top_k=6	37/86	43.0%	default
top_k=4 (rep 1)	38/86	44.2%	default
top_k=4 (rep 2)	38/86	44.2%	default
top_k=4 (official)	39/86	45.3%	t=1.0, p=0.95

Two-proportion z-test top_k=4 vs top_k=6: z ≈ 0.15–0.31 (not significant). Repeating the same top_k=4 config flips ~14–17 of the 86 instances per pair of runs (≈31% of the ever-solved union) purely from MoE-routing / fp8-kernel / batching non-determinism. Aggregated over 4 runs: 23 instances always pass (stable core), 37 always fail, and 26 are coin-flips. In other words, the per-task differences between top_k=4 and top_k=6 are the same magnitude as top_k=4 versus itself — i.e. noise, not a capability gap. top_k=4 gets the speed and parameter savings at no measurable accuracy cost.

On knowledge-heavy multiple-choice benchmarks (e.g. MMLU-Pro) narrower routing can even help slightly; on code generation top_k=6 may hold a fraction of a point. Both directions are inside the noise band on our runs — treat them as equivalent in quality.

How to enable top_k=4

Step 1 — set the config. In config.json:

"num_experts_per_tok": 4

Step 2 — patch vLLM weight loading (required). The checkpoint's tid2eid tensor (the hash-based expert-routing lookup table) was trained at top_k=6, so it has shape [vocab_size, 6]. With num_experts_per_tok=4 the model allocates a [vocab_size, 4] parameter, and loading fails with:

AssertionError: Attempted to load weight (torch.Size([129280, 6]))
into parameter (torch.Size([129280, 4]))

Fix it by slicing the checkpoint tensor to the first top_k columns during load. In vllm/models/deepseek_v4/nvidia/model.py, inside load_weights, in the final else branch just before the weight_loader(param, loaded_weight) call:

param = params_dict[name]
# top_k override: checkpoint's tid2eid is [vocab, 6] (trained at top_k=6);
# slice to the config's top_k columns so it matches the allocated parameter.
if "tid2eid" in name and loaded_weight.shape != param.shape:
    loaded_weight = loaded_weight[:, :param.shape[1]].contiguous()
weight_loader = getattr(param, "weight_loader", default_weight_loader)
weight_loader(param, loaded_weight)

The [:, :param.shape[1]] slice keeps the highest-priority expert columns and is a no-op when the shapes already match (top_k=6), so the patch is safe to leave in place for both configurations. No other weights change — total parameters, routing method (noaux_tc), shared experts, and attention are all identical.

Note: this is a weight-loading shim, not a re-training of the router table. The hash table's remaining 4 columns are the same top entries used at top_k=6, which is why accuracy is preserved.

How to Run Locally

Please refer to the inference folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos.

For local deployment, we recommend setting the sampling parameters to temperature = 1.0, with top_p = 0.95 for agentic scenarios and top_p = 1.0 otherwise. For the high and max reasoning effort levels, we recommend a maximum output length of 384K tokens.

License

This repository and the model weights are licensed under the MIT License.

Citation

@misc{deepseekai2026deepseekv4,
      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
      author={DeepSeek-AI},
      year={2026},
}