README.md

---
license: mit
library_name: transformers
---
# DeepSeek-V4-Flash-0731

<!-- markdownlint-disable first-line-h1 -->
<!-- markdownlint-disable html -->
<!-- markdownlint-disable no-duplicate-header -->

<div align="center">
  <img src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/logo.svg?raw=true" width="60%" alt="DeepSeek-V4" />
</div>
<hr>
<div align="center" style="line-height: 1;">
  <a href="https://www.deepseek.com/" target="_blank" style="margin: 2px;">
    <img alt="Homepage" src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/badge.svg?raw=true" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://chat.deepseek.com/" target="_blank" style="margin: 2px;">
    <img alt="Chat" src="https://img.shields.io/badge/🤖%20Chat-DeepSeek%20V4-536af5?color=536af5&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
</div>
<div align="center" style="line-height: 1;">
  <a href="https://huggingface.co/deepseek-ai" target="_blank" style="margin: 2px;">
    <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://twitter.com/deepseek_ai" target="_blank" style="margin: 2px;">
    <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-deepseek_ai-white?logo=x&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
</div>
<div align="center" style="line-height: 1;">
  <a href="LICENSE" style="margin: 2px;">
    <img alt="License" src="https://img.shields.io/badge/License-MIT-f5de53?&color=f5de53" style="display: inline-block; vertical-align: middle;"/>
  </a>
</div>

<p align="center">
  <a href="https://arxiv.org/abs/2606.19348"><b>Technical Report</b>👁️</a>
</p>

> **Note (this fork).** This is **not** a new or re-trained model. The weights are
> the official **DeepSeek-V4-Flash-0731**, byte-for-byte unchanged. The **only**
> modification is setting `num_experts_per_tok` from `6` to **`4`** in `config.json`
> (plus a one-line vLLM weight-loading shim to make it load). Everything else added
> here is **analysis**: a `top_k=4` vs `top_k=6` accuracy/speed comparison and the
> statistical evidence behind recommending `top_k=4`. See
> [Expert Routing: `top_k=4` vs `top_k=6`](#expert-routing-top_k4-vs-top_k6-recommended-top_k4).

## Introduction

**DeepSeek-V4-Flash-0731** is the official release of **DeepSeek-V4-Flash**, superseding the preview version, with substantially enhanced agentic capabilities. It has the same model structure as [DeepSeek-V4-Flash-DSpark](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark), i.e. it comes with a speculative decoding module attached.

DeepSeek-V4-Flash-0731 outperforms DeepSeek-V4-Pro (Preview) on benchmarks listed below despite its far smaller activated parameter count, and is broadly competitive with the strongest proprietary models available.

<div align="center">

| Benchmark | DeepSeek-V4-Flash-0731 | DeepSeek-V4-Flash (Preview) | DeepSeek-V4-Pro (Preview) | GLM-5.2 | Opus-4.8 |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Terminal Bench 2.1 | 82.7 | 61.8 | 72.1 | 81.0 | 85.0 |
| NL2Repo | 54.2 | 39.4 | 38.5 | 48.9 | 69.7 |
| Cybergym | 76.7 | 38.7 | 52.7 | - | 83.1 |
| DeepSWE | 54.4 | 7.3 | 12.8 | 46.2 | 58.0 |
| Toolathlon-Verified | 70.3 | 49.7 | 55.9 | 59.9 | 76.2 |
| Agents' Last Exam | 25.2 | 15.8 | 16.5 | 23.8 | 25.7 |
| AutomationBench Public | 25.1 | 10.8 | 12.8 | 12.9 | 27.2 |
| DSBench-FullStack † | 68.7 | 37.0 | 41.8 | 61.8 | 71.6 |
| DSBench-Hard † | 59.6 | 25.8 | 31.1 | 54.5 | 71.7 |

</div>

Notes:

1. For the Code Agent tasks among the public benchmarks above, DeepSeek-V4-Flash-0731 is evaluated with the minimal mode of DeepSeek Harness (to be released) as the agent framework, using the `max` reasoning effort level with `temperature = 1.0, top_p = 0.95`.
2. † DSBench-FullStack is an internal full-stack development test set; DSBench-Hard is an internal test set of difficult coding-agent problems.

## Chat Template

This release does not include a Jinja-format chat template. Instead, we provide a dedicated `encoding` folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the [`encoding`](encoding/README.md) folder for full documentation.

The `reasoning_effort` parameter now supports three levels — `low`, `high`, and `max` — which control how much deliberation the model spends before answering.

A brief example:

```python
from encoding_dsv4 import encode_messages, parse_message_from_completion_text

messages = [
    {"role": "user", "content": "hello"},
    {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
    {"role": "user", "content": "1+1=?"}
]

# messages -> string
prompt = encode_messages(messages, thinking_mode="thinking", reasoning_effort="max")

# string -> tokens
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
tokens = tokenizer.encode(prompt)
```

## How to Run with vLLM

DSpark speculative decoding is enabled with a single flag — add --speculative-config with method: dspark to your vLLM launch command:

`--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'`

For example, the command below serves the model with vLLM on a single 4×GB300 node. 
See the [vLLM recipe](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash?hardware=b300&features=tool_calling,reasoning) for detailed instructions and other hardware configurations.

```bash
vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'
```

## Expert Routing: `top_k=4` vs `top_k=6` (Recommended: **`top_k=4`**)

DeepSeek-V4-Flash-0731 ships with `num_experts_per_tok=6` (6 of 256 routed experts
activated per token, ~13B active params). We additionally evaluated
`num_experts_per_tok=4` (~11B active params) and **recommend `top_k=4` as the
default**: it is measurably faster at **statistically indistinguishable accuracy**.

### Why `top_k=4`

1. **~15% fewer activated parameters, for free.** Routing to 4 experts instead of
   6 drops per-token active params from ~13B to ~11B. Shared experts and attention
   are unchanged, so quality is preserved (see measurements below).
2. **Faster inference (~13–18%).** Fewer expert FFN computations plus less
   gather/scatter and softmax overhead in the router. Measured end-to-end:
   HumanEval wall-time ~15% lower, per-token generation ~13% faster.
3. **Power-of-2 dispatch alignment.** `6` is not a power of two; MoE dispatch,
   warp scheduling, and memory alignment on the GPU are more efficient when the
   expert count aligns to a power of two (`4`), improving tensor-core utilization
   for the dispatch/combine shapes.
4. **No accuracy regression.** On our internal SWE-bench-Lite and HumanEval runs
   the difference between `top_k=4` and `top_k=6` is within run-to-run noise
   (details below).

### Single-GPU test environment (used for the numbers below)

The measurements were produced on a **single NVIDIA B300 (SXM6, ~275 GiB)** — no
data/expert parallelism and **DSpark speculative decoding disabled** (the official
multi-GPU launch in [How to Run with vLLM](#how-to-run-with-vllm) enables DSpark;
that is orthogonal to the routing comparison here). Exact launch command:

```bash
CUDA_HOME=$HOME/.local/lib/python3.11/site-packages/nvidia/cu13 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
CUDA_VISIBLE_DEVICES=0 \
vllm serve /path/to/DeepSeek-V4-Flash-0731 \
  --served-model-name dsv4 --port 18002 \
  --trust-remote-code --kv-cache-dtype fp8 \
  --max-model-len 32768 --gpu-memory-utilization 0.85 \
  --reasoning-parser deepseek_v4 --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice
```

Notes for single-GPU:

- **No `--speculative-config`** — DSpark is left off, so throughput numbers reflect
  the base model. (DSpark is a decode-speed optimization and does not change which
  tasks pass; it can be re-enabled independently.)
- `--max-model-len 32768` fits the KV cache comfortably in 275 GiB alongside the
  fp8+MXFP4 weights; raise it if you have headroom.
- The `tid2eid` shim from [How to enable `top_k=4`](#how-to-enable-top_k4) must be
  applied before serving with `num_experts_per_tok=4`.
- Weight load takes ~6–9 min (fp8/fp4 MoE autotuning on first start).

**Test harnesses:** HumanEval via `/v1/chat/completions` (code-fence extraction,
`temperature=0.1`); SWE-bench-Lite via the [mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent)
minimal text-based agent (no Docker; each repo in an isolated `uv` venv), with the
official code-agent sampling `temperature=1.0, top_p=0.95`.

### Measured accuracy — the difference is within noise

All numbers below are from the single-B300 setup above on the native
fp8+MXFP4 weights.

**HumanEval** (pass@1, 164 problems, thinking mode, `temperature=0.1`):

| Config | Pass@1 | Note |
| :--- | :---: | :--- |
| `top_k=4` | **92.1% / 91.5% / 92.1%** (3 reps: 151 / 150 / 151) | run-to-run spread ±1 problem |
| `top_k=6` | 90.9% (149) | within the ±1-problem noise band |

**SWE-bench-Lite** (n=86 subset, mini-swe-agent harness, no Docker,
official code-agent sampling `temperature=1.0, top_p=0.95` unless noted):

| Config | Resolved | Rate | Sampling |
| :--- | :---: | :---: | :--- |
| `top_k=6` | 37/86 | 43.0% | default |
| `top_k=4` (rep 1) | 38/86 | 44.2% | default |
| `top_k=4` (rep 2) | 38/86 | 44.2% | default |
| `top_k=4` (official) | **39/86** | **45.3%** | `t=1.0, p=0.95` |

Two-proportion z-test `top_k=4` vs `top_k=6`: **z ≈ 0.15–0.31 (not significant)**.
Repeating the *same* `top_k=4` config flips ~14–17 of the 86 instances per pair of
runs (≈31% of the ever-solved union) purely from MoE-routing / fp8-kernel /
batching non-determinism. Aggregated over 4 runs: **23 instances always pass**
(stable core), **37 always fail**, and **26 are coin-flips**. In other words, the
per-task differences between `top_k=4` and `top_k=6` are the same magnitude as
`top_k=4` versus itself — i.e. **noise, not a capability gap**. `top_k=4` gets the
speed and parameter savings at no measurable accuracy cost.

> On knowledge-heavy multiple-choice benchmarks (e.g. MMLU-Pro) narrower routing
> can even help slightly; on code generation `top_k=6` may hold a fraction of a
> point. Both directions are inside the noise band on our runs — treat them as
> equivalent in quality.

### How to enable `top_k=4`

**Step 1 — set the config.** In `config.json`:

```json
"num_experts_per_tok": 4
```

**Step 2 — patch vLLM weight loading (required).** The checkpoint's `tid2eid`
tensor (the hash-based expert-routing lookup table) was trained at `top_k=6`, so
it has shape `[vocab_size, 6]`. With `num_experts_per_tok=4` the model allocates a
`[vocab_size, 4]` parameter, and loading fails with:

```
AssertionError: Attempted to load weight (torch.Size([129280, 6]))
into parameter (torch.Size([129280, 4]))
```

Fix it by slicing the checkpoint tensor to the first `top_k` columns during load.
In `vllm/models/deepseek_v4/nvidia/model.py`, inside `load_weights`, in the
final `else` branch just before the `weight_loader(param, loaded_weight)` call:

```python
param = params_dict[name]
# top_k override: checkpoint's tid2eid is [vocab, 6] (trained at top_k=6);
# slice to the config's top_k columns so it matches the allocated parameter.
if "tid2eid" in name and loaded_weight.shape != param.shape:
    loaded_weight = loaded_weight[:, :param.shape[1]].contiguous()
weight_loader = getattr(param, "weight_loader", default_weight_loader)
weight_loader(param, loaded_weight)
```

The `[:, :param.shape[1]]` slice keeps the highest-priority expert columns and is
a no-op when the shapes already match (`top_k=6`), so the patch is safe to leave
in place for both configurations. No other weights change — total parameters,
routing method (`noaux_tc`), shared experts, and attention are all identical.

> Note: this is a weight-loading shim, not a re-training of the router table.
> The hash table's remaining 4 columns are the same top entries used at `top_k=6`,
> which is why accuracy is preserved.

## How to Run Locally

Please refer to the [inference](inference/README.md) folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos.

For local deployment, we recommend setting the sampling parameters to `temperature = 1.0`, with `top_p = 0.95` for agentic scenarios and `top_p = 1.0` otherwise. For the `high` and `max` reasoning effort levels, we recommend a maximum output length of **384K** tokens.

## License

This repository and the model weights are licensed under the [MIT License](LICENSE).

## Citation

```
@misc{deepseekai2026deepseekv4,
      title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
      author={DeepSeek-AI},
      year={2026},
}
```