Instructions to use deepseek-ai/DeepSeek-V4-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V4-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V4-Flash")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V4-Flash", device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-V4-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V4-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash

SGLang

How to use deepseek-ai/DeepSeek-V4-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V4-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V4-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V4-Flash with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash
```

DeepSeek-V4-Flash fine-tuning: can a merged model keep the same FP4+FP8 format as the base model, or is BF16 the only supported merge output?

#40

by zhangshuai121 - opened Jun 15

Discussion

zhangshuai121

Jun 15

I am fine-tuning deepseek-ai/DeepSeek-V4-Flash with ms-swift / Megatron LoRA.
The official base model uses a mixed low-precision format: MoE expert weights are FP4 and most other weights are FP8. After LoRA fine-tuning, if I use merge_lora, the merged full model becomes BF16/F16 and is much larger than the original base checkpoint.
I want to know whether anyone has successfully produced a fine-tuned / merged DeepSeek-V4-Flash checkpoint that keeps the same format and size characteristics as the official base model, namely FP4 experts + FP8 non-expert weights.
Questions:
Is merged BF16/F16 currently the expected and only supported output after LoRA merge?
Is there an official or community workflow to merge LoRA back into the original FP4+FP8 mixed checkpoint format?
Can post-training quantization restore the exact official DeepSeek-V4-Flash format, or would it only produce a different quantized format such as generic FP8/AWQ/GPTQ?
Are there known serving engines that support DeepSeek-V4-Flash LoRA adapters directly, including modules such as wq_a, wkv, wo_b, linear_fc1, or linear_fc2?
In my tests, the base model can start successfully, but serving the LoRA adapter separately has compatibility issues depending on the target modules. I am trying to understand whether preserving the base model’s original precision layout after fine-tuning is possible, or whether the practical options are currently limited to BF16 merged weights or base-plus-LoRA serving.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment