Instructions to use 6block/DeepSeek-V4-Flash-0731-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use 6block/DeepSeek-V4-Flash-0731-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M
# Run inference directly in the terminal:
llama cli -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M
# Run inference directly in the terminal:
llama cli -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M
# Run inference directly in the terminal:
./llama-cli -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Use Docker

docker model run hf.co/6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

LM Studio
Jan

vLLM

How to use 6block/DeepSeek-V4-Flash-0731-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "6block/DeepSeek-V4-Flash-0731-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "6block/DeepSeek-V4-Flash-0731-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Ollama
How to use 6block/DeepSeek-V4-Flash-0731-GGUF with Ollama:
```
ollama run hf.co/6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M
```

Unsloth Studio

How to use 6block/DeepSeek-V4-Flash-0731-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for 6block/DeepSeek-V4-Flash-0731-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for 6block/DeepSeek-V4-Flash-0731-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for 6block/DeepSeek-V4-Flash-0731-GGUF to start chatting

How to use 6block/DeepSeek-V4-Flash-0731-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use 6block/DeepSeek-V4-Flash-0731-GGUF with Docker Model Runner:
```
docker model run hf.co/6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M
```

Lemonade

How to use 6block/DeepSeek-V4-Flash-0731-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Run and chat with the model

lemonade run user.DeepSeek-V4-Flash-0731-GGUF-IQ1_M

List all available models

lemonade list

Hermes Agent

How to use 6block/DeepSeek-V4-Flash-0731-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use 6block/DeepSeek-V4-Flash-0731-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf 6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "6block/DeepSeek-V4-Flash-0731-GGUF:IQ1_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

DeepSeek-V4-Flash-0731 GGUF

GGUF quantizations of deepseek-ai/DeepSeek-V4-Flash-0731, made with llama.cpp.

Chinese version: README_zh.md

Why there is no Q8_0 / Q6_K / Q5_K_M / Q4_K_M

The expert layers ship as FP8 (quantization_config.fmt = e4m3, block 128x128). After conversion to GGUF they land in MXFP4, taking 137.06 GiB (94.1%) of the 156.4 GB master; everything else adds up to roughly 8.6 GiB.

Re-quantizing 4.25 bpw expert weights up to 8.5 bpw only inflates the file — it cannot recover precision that was already discarded upstream. Measured with --dry-run against the same master:

Global type	Size	vs master
Q8_0	281.4 GiB	+93%
Q6_K	217.3 GiB	+49%
Q5_K_M	187.8 GiB	+29%
Q4_K_M	160.0 GiB	+10%
IQ4_XS	142.5 GiB	-2%

So this repo starts at Q3_K_M. If you want near-lossless, use the original weights directly — an intermediate tier buys you nothing here.

Why there is no IQ2_M

It was built, and it failed testing. Quantized with the same settings, IQ2_M came out at PPL = 1.08e9, while the smaller IQ2_XXS sits at 7.47. Every chunk from the very first one is a 9-digit number, reproducible on both CPU and GPU. Tensor count and the per-layer type map match the working tiers line for line, so this is not a misconfiguration — the IQ2_S code path fails on this model (likely related to imatrix columns left empty by 256-expert top-6 sparse routing). IQ2_XS covers the same size range instead.

Quantizations

Tier	Size	Shards	PPL (wikitext-2, n_ctx=512, 12 chunks)
master (expert layers = factory MXFP4)	145.6 GiB	4	2.7562 ± 0.12273
`Q3_K_M`	129.9 GiB	4	3.1992 ± 0.14989
`IQ3_XXS`	105.9 GiB	4	3.7911 ± 0.18074
`Q2_K`	100.5 GiB	4	4.4583 ± 0.22758
`IQ2_XS`	82.1 GiB	4	5.6371 ± 0.29464
`IQ2_XXS`	74.3 GiB	4	7.4718 ± 0.41588
`IQ1_M`	64.6 GiB	4	10.2099 ± 0.58384
`IQ1_S`	58.8 GiB	4	11.7962 ± 0.69607

The master row is not a file in this repo. It is listed so the numbers above have a reference point: its expert layers are the factory MXFP4 weights with no further quantization, which is the accuracy ceiling for this model.

Every tier is imatrix-calibrated (220 chunks, mixed code / English / Chinese), and the non-expert layers are protected:

Tensors	Type	Reason
`attn_*`	Q8_0	source weights are already FP8; compressing further gains nothing
`ffn_*_shexp`	Q8_0	the shared expert runs on every token
`indexer` / `compressor*`	Q8_0	sparse-attention index; compressing it picks the wrong KV
`ffn_gate_inp`	F32	router; compressing it routes to the wrong experts
`hc_*`	F32	per-layer control signals, negligible size
`token_embd` / `output`	Q6_K	a global type would otherwise squeeze these hard

Usage

# Plain decoding — fine across multiple GPUs
llama-cli -hf 6block/DeepSeek-V4-Flash-0731-GGUF:Q3_K_M -ngl 99 -c 8192

# Or point at the files yourself
llama-cli -m DeepSeek-V4-Flash-0731-Q3_K_M-00001-of-00004.gguf -ngl 99 -c 8192

For sharded tiers, download all shards and point at -00001-of-*; llama.cpp finds the rest on its own.

DSpark speculative decoding

The base model ships a DSpark speculative-decoding module. This repo exports it separately as dspark-DeepSeek-V4-Flash-0731-BF16.gguf (10.1 GiB, 81 tensors, architecture dflash). One draft file serves every tier.

# --spec-type is required, and everything has to fit on ONE GPU (see below)
CUDA_VISIBLE_DEVICES=0 llama-cli \
  -m DeepSeek-V4-Flash-0731-IQ1_S-00001-of-00004.gguf \
  -md dspark-DeepSeek-V4-Flash-0731-BF16.gguf \
  --spec-type draft-dspark \
  -ngl 99 -ngld 99 -c 2048

Two things worth knowing, both measured here on 8xH100 with llama.cpp 4ed2b13:

-md on its own does not turn it on. Without --spec-type draft-dspark the draft file loads, the log prints no implementations specified for speculative decoding, and the slot reports "speculative":false — you get plain decoding at full speed and no error. --spec-default does not help either; it only enables ngram-mod and never touches the draft model. With the flag set, the log shows "speculative":true and repeated called impl draft-dspark lines.
One GPU at a time, for now. Split over 2 or more GPUs it aborts on GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) (the cap is 30): the graph wants more cross-backend copies than the scheduler allows. Pinning the draft to one device with -devd does not avoid it, because the target model is what gets split. This is a llama.cpp scheduler limit, not a problem with these files. So the tier plus the 10.1 GiB draft has to fit in a single card — on an 80 GB card that means IQ1_S; the larger tiers need plain decoding until the limit is lifted.

About the PPL numbers

Measured in-house with one fixed setup: wikitext-2 test, n_ctx=512, 12 chunks, -ngl 99, and the master baseline run through the exact same command. These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make the absolute values meaningless across setups.

License

MIT, inherited from deepseek-ai/DeepSeek-V4-Flash-0731. Quantized by 6block.

Downloads last month: 3,048

GGUF

Model size

284B params

Architecture

deepseek4

Hardware compatibility

1-bit

2-bit

3-bit

16-bit

Model tree for 6block/DeepSeek-V4-Flash-0731-GGUF

Base model

deepseek-ai/DeepSeek-V4-Flash-0731

Quantized

(181)

this model

Collection including 6block/DeepSeek-V4-Flash-0731-GGUF

DeepSeek-V4

Collection

Sub-4-bit GGUF quants of DeepSeek-V4: Flash-0731 (seven PPL-tested tiers + DSpark draft) and Pro-0813 (1.57T, factory FP4 master). • 2 items • Updated 7 days ago