kingjones777/DeepSeek-V4-Flash-0731-ROCmFP4

DeepSeek-V4-Flash-0731 — ROCmFP4 (Strix Halo) GGUF

This is a ROCmFP4 quant of deepseek-ai/DeepSeek-V4-Flash-0731, built to fit a single AMD Strix Halo box (128 GB unified memory) with full GPU offload. As far as I can tell it's the first ROCmFP4 quant of this model. I made it with the ROCmFPX fork of llama.cpp for the gfx1151 (Radeon 8060S / Ryzen AI MAX+ 395) Vulkan/ROCm stack.

Base model	deepseek-ai/DeepSeek-V4-Flash-0731
Quant	Q3 — mixed ROCmFP4, experts ~3.14 bpw, 2.92 BPW overall
Size	~101 GB (fits 128 GB unified memory with headroom)
Arch	deepseek4 (sparse MoE, 256 experts, indexer/DSA attention)
Target HW	AMD Strix Halo gfx1151 iGPU (Ryzen AI MAX+ 395), Vulkan RADV
Loader	ROCmFPX fork — stock llama.cpp cannot load ROCmFP4 tensors

Why I made it

A standard 4-bit GGUF of this model comes out around 141 GB, which overflows a 128 GB Strix Halo's shared pool and spills to CPU. I wanted the largest-quality quant that still fully offloads on a single box and stays coherent, so I mixed the expert tensors down to land it at ~101 GB.

Recipe (quantized from the F16 with the fork's llama-quantize):

• base type Q2_0_ROCMFPX
• ffn_down_exps → q3_0_rocmfpx (3.5 bpw)
• ffn_gate_exps, ffn_up_exps → q2_0_rocmfpx (2.5 bpw)
• attention / embeddings → ROCmFPX; norms kept in fp32

The ROCmFP4 (_ROCMFPX) types hold quality better than equivalent-bit k-quants on this hardware while using the FP4 paths on gfx1151.

Running it

Build the ROCmFPX fork (llama-server / llama-cli) for gfx1151, then:

export HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1
export AMD_VULKAN_ICD=RADV VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/radeon_icd.json

./llama-server \
  -m DeepSeek-V4-Flash-0731-Q3-ROCmFP4-00001-of-00004.gguf \
  -dev Vulkan0 -ngl 999 -fa on -fit off --no-mmap \
  -c 8192 -n 2048 -np 1 -b 1024 -ub 512 -t 16 --poll 50 --jinja \
  --reasoning-format deepseek \
  --chat-template-kwargs '{"enable_thinking":false}' \
  --host 0.0.0.0 --port 8084

Notes from getting it stable on my box:

• -fit off — the fork's auto-fit step crashed on this arch for me; pin -ngl 999 and turn it off.
• --no-mmap — important for MoE speed. With mmap, experts page-fault per token and throughput roughly halves.
• -c 8192 with -b 1024 -ub 512 keeps the graph pool under its limit; larger context can overflow it.
• -n 2048 caps runaway generations so one request can't hold the single slot forever.
• --chat-template-kwargs '{"enable_thinking":false}' gives fast, direct answers. Drop it (or pass enable_thinking:true per request) for the model's reasoning mode.
• Expect roughly 5–8 tok/s — it's a 101 GB model on one iGPU. Use streaming for a usable feel.

A note on MTP

This checkpoint ships a multi-token-prediction (nextn) head, and I kept those tensors in this quant. I got a working MTP inference path running on this arch and tested it thoroughly, but on this hardware/loader combination MTP nets out slightly slower than plain decoding — the draft head's acceptance is low and the sparse-MoE verify step can't amortize its weight reads across draft tokens. I ran it against draft depth, the probability threshold, and draft-head precision; none of them turned it into a win here. So I ship it with MTP off. If you want the model's advertised MTP speedup, run it on a CUDA/vLLM stack instead of this one.

License

Derived from deepseek-ai/DeepSeek-V4-Flash-0731; the original model's license applies (see license_link). This upload is only a quantization — all capabilities and limitations are the base model's.