antirez

/

deepseek-v4-gguf

like 460

Summary

This PR publishes the AProjQ4 DeepSeek V4 Flash GGUFs and the 220k routed-and-dense DS4 importance matrix used to build them.

The recommended artifact for new testing is:

DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

It is a same-checkpoint requantization of the existing DeepSeek V4 Flash -0731 AProjQ8 GGUF. Only the dense attention projections were changed: 215 tensors were requantized from Q8_0 to Q4_K, guided by the included 220k imatrix.

The matching runtime implementation, backend kernels and validation work are tracked in DS4 GitHub PR #621.

---

Artifacts included in this PR

The artifacts are currently stored under the Hugging Face revision:

refs/pr/22

They should not be assumed to exist on the repository’s main branch until this PR is merged.

Recommended GGUF metadata

File:
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

Size:
84,420,584,288 bytes
78.62 GiB

SHA-256:
413cf0a68ca8d084e89f3f810eef5046b5308174d441a80017a0ff388933c767

---

How the AProjQ4 GGUF was built

The -0731 AProjQ4 GGUF was not rebuilt from the original DeepSeek safetensors and it does not use a different checkpoint.

It was produced by directly requantizing the existing same-checkpoint AProjQ8 GGUF:

Source GGUF:
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

Importance matrix:
imatrix/DeepSeek-V4-Flash-chat-v2-routed-and-dense-ds4-220k.dat

Transformation:
215 dense attention projection tensors
Q8_0 → Q4_K

Output GGUF:
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

In simplified form:

existing AProjQ8 0731 GGUF
        +
220k routed-and-dense DS4 imatrix
        +
requantization of the dense attention projections
from Q8_0 to Q4_K
        =
AProjQ4 0731 GGUF

This GGUF-to-GGUF process avoids downloading and reconverting the full original safetensor checkpoint. It also creates a clean same-checkpoint A/B pair for quality, memory and performance comparisons.

---

Which tensors were requantized

DeepSeek V4 Flash contains five relevant dense attention projections per layer:

• attn_q_a
• attn_q_b
• attn_kv
• attn_output_a
• attn_output_b

The model has 43 layers, therefore:

5 projections × 43 layers = 215 requantized tensors

Those 215 tensors were changed from:

Q8_0 → Q4_K

All tensors outside these dense attention projections retain the layout and quantization types of the source AProjQ8 GGUF.

In particular, this is not a generic all-Q4 model. The routed experts, shared experts, output tensors and the rest of the quantization mix remain those identified by the filename:

IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8

---

Role of the imatrix

The included importance matrix is:

imatrix/DeepSeek-V4-Flash-chat-v2-routed-and-dense-ds4-220k.dat

It contains activation-importance statistics collected from DS4 calibration runs covering both routed and dense model paths.

During requantization, these statistics allow the quantizer to weight reconstruction error according to how important each input dimension is during actual model execution. This is preferable to treating every dimension as equally important when converting the dense projections to Q4_K.

The imatrix is included in this PR so that the AProjQ4 artifact has reproducible provenance and so that future requantizations can use the same calibration data.

---

Why AProjQ4 exists

The goal is to reduce the resident memory occupied by the dense attention projections while keeping the rest of the model layout unchanged.

The matched DS4 comparison reports approximately:

AProjQ8 model: 80.76 GiB
AProjQ4 model: 78.62 GiB
Saving:           2.14 GiB

That memory saving is useful in two main scenarios.

Fully resident inference

On unified-memory systems such as DGX Spark or Strix Halo, the saved memory can become additional KV-cache or context headroom.

SSD-streaming inference

On Macs that cannot keep the entire model resident, the saved memory can be assigned to the routed-expert cache. A larger expert cache reduces repeated SSD reads and expert-cache churn during generation.

The relevant trade-off is therefore not only Q4 kernel throughput. It is the complete combination of:

memory
+ correctness
+ quality
+ prefill speed
+ generation speed

---

Runtime support

This Hugging Face PR publishes the model artifacts.

The corresponding DS4 runtime changes are implemented and validated in:

antirez/ds4 GitHub PR #621 — Support AProjQ4 GGUFs

That runtime work adds type-aware execution for Q4_K dense attention projections across:

• Metal;
• CUDA;
• ROCm;
• the CPU reference backend.

Before this work, DS4 could accept the tensors while loading the GGUF, but some execution paths still interpreted their data as Q8_0. That could result in failed prefill, invalid numerical output or BOS loops.

The GitHub PR adds the required dispatch, kernels, CPU reference implementation, backend tests and performance work.

Support in arbitrary GGUF runtimes is not implied by this artifact PR. A runtime must explicitly support this DeepSeek V4 Flash tensor layout and the Q4_K dense attention paths.

---

Validation

Official-continuation scoring

The -0731 artifact was evaluated with the DS4 official-continuation scorer:

Cases:         100
Target tokens: 2,313
avg_nll:       0.398263336

Lower avg_nll is better.

This number should be interpreted only within a matched comparison using the same:

• checkpoint;
• continuation manifest;
• scorer version;
• backend;
• context settings.

Scores collected from different checkpoints or fixture versions are not directly comparable.

The complete matched AProjQ8/AProjQ4 quality analysis and subsequent backend revalidations are documented in GitHub PR #621.

Runtime validation

Validation reported during development includes:

• coherent greedy generation;
• absence of the original BOS-loop corruption;
• official continuation vectors;
• decode-consistency checks;
• Metal resident inference;
• Metal SSD streaming;
• CUDA on DGX Spark / GB10;
• ROCm on Strix Halo;
• CPU reference execution;
• Q4_K dense and paired kernel parity tests;
• separate prefill and generation benchmarks;
• memory and long-context measurements.

The exact commit, machine, backend, commands, results and known caveats are recorded in the GitHub PR discussion.

---

Downloading before merge

Until this PR is merged, use revision refs/pr/22.

For example, with the Hugging Face CLI:

hf download antirez/deepseek-v4-gguf \
  DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf \
  --revision refs/pr/22 \
  --local-dir ./gguf

The corresponding browser path is:

https://huggingface.co/antirez/deepseek-v4-gguf/blob/refs%2Fpr%2F22/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

After downloading, verify:

sha256sum \
  DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

Expected:

413cf0a68ca8d084e89f3f810eef5046b5308174d441a80017a0ff388933c767

---

Merge checklist

Artifacts

• AProjQ4 -0731 GGUF uploaded
• 220k routed-and-dense imatrix uploaded
• Source AProjQ8 checkpoint identified
• Requantized tensor count documented
• GGUF size documented
• GGUF SHA-256 documented
• Recommended artifact identified
• Decide whether the pre-0731 GGUF should remain
• Mark the pre-0731 GGUF as legacy if retained
• Resolve the .gitattributes merge conflict

Requantization provenance

• Same-checkpoint AProjQ8 source used
• Direct GGUF-to-GGUF requantization documented
• Original safetensors were not required
• 215 dense attention projection tensors documented
• Q8_0 → Q4_K conversion documented
• Exact imatrix path documented
• Unchanged portions of the source layout explained

Quality and runtime validation

• Official-continuation scorer run
• Case and target-token counts documented
• Initial avg_nll documented
• Metal SSD-streaming validation completed
• Matching DS4 runtime PR linked
• CUDA validation documented in GitHub PR #621
• ROCm validation documented in GitHub PR #621
• CPU reference validation documented in GitHub PR #621
• Link the final exact-SHA validation summary before merge
• Confirm the final quality result with the final scorer and manifest versions

Documentation

• Recommended -0731 artifact clearly identified
• Relationship between the Hugging Face and GitHub PRs explained
• Requantization process explained
• Download instructions for refs/pr/22 provided
• Update the repository model card after merge
• Add the final artifact to the normal download instructions after merge

Added the 0731 AProjQ4 GGUF in commit 0d193661:

DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ4-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf

• Size: 84,420,584,288 bytes
• Hub SHA-256: 413cf0a68ca8d084e89f3f810eef5046b5308174d441a80017a0ff388933c767
• Source: 0731 AProjQ8 GGUF
• Requantization: 215 dense attention projection tensors, Q8_0 -> Q4_K
• Imatrix: imatrix/DeepSeek-V4-Flash-chat-v2-routed-and-dense-ds4-220k.dat
• Official-continuation score: 100 cases / 2,313 target tokens, avg_nll=0.398263336

The upload was committed directly to refs/pr/22; Xet deduplicated the 84.4 GB file down to about 2.87 GB of new uploaded data.