README.md

---
library_name: hipfire
license: mit
base_model: deepseek-ai/DeepSeek-V4-Flash
base_model_relation: quantized
pipeline_tag: text-generation
tags:
  - hipfire
  - mq2lloyd
  - mixed-precision
  - amd
  - rocm
  - rdna
  - moe
  - deepseek
  - deepseek-v4
  - mtp
  - dspark
  - speculative-decoding
---

# hipfire DeepSeek V4 Flash (mq2lloyd)

A mixed-precision packaging of [`deepseek-ai/DeepSeek-V4-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) (284 B total / 13 B active per upstream) for the [hipfire](https://github.com/Kaden-Schutt/hipfire) Rust-native inference engine on AMD RDNA GPUs.

Upstream is shipped in FP8 (`e4m3`) with FP4 routed-expert weights. This packaging rewrites the dominant weight class — the 256 routed MoE experts per layer — as 2-bit MagnumQuant-Lloyd (MQ2-Lloyd), and keeps everything else as Q8F16 or F16. The container suffix `.mq2lloyd` names that dominant class, but the file is not a uniform 2-bit dump.

The file format is hipfire's HFQ container and is **not GGUF / safetensors / AWQ compatible** — it only loads in hipfire.

## What's inside (verified by enumerating the file's tensor table)

The 86.2 GB file contains 34 223 tensors. By hipfire `QuantType` enum:

```
  33024  qt=19 (MQ2G256Lloyd)        – 256 routed experts × 3 (w1/w2/w3) × 43 layers
    389  qt=3  (Q8F16)               – shared experts, main attn, embed, lm_head, router gates
    807  qt=1  (F16)                 – norms, compressor, indexer, hc_*, attn_sink
      3  qt=22 (TidI32)              – hash-router fast-path (3 layers only)
```

| Tensor class | Storage | Weights (this file) | Storage bytes |
|---|---|---:|---:|
| Routed MoE experts | **MQ2G256Lloyd, 2.25 bpw** (G=256 → 64 B 2-bit indices + 8 B fp16 Lloyd codebook = 72 B/group) | 277.025 B | 77.913 GB |
| Shared expert + main attention (wq_a/b, wkv, wo_a/b) + embed + lm_head + MoE router gates | **Q8F16, 8.5 bpw** (GGML Q8_0 block: 2 B F16 scale + 32 B Q8 data per 32 weights = 34 B / 32 w) | 6.785 B | 7.209 GB |
| Compressor / indexer matrices, RMSNorm scales, attn_sink, HC gating (`hc_attn_*`, `hc_ffn_*`, `hc_head_*`) | **F16, 16 bpw** | 0.522 B | 1.043 GB |
| `tid2eid` hash-router table (layers 0–2 only — `num_hash_layers = 3`) | **TidI32, 32 bpw** | 2.3 M | 9.3 MB |
| **Total** | mixed → **2.425 bpw avg** | **284.335 B** | **86.175 GB weights + 9.2 MB header/metadata = 86.184 GB on disk** (estimate matches the actual file to within 0.01 %) |

The runtime SWA K and V state (window = 128) is **F32** and is allocated live at session start — it is not persisted in the file.

## Files

| File | Size | Purpose |
|------|------|---------|
| `deepseek-v4-flash.mq2lloyd` | 86,184,307,283 B (≈86.2 GB) | Main model — 43 layers, 256 routed + 1 shared expert per layer, attention, embed, lm_head |
| `deepseek-v4-flash-mtp.mq2lloyd` | 1,998,047,355 B (≈2.0 GB) | Single MTP layer (`num_nextn_predict_layers = 1`), opened automatically when present alongside the main file. Used for optional speculative decode. |
| `deepseek-v4-flash-dspark.mq2lloyd` | 5,996,334,814 B (≈6.0 GB) | **DSpark 3-stage speculative drafter** (`mtp.0/1/2`), opened automatically via the `<stem>-dspark.<ext>` sibling convention. Wins over the single MTP head under `--spec dspark` / `--spec auto`. See the DSpark section below. |

## DSpark speculative drafter (`-dspark` sidecar)

`deepseek-v4-flash-dspark.mq2lloyd` is an optional multi-token draft module that replaces the single in-trunk MTP head with a **3-stage chain** (`mtp.0 → mtp.1 → mtp.2`) proposing a block of 5 tokens per step. It is additive to the main file and to the MTP companion — attach whichever you want; DSpark auto-wins over MTP when both are present.

Verified by enumerating the file's tensor table (`hfq_dump`): **2 376 tensors, arch_id = 9**, same mixed-precision scheme as the main file:

```
  2304  qt=19 (MQ2G256Lloyd)   – 256 routed experts × 3 (w1/w2/w3) × 3 draft stages
    31  qt=3  (Q8F16)          – per-stage attention, shared experts, main_proj, markov heads, confidence head
    41  qt=1  (F16)            – norms, mHC gating (hc_head_*), main_norm
```

The three stages are near-symmetric full DeepSeek-V4 blocks (791 / 789 / 796 tensors) with role-specific heads:

- **`mtp.0`** — owns the embed/main-hidden entry: `main_proj` (Linear 12288→4096, no bias) + `main_norm` (RMS). Draft input comes from `main_proj` + a noise block (`dspark_noise_token_id = 128799`), not the MTP head's `e_proj`/`h_proj` mixing.
- **`mtp.1`** — middle draft block.
- **`mtp.2`** — owns the output heads: `hc_head_{base,fn,scale}`, `markov_head.{markov_w1, markov_w2}` (Embed 129280→256 / Linear 256→129280) for the Markov token-bias prior, and `confidence_head.proj` (Linear 4352→1, fp32) driving confidence-truncated acceptance.

Config (from the file's metadata blob): `dspark_block_size = 5`, `dspark_target_layer_ids = [40, 41, 42]`, `dspark_markov_rank = 256`, `dspark_noise_token_id = 128799`.

Enable it with `hipfire run deepseek-v4-flash --spec dspark` (force) or `--spec auto` (DSpark preferred when the sidecar is present). Acceptance can be tuned with `--dspark-conf-threshold <0..1>` (deepseek4 default 0.3).

## Loading

This model is registered in [hipfire's CLI registry](https://github.com/Kaden-Schutt/hipfire/blob/master/cli/registry.json) under `deepseek-v4-flash` (aliases: `deepseek4`, `deepseek-v4`). The registry entry pulls the main file together with both the MTP and DSpark companions.

```bash
# 1. Install hipfire (one-time):
curl -L https://raw.githubusercontent.com/Kaden-Schutt/hipfire/master/scripts/install.sh | bash

# 2. Pull the model (downloads both .mq2lloyd files into ~/.hipfire/models/):
hipfire pull deepseek-v4-flash

# 3. Run it:
hipfire run deepseek-v4-flash "Write a fibonacci function in C"
#    or interactive chat:
hipfire run deepseek-v4-flash
#    or expose an OpenAI-compatible HTTP server on :11435:
hipfire serve
```

`hipfire run` auto-pulls if the files aren't present, so step 2 is optional. Both `HIPFIRE_DEEPSEEK4_MOE` and `HIPFIRE_DEEPSEEK4_UPLOAD_EXPERTS` default to **on** at the engine level — no env vars are required.

### From source (development path)

```bash
git clone https://github.com/Kaden-Schutt/hipfire
cd hipfire
cargo build --release -p hipfire-arch-deepseek4 --example deepseek4_chat

# Pull the files (uses `hf` CLI or hipfire's registry-driven pull above):
hf download nwoolmer/hipfire-deepseek-v4-flash \
    deepseek-v4-flash.mq2lloyd deepseek-v4-flash-mtp.mq2lloyd \
    deepseek-v4-flash-dspark.mq2lloyd \
    --local-dir ~/.hipfire/models/

# Direct chat binary (DSML chat template, EOS stop, multi-turn KV):
HIPFIRE_DEEPSEEK4_MODEL=~/.hipfire/models/deepseek-v4-flash.mq2lloyd \
    ./target/release/examples/deepseek4_chat
```

For programmatic access, `crates/hipfire-runtime/examples/daemon.rs` exposes the engine as a JSON-lines IPC service over stdin/stdout (spawned by the Bun CLI front-end); it dispatches DeepSeek V4 Flash via `arch_id = 9`.

## Architecture

The packaging follows the V4F architecture as described in the DeepSeek V4 paper. Every value below is read from the `config` JSON embedded in the HFQ file's metadata blob:

| Field | Value | Notes |
|---|---|---|
| `architectures` | `["DeepseekV4ForCausalLM"]` | upstream class |
| `num_hidden_layers` | 43 | |
| `hidden_size` | 4096 | |
| `vocab_size` | 129 280 | |
| `num_attention_heads` / `num_key_value_heads` / `head_dim` | 64 / 1 / 512 | 64 query heads; KV is a single latent stream of dim 512 (MLA, joint K+V via `wkv [512, 4096]`) |
| `q_lora_rank` / `o_lora_rank` / `o_groups` | 1024 / 1024 / 8 | Q low-rank factorisation `wq_a [1024,4096] → wq_b [32768,1024]`; grouped O projection `wo_a [8192,4096] → wo_b [4096,8192]` with intermediate = `o_groups · o_lora_rank = 8192` |
| `qk_rope_head_dim` | 64 | tail-split RoPE: only the last 64 channels per head get rotated |
| `n_routed_experts` / `n_shared_experts` / `num_experts_per_tok` | 256 / 1 / 6 | top-6 routing per token |
| `moe_intermediate_size` | 2048 | per-expert width |
| `hc_mult` / `hc_sinkhorn_iters` / `hc_eps` | 4 / 20 / 1e-6 | 4-stream Hyper-Connections, 20-iter Sinkhorn normalisation |
| `index_n_heads` / `index_head_dim` / `index_topk` | 64 / 128 / 512 | indexer-gated attention: top-512 over compressed-K |
| `compress_ratios` | `[0, 0, 4, 128, 4, 128, …, 4, 0]` (len 43) | layers 0–1 are dense; compressed-KV attention from layer 2 onward |
| `num_hash_layers` | 3 | first three layers carry the `tid2eid` router fast-path |
| `num_nextn_predict_layers` | 1 | one MTP head (in the companion file) |
| `sliding_window` | 128 | SWA window for the main attention path |
| `rope_theta` / `compress_rope_theta` | 10 000 / 160 000 | |
| `rope_scaling` | YaRN, factor 16, original_max 65 536 → max_position 1 048 576 | |
| `expert_dtype` (upstream) / `quantization_config` | `fp4` / `fp8 e4m3 [128,128]` | upstream's quant; superseded here by MQ2-Lloyd + Q8F16 + F16 |

Tensor-presence cross-check against the file (also verified above):

- Layers 0 and 1: dense attention (`wq_a/b`, `wkv`, `wo_a/b` only — no compressor / no indexer).
- Layers 2–42: compressed-KV attention (every layer has `attn.compressor.{wkv,wgate,ape,norm}`).
- Even layers 2, 4, …, 42 (21 layers): also carry the indexer block (`attn.indexer.{wq_b, weights_proj, compressor.*}`).
- Layers 0, 1, 2: also ship the `ffn.gate.tid2eid` hash-router fast-path table.
- The MTP companion file contains a single layer (`mtp.0.*`) with its own attention block, 256 routed experts, a shared expert, a full mHC stack (`hc_attn_*`, `hc_ffn_*`, `hc_head_*`), and input-side projection/norm tensors (`e_proj`, `h_proj`, `enorm`, `hnorm`) used by the MTP head's input mixing.

## Performance

Measured 2026-05-28 on **AMD Radeon 8060S (gfx1151, Strix Halo APU, 128 GB UMA)**, hipfire at tag [`v0.2.0`](https://github.com/Kaden-Schutt/hipfire/releases/tag/v0.2.0) (commit `3d456e5c`), ROCm 7.2.1, SWA attention path, `temp=0.7 top_k=40`, prompt_normalize on.

| Mode | Throughput | How measured |
|---|---|---|
| Plain decode (TG) | **~13.9 tok/s** | 13 warm-turn measurements across two chat processes (turn ≥ 2 of each, both 256-tok and 16-tok generations): median 13.86, range 13.61 – 14.00, σ ≈ 0.7 %. |
| Batched prefill (PP) | **~55 tok/s** | 1 235-token fresh-KV prefill of a single-chunk system prompt (no internal blank lines). `HIPFIRE_DEEPSEEK4_PP_BATCH` default 1024. Sub-100-token chat-turn prompts are overhead-dominated (~40 tok/s) and are not the right point to cite. |
| Spec decode (MTP, K=3) | **14–19 tok/s** | Median 16.4 tok/s, +19 % over plain. Draft accept rate ranged 41–62 % across the three turns of the same chat (highest on direct code generation, lowest on conversational follow-ups). Enable with `HIPFIRE_DEEPSEEK4_SPEC_DECODE=1 HIPFIRE_DEEPSEEK4_SPEC_K=3`. |

Cold-process load — weight upload from both HFQ files — is **~44 s** on the 8060S (measured by an inline timestamping wrapper from process start until the engine prints `DeepSeek V4 ready.`). Plain decode is DRAM-bandwidth-bound; faster memory (RDNA3 desktop GDDR6, future RDNA4) scales decode roughly proportionally.

## Compatibility

- **GPU**: AMD RDNA3 / RDNA3.5 with HIP + WMMA. **Validated on gfx1151 (Radeon 8060S, Strix Halo)** for this build of the V4F weights. The engine has gfx1100 (RX 7900-class) kernels in tree but they were not exercised against this specific file; RDNA1/2 (gfx1010 / gfx103x) and gfx12 are tracked targets of the broader hipfire project but not recommended for running this model.
- **OS**: Linux with `amdgpu` kernel driver. Built and measured against **ROCm 7.2.1 / HIP 7.2**.
- **Memory**: 86.2 GB for the main file + ~2 GB for the MTP companion + ~6 GB for the optional DSpark drafter + several GB working set during decode. Strix Halo class systems with 128 GB UMA are the comfortable target; discrete-GPU configs need ~96 GB+ of VRAM.
- **Tested context length**: end-to-end inference was exercised at prompts up to ~1.2 k tokens and generations up to 256 tokens. Upstream's stated max position (1 048 576 via YaRN) was not exercised here.

## License

The upstream model ([`deepseek-ai/DeepSeek-V4-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)) is **MIT-licensed**; the weights in this packaging inherit those terms.

The hipfire engine that produced and consumes this format is **dual-licensed MIT / Apache-2.0** at the user's option (see `LICENSE-MIT`, `LICENSE-APACHE`, and `NOTICE` in the engine repo).

## Acknowledgements

- DeepSeek AI — original DeepSeek V4 Flash weights and architecture.
- [Salvatore Sanfilippo (antirez)](https://github.com/antirez/ds4) — **DwarfStar**, a focused C/Metal/CUDA/ROCm reference engine for DeepSeek V4 Flash. The hipfire impl cross-validates against it for MTP wiring, HC reduction, and KV layout.

## Citation

If you use this packaging, please cite the upstream model, the hipfire engine (per its `CITATION.cff`), and this HF release:

```bibtex
@misc{deepseekai2026deepseekv4,
  title  = {DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
  author = {DeepSeek-AI},
  year   = {2026},
  url    = {https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash}
}

@software{hipfire,
  title   = {hipfire — Rust-native LLM inference for AMD RDNA / CDNA},
  author  = {Schutt, Kaden},
  year    = {2026},
  version = {0.2.0},
  url     = {https://github.com/Kaden-Schutt/hipfire}
}

@misc{hipfire-deepseek-v4-flash,
  title  = {hipfire DeepSeek V4 Flash (MQ2-Lloyd)},
  author = {Woolmer, Nick},
  year   = {2026},
  url    = {https://huggingface.co/nwoolmer/hipfire-deepseek-v4-flash}
}
```