primitive-ai/DeepSeek-V4-Flash-Vision-Exp-REAP-145B

DeepSeek-V4-Flash-Vision-Exp REAP-145B — one 96 GB GPU

DeepSeek-V4-Flash-Vision-Exp is 284B parameters in 167.8 GB — the experts ship 4-bit (MXFP4) from the source, so further quantization has nothing left to take, and no single-GPU artifact existed. This one fits: half the routed experts pruned (K128 of 256, REAP), 83.3 GB, serves on one 96 GB card with stock vLLM 0.28.0. Expert bytes are untouched source MXFP4 — pruning removes experts, it never requantizes the survivors.

The expert selection transplants ludo-tech's REAP plan (computed on the text-only 0731 base with their published toolchain, MIT) onto the Vision-Exp weights; the frozen hash-router tables (tid2eid, layers 0–2) are remapped from this checkpoint's own gate tensors. Two Vision-Exp-only gate tensors the text runtime cannot load (gate.bias on hash layers, gate.bias_vl everywhere) are sliced to the kept experts and stored in vision_tower.safetensors with the vision tower and projector — shipped for the day vLLM grows deepseek_v4 vision support, outside the weight index until then. MTP is dropped (inherited from the REAP plan). Text-only serving today, by runtime necessity: vLLM has no vision path for this architecture yet.

Serve (vLLM 0.28.0, one 96 GB Blackwell card)

Recommended (CUDA graphs, 20K context — 73.1 tok/s single-stream; the model reasons long, so the full context budget matters):

vllm serve primitive-ai/DeepSeek-V4-Flash-Vision-Exp-REAP-145B \
  --max-model-len 20480 --gpu-memory-utilization 0.95 \
  --max-num-batched-tokens 2048 --max-num-seqs 4 \
  --kv-cache-dtype fp8

Batch throughput (eager, concurrency 16 — 150–172 tok/s aggregate):

vllm serve primitive-ai/DeepSeek-V4-Flash-Vision-Exp-REAP-145B \
  --max-model-len 20480 --gpu-memory-utilization 0.95 \
  --max-num-batched-tokens 4096 --max-num-seqs 16 \
  --kv-cache-dtype fp8 --enforce-eager

Every flag is load-bearing on 96 GB:

• --kv-cache-dtype fp8 — the architecture's fp8_ds_mla KV layout accepts nothing else.
• --gpu-memory-utilization 0.95 with tight --max-num-batched-tokens / --max-num-seqs — weights take 83.3 GB; the batching bound shrinks activation workspace enough to fit the context and, in the first config, the CUDA graphs. At vLLM's default batching, even 8K does not fit. Graphs are worth 5× on single-stream decode here (75.7 vs 13.9 tok/s eager): this architecture launches many small kernels per token.
• vLLM 0.28.0 or newer. Older builds (including recent dev snapshots) fail on RTX-class Blackwell (SM120) at five distinct layers: DeepGEMM's MegaMoE and hyper-connection kernels assert (they require SM100), cutlass c3x has no SM120 block-FP8 GEMM, the triton fallback lacks the e8m0 scale dtype, and tilelang 0.1.12 aborts on import next to apache-tvm-ffi (0.1.13 fixes that). 0.28.0 routes all of it correctly out of the box.

Measured

One RTX PRO 6000 Blackwell (96 GB). Our standard 1,170-item knowledge + 200-item tool-calling suites, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, 16,384-token budget, auto-scored. Suites ran at concurrency 16 in the eager 20K configuration (our other cards run 32; the memory-bound --max-num-seqs cap decides here).

suite	score
knowledge, strict (1,170 items; truncated = 0)	82.7
knowledge, completed-only (n=1,067)	88.7
knowledge completion rate	91.2%
tool-calling pooled (200)	74.5
— call accuracy (160 call rows)	81.9
— abstain (40 no-call rows)	45.0

Highlights: gsm8k 100.0, arc_challenge 90.7, mmlu_pro 90.8 completed-only. The strict/completed gap is verbosity, not wrongness: mean answer runs 2,761 tokens and 8.8% of items hit the 16,384-token budget — use the 20K serve config if that matters to you.

config	tok/s @ 1	tok/s @ 16	TTFT
graphs 20K (seqs 4), 2K in / 256 out	73.1	—	~220 ms @ 1
graphs 8K (seqs 8), 2K in / 256 out	75.7	—	217 ms @ 1
eager 20K (seqs 16), 8K in / 512 out, cache-free	13.9	150.5 / 171.9	864 ms @ 1, 1.86 s @ 16

The KV cache is DeepSeek's fp8_ds_mla compressed-latent format — the architecture accepts no other layout (4-bit KV schemes like TurboQuant don't apply here; there is far less cache to squeeze than on a GQA model to begin with).

No unpruned baseline column: the full model cannot serve on this hardware, which is the point of this artifact — treat the numbers as absolute, not as a delta. Zero request errors across both suites (4.3 h of continuous serving).

What pruning half the experts looks like

Call accuracy held up (81.9 beats several unpruned models on our same-box leaderboard); abstention is the visible casualty (45.0 — the model rarely declines to call a tool). If your workload punishes spurious tool calls, weight that column.

Provenance

• Base: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp (MIT), experts MXFP4 + FP8 attention as shipped.
• REAP plan and rewriter: ludo-tech/DeepSeek-V4-Flash-REAP-145B-A13B tools/v4prune (MIT) — the transplant recomputes only the hash-table remaps against this checkpoint's router.
• Files: 48 weight shards (83.3 GB) + vision_tower.safetensors (vision tower, aligner, image-token embeddings, sliced gate.bias_vl) + tokenizer/config.

---

_{primitive · more models · inference economics for production LLM systems}