DeepSeek-R1-Distill-Llama-70B-NVFP4

NVFP4 quantized version of DeepSeek-R1-Distill-Llama-70B using custom Blackwell NVFP4 GEMM kernels.

140 GB → 40 GB (0.29x) with vision tower excluded.

NVFP4 Quantization Details

Property	Value
Base model	deepseek-ai/DeepSeek-R1-Distill-Llama-70B
Quantization	NVFP4 (W4A4 — weights FP4 E2M1, activations FP4, scales FP8 E4M3)
Format	compressed-tensors (native vLLM support)
Tool	vllm-project/llm-compressor v0.10.0.2
Calibration	128 samples, ultrachat-200k (train_sft split), max_seq_length 2048
Size	40 GB (single safetensors shard set)
Requires	NVIDIA Blackwell GPU (SM 120), vLLM >= 0.19

Recipe

QuantizationModifier:
  targets: [Linear]
  ignore: [lm_head]
  scheme: NVFP4

Usage

vLLM

vllm serve PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4 \
  --host 0.0.0.0 \
  --port 8081 \
  --max-model-len 8192

Python

from vllm import LLM, SamplingParams

llm = LLM(model="PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4")
output = llm.generate("What is the meaning of life?", SamplingParams(max_tokens=256))
print(output[0].outputs[0].text)

Benchmarks

Tested on RTX PRO 6000 Blackwell 96GB:

Backend	Generation tok/s	Prompt tok/s
vLLM 0.19.0	25.0	176.3
llama.cpp (GGUF variant)	33.6	196.5

GGUF Version

A GGUF version of this model is available at PiehSoft/DeepSeek-R1-Distill-Llama-70B-NVFP4-GGUF for use with llama.cpp.

Credits

Quantized by PiehSoft (William Pieh) on NVIDIA RTX PRO 6000 Blackwell 96GB.