Qwen3-Reranker-4B-seq-cls-NVFP4

NVFP4 (W4A4, FP4 weights and activations, group-16 FP8 scales) quantization of Qwen3-Reranker-4B, in the vLLM-ready sequence-classification layout from danielchalef/Qwen3-Reranker-4B-seq-cls-vllm-fixed.

As far as we know this is the first published NVFP4 quant of a Qwen3 reranker. It exists because, on NVIDIA GB10 (DGX Spark) unified-memory hardware, reranking is a compute-bound prefill workload: weight-only quants (W4A16/GPTQ) and FP8 W8A8 give no prefill speedup there — in our embedding-model bake-off on the same silicon, FP8 was a net loss (~~0.7× bf16) while NVFP4 was the only quantization that actually ran faster (~~2× bf16). This checkpoint brings that same trade to reranking: ~⅓ the memory, substantially higher throughput, with a measured (small) quality cost quantified below.

Quantization recipe

Toolchain: llmcompressor==0.9.0.3 / compressed-tensors==0.13.0 (config format verified to load in vLLM v0.25.0), transformers==4.57.3
Scheme: NVFP4 — FP4 weights + FP4 activations, group_size=16, FP8 (e4m3) scales, static minmax observers, dynamic="local" activation scales
Targets: all Linear layers; the classification head (score) is kept in bf16
Calibration: 256 MS MARCO v1.1 passages formatted with the official Qwen3-Reranker prompt template (mixed relevant/irrelevant pairs), max_seq_length=1024

Serving with vLLM

vllm serve <this-repo> \
  --runner pooling \
  --dtype bfloat16 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.10

No --hf_overrides needed — the seq-cls conversion (classifier_from_token: ["no","yes"], method: from_2_way_softmax) is baked into config.json.

Queries must use the Qwen3-Reranker instruction format. Against the /rerank endpoint:

{
  "model": "...",
  "query": "<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: your query",
  "documents": ["..."]
}

Quality & throughput vs bf16 (NVIDIA GB10, vLLM v0.25.0)

Measured on MS MARCO v1.1 validation, 100 queries × ~8 candidate passages, identical fixed eval set for both backends.

Metric	bf16	NVFP4 (this)
Global score Pearson vs bf16	1.000	0.920
Mean per-query Spearman vs bf16	1.000	0.795
Top-1 agreement vs bf16	100%	56%*
MRR (labeled relevant passage)	0.4362	0.4313 (−1.1% rel)
Throughput (docs/s, sequential)	26.3	34.1 (+30%)
p50 / p95 latency per query	132 / 177 ms	62 / 82 ms
Weights on disk	~8 GB (bf16)	3.4 GB

* Top-1 disagreements are concentrated in near-tied scores between MS MARCO's many near-duplicate passages; the near-identical MRR shows they rarely change which labeled-relevant passage ranks first. Judge quality by MRR + Pearson, not top-1.

Notes

NVFP4 activation quantization requires hardware FP4 support (Blackwell-class GPUs, e.g. GB10/GB200/B200). On older GPUs vLLM falls back to slow emulation or refuses.
Base model: Apache-2.0 (Qwen3-Reranker-4B). This derivative keeps Apache-2.0.