Qwen3-Embedding-4B-NVFP4

Qwen/Qwen3-Embedding-4B quantized to NVFP4 (4-bit weights and 4-bit activations) with llm-compressor, for serving on NVIDIA Blackwell (SM120) under vLLM. Built to back a scripture-retrieval application, so it is additionally patched to expose Matryoshka output dimensions and serve 768-dim vectors directly.

Base model	Qwen/Qwen3-Embedding-4B (Apache-2.0)
Quantization	NVFP4, W4A4
Size on disk	2.7 GB (from 7.6 GB BF16, ~2.8x smaller)
Native embedding dim	2560 (Matryoshka rungs down to 128)
Tooling	llm-compressor 0.12.0.1, compressed-tensors 0.17.1
Checkpoint format	nvfp4-pack-quantized
Verified on	vLLM 0.26.0, torch 2.11.0+cu130, RTX PRO 6000 Blackwell

Quantization details

QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head"])

• Weights — FP4, group_size=16, strategy=tensor_group, symmetric, FP8 (e4m3) group scales.
• Activations — FP4, group_size=16, dynamic="local" per-group scales with a static global FP8 scale fitted from calibration data. This is why the calibration set matters: W4A4 has a data-dependent activation scale, unlike a weights-only scheme.
• Excluded — lm_head only. The model is dense (no MoE), so no router/gate exclusions apply. lm_head is unused by the pooling runner and is tied to the input embeddings.
• Architecture is Qwen3ForCausalLM, so it loads with AutoModelForCausalLM; the embedding behaviour comes from last-token pooling, not a separate head.

Calibration

512 samples, max_seq_length=512, English only:

Source	Samples	Shape
HuggingFaceH4/ultrachat_200k (MIT)	256	instruction / dialogue
google-research-datasets/natural_questions (CC BY-SA 3.0)	128	questions, carrying the query instruction prefix
natural_questions	128	passages, bare

Query-shaped samples carry the model's real query prefix (below) and passages do not, so the calibrated activation scales match the asymmetric way the model is used at inference.

Matryoshka patch — read this if you fork the recipe

Upstream Qwen3-Embedding-4B supports MRL truncation but its config.json declares neither is_matryoshka nor matryoshka_dimensions. vLLM refuses the dimensions pooling parameter unless ModelConfig.is_matryoshka is true, so this checkpoint adds:

{ "is_matryoshka": true,
  "matryoshka_dimensions": [128, 256, 512, 768, 1024, 1536, 2048, 2560] }

vLLM truncates after pooling and renormalizes, so a 768-dim response is unit-norm.

Serving with vLLM

vllm serve ironbcc/Qwen3-Embedding-4B-NVFP4 \
  --served-model-name qwen3-embedding-4b \
  --runner pooling --convert embed \
  --pooler-config '{"dimensions": 768}' \
  --max-model-len 8192

--pooler-config '{"dimensions": 768}' makes every response 768-dim, so OpenAI-compatible clients need no dimensions parameter. Drop it to get the native 2560.

Measured on one Blackwell GPU at gpu_memory_utilization=0.09 (~8.8 GB resident): a 64-text batch (1654 prompt tokens) returns in 57-59 ms, about 0.9 ms/text.

Asymmetric use matters

Embed documents bare, and prefix queries with the string from the base model's config_sentence_transformers.json:

Instruct: Given a web search query, retrieve relevant passages that answer the query
Query: {your query}

This is not cosmetic. On a 31k-verse Bible corpus, "How can I be saved?" retrieves Psalm 55:16 / Genesis 49:18 unprefixed versus Acts 15:11 / Romans 10:10 / Titus 3:5 prefixed; "I am so lonely" goes from Job 7:19 to Psalm 25:16 / 102:7 / 88:8.

Official MTEB subset evaluation

This checkpoint was evaluated with official MTEB 2.18.13 evaluators over five complete English test tasks. This is a targeted subset, not the full MTEB English v2 suite.

Task	Main metric	W4A4	W4A16	W4A16 delta
ArguAna	nDCG@10	0.69040	0.71136	+0.02096
ArXivHierarchicalClusteringP2P	V-measure	0.58969	0.59041	+0.00072
Banking77Classification.v2	Accuracy	0.83228	0.84044	+0.00816
BIOSSES	Cosine Spearman	0.81693	0.82780	+0.01087
CQADupstackGamingRetrieval	nDCG@10	0.63603	0.65026	+0.01423

The W4A16 checkpoint was higher on all five tasks. W4A4 completed the suite in 1,659.43 seconds versus 1,713.96 seconds for W4A16: 39.10 versus 37.85 encoded texts/second, so W4A4 was about 3.3% faster end to end in this run. The benchmark did not record the total number of tokenizer input tokens, so these results cannot support a tokens/second comparison.

Matched protocol

• Same five dataset revisions and complete test splits.
• 64,880 encoded texts in 2,033 API requests; batch size 32.
• 768-dimensional embeddings through the same vLLM/OpenAI-compatible runner.
• Retrieval instruction applied only to queries; documents remained bare.
• Inputs longer than 8,192 tokens were right-truncated identically in both arms.
• MTEB cache disabled, isolated per-arm prediction artifacts, and no result exceptions.
• Runner SHA-256: b3506dc3af9a271c77ef5d3b51b49700f3f3f88aa55b3b7303172292eedf887d.

The raw manifest, MTEB results, and W4A4/W4A16 comparison are included in this repository.

W4A4 versus W4A16 embedding geometry

On 100 matched English texts at 768 dimensions, W4A16 versus W4A4 had mean cosine similarity 0.97610, minimum 0.95516, and mean L2 distance 0.21797. No BF16 baseline was measured in this run, so these numbers must not be interpreted as BF16 fidelity.

Downstream check on the target application: with W4A4 vectors, dense retrieval over a 31,086-verse corpus returns John.3.16 as top-1 for "For God so loved the world" (cos 0.7865) and Psalm.23.1 for "the Lord is my shepherd" (0.7936).

Limitations

• Reported MTEB results cover five English tasks, not the full English v2 or multilingual suites; do not treat them as a leaderboard aggregate.
• No BF16 baseline was measured in the reported run. The geometry comparison is W4A4 versus W4A16 only.
• W4A4 is lossy in a way that shows up directly in embedding space — see above. Validate on your own retrieval set before trusting it in production.
• Blackwell/SM120-oriented. FP4 activation kernels need recent flashinfer; on vLLM 0.26.0 the flashinfer-python==0.6.14 pin has no matching flashinfer-cubin release, which may require FLASHINFER_DISABLE_VERSION_CHECK=1.
• English calibration only. For multilingual retrieval, recalibrate on your languages.
• Inherits the base model's Apache-2.0 license and its biases.

Provenance

Quantized with llm-compressor 0.12.0.1 in an environment isolated from the serving stack (transformers 5.10.1, torch 2.11.0+cu130). Conversion and calibration ran on a single Blackwell GPU. recipe.yaml and the oneshot_*.log in this repo are the actual artifacts emitted by the run.