andrebadini/Qwen3-Embedding-4B-NVFP4

Qwen3-Embedding-4B (NVFP4 quantized)

Quantization of Qwen/Qwen3-Embedding-4B to NVFP4 (NVIDIA FP4 with FP8 micro-block scales) for Blackwell GPUs (RTX 50-series, B200).

Format

• Quantization scheme: NVFP4 — weights in FP4 E2M1, scales in FP8 E4M3 per group of 16 elements
• Storage: compressed-tensors (nvfp4-pack-quantized)
• Layers quantized: all Linear layers except lm_head
• Original size (BF16): ~8 GB
• Quantized size: 2.7 GB (~3.4x compression)

Usage

This model is in compressed-tensors format. To load on a Blackwell GPU:

import torch
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "andrebadini/Qwen3-Embedding-4B-NVFP4",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("andrebadini/Qwen3-Embedding-4B-NVFP4", trust_remote_code=True)

inputs = tok("Your text here", return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model(**inputs)
# last-token pooling (matches Qwen3-Embedding-4B default)
emb = outputs.last_hidden_state[:, -1, :]
print(emb.shape)  # torch.Size([1, 2560])

Inference server (vLLM)

Recommended (works out of the box on Blackwell):

vllm serve andrebadini/Qwen3-Embedding-4B-NVFP4 \
  --runner pooling \
  --convert embed \
  --trust-remote-code

The --convert embed flag tells vLLM to run in embedding mode; the OpenAI-compatible endpoint becomes POST /v1/embeddings.

Quantization recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: NVFP4

Calibrated oneshot on 20 samples from neuralmagic/calibration (LLM split).

Hardware notes

• NVFP4 tensor cores require Blackwell (SM 120). On Hopper (H100) or older, kernels will JIT-compile through CUTLASS via FlashInfer (slower first run, cached after).
• The flashinfer cache (~/.cache/flashinfer/0.6.12/120f/cached_ops/fp4_gemm_cutlass_sm120/) holds the JIT kernels; keep it across runs to avoid recompilation.

Limitations

• LM Studio does not currently load compressed-tensors format. Use transformers directly or run via vLLM / TGI.
• Other inference engines (llama.cpp, exllamav2, etc.) do not yet support NVFP4.

License

Inherits Apache 2.0 from the original Qwen3-Embedding-4B.