Qwen3-VL-4B-Instruct-NVFP4

This is a quantized version of Qwen/Qwen3-VL-4B-Instruct using SmoothQuant + NVFP4 across all text linear layers.

Quantization Strategy

Component	Scheme	Details
All Text Linear Layers	NVFP4	W4A4 (W4A16 via Marlin on Ampere)
Vision Encoder	BF16 (unquantized)	Full precision for visual understanding
LM Head	BF16 (unquantized)	Full precision for output quality

SmoothQuant

Applied with strength 0.8 for activation smoothing before quantization:

• Q/K/V projections ← input_layernorm
• Gate/Up projections ← post_attention_layernorm

Model Details

• Base Model: Qwen/Qwen3-VL-4B-Instruct (4.4B parameters)
• Quantization Method: compressed-tensors (llm-compressor)
• Model Size: ~4.1 GB (reduced from ~8.9 GB BF16, ~54% compression)
• Calibration: 512 samples from flickr30k, max_seq_length=2048

Usage with vLLM

vllm serve JEILDLWLRMA/Qwen3-VL-4B-Instruct-NVFP4 \
    --quantization compressed-tensors \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.9 \
    --max-model-len 8192

from vllm import LLM, SamplingParams

llm = LLM(
    model="JEILDLWLRMA/Qwen3-VL-4B-Instruct-NVFP4",
    quantization="compressed-tensors",
    trust_remote_code=True,
    kv_cache_dtype="fp8",
    max_model_len=8192,
)

sampling_params = SamplingParams(temperature=0.7, top_p=0.95)
prompts = ["Your prompt here"]
outputs = llm.generate(prompts, sampling_params)

Usage with Transformers

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
import torch

model_id = "JEILDLWLRMA/Qwen3-VL-4B-Instruct-NVFP4"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

License

Apache 2.0, same as the base model.