gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit

Qwen3-VL-Embedding-8B — AWQ INT4 (W4A16)

4-bit AWQ quantization of Qwen/Qwen3-VL-Embedding-8B, exported in the compressed-tensors format for fast serving with vLLM (Marlin INT4 kernels on Ampere/Ada/Hopper and Jetson Orin sm_87).

The model produces dense multimodal embeddings (text, image, video) in a shared 4096-dim space using last-token pooling. This quant keeps the entire vision tower in BF16 and only quantizes the language decoder, which preserves embedding quality (see Evaluation).

Base model	Qwen/Qwen3-VL-Embedding-8B (8B, qwen3_vl)
Method	AWQ (activation-aware), llm-compressor
Scheme	W4A16, 4-bit int weights / 16-bit activations, asymmetric
Granularity	group-wise, group_size = 128, pack-quantized
Kept in BF16	full vision tower (visual.*, merger, patch_embed), lm_head
Format	compressed-tensors (vLLM-native)
Embedding dim	4096 (Matryoshka: usable down to 64)
Pooling	last-token, include_prompt: true
Size on disk	~5.7 GB (vs ~16 GB BF16, ≈64% smaller)

What is and isn't quantized

Only the language-model Linear layers are quantized to INT4. Quantizing the vision encoder of a VLM embedding model causes severe "cosine drift", so the following are explicitly excluded and remain bit-identical to the BF16 reference:

ignore = ["re:.*lm_head", "re:.*visual.*", "re:.*vision_tower.*",
          "re:.*merger.*", "re:.*patch_embed.*"]

The AWQ smoothing uses the standard Qwen3 mappings (input_layernorm→q/k/v, v→o, post_attention_layernorm→gate/up, up→down). The v_proj→o_proj smoothing is skipped per layer because of grouped-query attention (different head dims) — this is expected and harmless.

Usage

vLLM (recommended — OpenAI-compatible /v1/embeddings)

vllm serve gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit \
  --runner pooling --convert embed \
  --max-model-len 8192 --gpu-memory-utilization 0.40

Notes

• vLLM ≥ 0.15 uses --convert embed (older docs say --task embed).
• Do not pass --load-format fastsafetensors — it regresses multimodal embedding similarity (vLLM issue #33954).
• compressed-tensors quantization is auto-detected; no --quantization flag needed.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")
e = client.embeddings.create(model="gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit",
                             input=["A dog on the beach", "A cat on a sofa"])
print(len(e.data[0].embedding))  # 4096

Transformers / compressed-tensors

import torch
from transformers import AutoConfig, AutoProcessor, Qwen3VLForConditionalGeneration

repo = "gonuit/Qwen3-VL-Embedding-8B-AWQ-4bit"
cfg = AutoConfig.from_pretrained(repo); cfg.tie_word_embeddings = True
model = Qwen3VLForConditionalGeneration.from_pretrained(
    repo, config=cfg, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

msgs = [{"role": "user", "content": [{"type": "text", "text": "Represent this sentence."}]}]
text = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inp = processor(text=[text], return_tensors="pt").to(model.device)
with torch.no_grad():
    h = model(**inp, output_hidden_states=True).hidden_states[-1][:, -1, :]
emb = torch.nn.functional.normalize(h, dim=-1)  # last-token pooling

(compressed-tensors must be installed to load the packed INT4 weights.)

Calibration

Activation statistics were collected with llm-compressor (AWQModifier + QuantizationModifier, duo_scaling, n_grid=20) using the sequential pipeline (one decoder subgraph at a time).

Dataset	lmms-lab/flickr30k (multimodal: image + caption)
Samples	256, batch_size = 1 (M-ROPE → variable vision-token count)
Max sequence	2048 tokens, image max_pixels = 1638400

Evaluation

BF16 vs. this AWQ model, both served through vLLM, comparing /v1/embeddings output on mixed EN/PL text:

Metric	Value
mean cosine(BF16, AWQ), same input	0.973
min cosine	0.966
similar-pair cosine (BF16 → AWQ)	0.77 → 0.75, 0.60 → 0.57
unrelated-pair cosine	~0.20 (preserved)

The similarity geometry is preserved — similar inputs stay close, unrelated inputs stay far — so retrieval ranking matches the BF16 model. (Validation shown here is text-only; the vision tower is unquantized BF16 and runs identically to the base model.)

Reproduce

Quantized with llm-compressor 0.11 on an NVIDIA Jetson AGX Orin (JetPack 6.2, CUDA 12.6). One upstream workaround is required before loading the base model: set tie_word_embeddings = true to handle the missing lm_head.weight shard (Qwen3-VL-Embedding bug #89). The full Dockerized recipe (quantize.py, recipe, serve script) is included as recipe.yaml in this repo.

Limitations

• INT4 introduces ~2-3% per-vector cosine drift vs BF16; for most RAG retrieval this does not change top-k ranking, but exact-similarity-threshold workflows should re-tune thresholds.
• Validation above is text-only.
• Inherits all capabilities and limitations of the base Qwen/Qwen3-VL-Embedding-8B.

License & citation

Released under Apache-2.0, inherited from the base model. Please cite the original Qwen3-VL-Embedding work and credit Qwen/Qwen3-VL-Embedding-8B as the base model.