Shashwat42/Qwen3.5-9B-FP8

Qwen3.5-9B VLM — FP8 W8A8 Quantization

FP8 dynamic (W8A8) quantization of Qwen/Qwen3.5-9B produced with llmcompressor on a B200 GPU.

Quantization Details

Property	Value
Method	FP8_DYNAMIC (W8A8)
Tool	llmcompressor QuantizationModifier
Format	compressed-tensors
Vision encoder	BF16 (not quantized)
MTP head	BF16 (not quantized)
Calibration	allenai/c4 (512 samples, seq_len 512)
Hardware	NVIDIA B200 (Blackwell)

Benchmarks (BF16 vs FP8)

Evaluated on a B200 GPU with AutoModelForImageTextToText (full VLM).

Metric	BF16	FP8	Delta
Perplexity ↓ (wikitext-2)	8.17	7.97	-0.19
ARC-Challenge 0-shot % ↑	44.3	44.3	+0.0

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "Shashwat42/Qwen3.5-9B-FP8",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("Shashwat42/Qwen3.5-9B-FP8", trust_remote_code=True)

vLLM

vllm serve Shashwat42/Qwen3.5-9B-FP8 --quantization compressed-tensors

Files

• model.safetensors — FP8 compressed-tensors weights (~12 GB)
• recipe.yaml — llmcompressor quantization recipe
• Processor / tokenizer files included