Instructions to use axteam/Qwen3.5-4B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/Qwen3.5-4B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="axteam/Qwen3.5-4B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("axteam/Qwen3.5-4B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("axteam/Qwen3.5-4B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/Qwen3.5-4B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/Qwen3.5-4B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/Qwen3.5-4B-NVFP4

SGLang

How to use axteam/Qwen3.5-4B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/Qwen3.5-4B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/Qwen3.5-4B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/Qwen3.5-4B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use axteam/Qwen3.5-4B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/axteam/Qwen3.5-4B-NVFP4
```

Qwen3.5-4B · NVFP4 (W4A4, FP4)

4-bit quantization of Qwen/Qwen3.5-4B, produced for efficient vLLM serving. Quantized by axteam.


Base model	`Qwen/Qwen3.5-4B`
Method	NVFP4 (W4A4, FP4)
Format	compressed-tensors `nvfp4-pack-quantized`
Weights / activations	W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations
Architecture	Hybrid (linear-attention + full-attention) · Dense · multimodal (vision) · MTP head
Size on disk	~5.6 GB
Calibration	256 samples (Korean-heavy mixed corpus)
License	Apache-2.0

Quantization details

Scheme — W4A4 — FP4 (E2M1) weights group 16 (FP8 E4M3 scale + per-tensor FP32 global scale), FP4 activations, stored as compressed-tensors nvfp4-pack-quantized.
Kept in BF16 — the vision tower & projector (model.visual.*), the multi-token-prediction head (mtp.*), all RMSNorms, token embeddings, the LM head, the entire linear-attention (GatedDeltaNet/SSM) path — FP4 is too coarse for the SSM weights, so the full linear_attn.* stays BF16. Left unquantized to preserve quality (about 44% of all parameters in this build).
Tokenizer / chat_template / generation_config are identical to the original. config.json is the original plus a quantization_config block.

This is post-training quantization (PTQ) — no retraining. AWQ uses a small calibration set (no gradient) to protect salient weight channels before rounding.

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic	AWQ-8bit	NVFP4A16	NVFP4 (this repo)	AWQ-4bit
MMLU (5-shot)	55.18	57.99 (105.1%)	54.79 (99.3%)	60.58 (109.8%)	58.61 (106.2%)	60.89 (110.3%)
GSM8K (5-shot, strict)	74.07	73.92 (99.8%)	73.84 (99.7%)	76.72 (103.6%)	73.31 (99.0%)	76.42 (103.2%)
ARC-Challenge (0-shot)	54.61	53.92 (98.8%)	54.52 (99.8%)	54.86 (100.5%)	53.92 (98.8%)	54.35 (99.5%)
Winogrande (5-shot)	68.75	67.96 (98.9%)	69.14 (100.6%)	68.90 (100.2%)	69.38 (100.9%)	68.90 (100.2%)
KMMLU (5-shot, direct)	50.47	50.13 (99.3%)	50.48 (100.0%)	51.49 (102.0%)	50.26 (99.6%)	49.02 (97.1%)
HAE-RAE (0-shot)	62.42	61.50 (98.5%)	62.05 (99.4%)	61.69 (98.8%)	59.76 (95.7%)	58.48 (93.7%)
Average	60.92	60.90 (100.0%)	60.81 (99.8%)	62.37 (102.4%)	60.87 (99.9%)	61.34 (100.7%)

Evaluation (measured)

Fidelity to the BF16 original on an internal mixed English/Korean set (100 prompts, BF16 greedy answers scored under each build — same harness for every row).

Metric	This model	BF16 original
Perplexity (lower = closer)	1.7157	1.5703 (+9.3%)
Top-1 next-token agreement	88.1%	99.2%

These measure fidelity to the base model, not absolute task scores.

Requirements


Runtime	vLLM — the architecture is in vLLM ≥ 0.17.0; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	Full FP4 (W4A4) speed on Blackwell — B200/GB200 (sm100), B300 (sm103), RTX 50xx / RTX PRO 6000 (sm120). Turing–Hopper (sm75–90) fall back to weight-only FP4 (slower, vLLM warns). V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 5.6 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU.
Transformers (without vLLM)	`transformers>=5.2.0` + `compressed-tensors>=0.15.0` (checked on CPU with Qwen3.5-0.8B FP8 / AWQ-4bit / NVFP4A16 / NVFP4).
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/Qwen3.5-4B-NVFP4 --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/Qwen3.5-4B-NVFP4",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/Qwen3.5-4B-NVFP4", max_model_len=32768)
params = SamplingParams(max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

# image input (URL or data: URI)
messages = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "https://example.com/your-image.png"}},
    {"type": "text", "text": "Describe this image."},
]}]
out = llm.chat(messages, params)
print(out[0].outputs[0].text)

Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body). Text-only serving: add --limit-mm-per-prompt '{"image": 0, "video": 0}' to skip loading the vision encoder's memory budget.

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/Qwen3.5-4B-NVFP4 --kv-cache-dtype fp8

License

Quantized derivative of Qwen/Qwen3.5-4B (Apache-2.0). All original weights, tokenizer and chat template belong to the Qwen team.

한국어 요약

Qwen/Qwen3.5-4B 를 NVFP4 (W4A4, FP4) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.

방식 — 가중치·활성화 모두 FP4(W4A4). Blackwell(B200) FP4 텐서코어에서 제 속도. 품질 우선이면 NVFP4A16 권장. 형식 compressed-tensors nvfp4-pack-quantized.
BF16 유지 — 비전 타워·MTP 헤드·모든 norm·임베딩/LM 헤드·SSM(linear_attn) 경로 전체(FP4가 SSM 가중치엔 너무 거칠어서).
PTQ — 재학습 없음.
품질(실측) — BF16 원본 대비 perplexity 1.7157(원본 1.5703, +9.3%), top-1 일치도 88.1%(원본 99.2%). 내부 영/한 100문항, 같은 하네스.
벤치마크(실측) — MMLU · GSM8K · ARC-Challenge · Winogrande · KMMLU · HAE-RAE 평균 BF16 60.92 → 이 모델 60.87 (회복률 99.9%). lm-evaluation-harness · vLLM, 모든 형식 같은 설정.
라이선스 — 원본 Apache-2.0 파생물. 가중치·토크나이저·chat_template 권리는 Qwen 팀.
요구사항 — vLLM ≥0.17.0 (0.32.0 에서 실측); GPU FP4(W4A4) 제 속도는 Blackwell(B200·B300·RTX 50xx/PRO 6000). Turing~Hopper 는 가중치 전용 FP4 로 자동 전환(느림); 가중치 5.6 GB + KV 캐시; Transformers 직접 사용: transformers ≥5.2.0 + compressed-tensors ≥0.15.0.
실행 — vllm serve axteam/Qwen3.5-4B-NVFP4 --max-model-len 32768 로 OpenAI 호환 서버(포트 8000), 또는 Python LLM(model="axteam/Qwen3.5-4B-NVFP4").chat(...) (위 예시). thinking 끄기: chat_template_kwargs={"enable_thinking": False}. 이미지 입력은 image_url 메시지로.
KV 캐시 절감(런타임) — --kv-cache-dtype fp8.