Instructions to use CodeStreet/Qwen-3.5-4B-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CodeStreet/Qwen-3.5-4B-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="CodeStreet/Qwen-3.5-4B-NVFP4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("CodeStreet/Qwen-3.5-4B-NVFP4A16")
model = AutoModelForMultimodalLM.from_pretrained("CodeStreet/Qwen-3.5-4B-NVFP4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use CodeStreet/Qwen-3.5-4B-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CodeStreet/Qwen-3.5-4B-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CodeStreet/Qwen-3.5-4B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CodeStreet/Qwen-3.5-4B-NVFP4A16

SGLang

How to use CodeStreet/Qwen-3.5-4B-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CodeStreet/Qwen-3.5-4B-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CodeStreet/Qwen-3.5-4B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CodeStreet/Qwen-3.5-4B-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CodeStreet/Qwen-3.5-4B-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use CodeStreet/Qwen-3.5-4B-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/CodeStreet/Qwen-3.5-4B-NVFP4A16
```

Qwen-3.5-4B-NVFP4A16

Weight-only NVFP4A16 quantization of Qwen/Qwen3.5-4B (revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a), produced with llm-compressor 0.12.0.

This is not a fine-tune — the weights are the official Qwen3.5-4B weights, quantized. It is packaged in the Qwen3_5ForConditionalGeneration layout; the vision tower is bundled unquantized and unused when served with --language-model-only.

Scheme

4-bit NVFP4 weights (E2M1 with an FP8 scale per block of 16), bf16 activations. input_activations is null, so vLLM selects CompressedTensorsW4A16Fp4 → MarlinNvFp4LinearKernel, not the W4A4 path that most public NVFP4 checkpoints take.

Left in bf16 via ignore: lm_head, embed_tokens, the full visual tower, every linear_attn (GDN) projection, and mtp. Only self_attn and mlp in the full-attention layers are quantized.

GPTQModifier:
  targets: [Linear]
  scheme: NVFP4A16
  block_size: 128
  dampening_frac: 0.01
  actorder: static
  ignore: ['re:.*lm_head', 're:.*embed_tokens', 're:.*visual.*',
           're:.*linear_attn.*', 're:^mtp.*']

Calibration

512 prompts sampled from a 2342-prompt pool of short real-world chat messages wrapped in a few-shot language-identification prompt — not C4 or Pile. Locale mix of the pool: es 1360, en 471, und 131, ar 93, de 68, uk 47, ru 36, and a long tail. Length buckets skew short (860 samples at 11-20 chars, 541 at 4-10). Sequence length 512.

Evaluation

Language identification, top-1 accuracy, greedy, scores read from first-token logprobs. Compared against the same base model served with vLLM's online per-tensor FP8 forced into Marlin W8A16 (VLLM_TEST_FORCE_FP8_MARLIN=1).

checkpoint	uk/ru bench (n=347)	hard-cases bench (n=1309)
this (NVFP4A16)	330 (95.10%)	1146 (87.55%)
base, online FP8 W8A16	333 (95.97%)	1144 (87.39%)
int8 W8A16 group-32	329 (94.81%)	1144 (87.39%)
int4 W4A16 group-128	331 (95.39%)	1135 (86.71%)
NVFP4 W4A4	326 (93.95%)	1132 (86.48%)
bf16, no quantization	329 (94.81%)	—

No pair differs significantly (McNemar exact, all p > 0.4; vs the FP8 baseline p=0.45 and p=0.86). Notably bf16 does not top the table — on this task the spread across schemes is measurement noise on the uk↔ru boundary, so treat the ordering above as ties, not a ranking.

Both benches are disjoint from the calibration pool.

Serving

vllm serve CodeStreet/Qwen-3.5-4B-NVFP4A16 \
  --language-model-only --max-model-len 512 --max-logprobs 128 \
  --gpu-memory-utilization 0.12 --trust-remote-code

gpu-memory-utilization needs more headroom than a fully-quantized NVFP4 checkpoint: the conservative ignore list keeps ~3.3 GB in bf16 (GDN projections 2.0 GB, embed_tokens 1.3 GB). At 0.07 on a 96 GB card the engine fails with max_num_seqs exceeds available Mamba cache blocks. Requires Blackwell (sm_120) or newer for the NVFP4 kernels.

Downloads last month: 22

Safetensors

Model size

5B params

Tensor type

BF16

Model tree for CodeStreet/Qwen-3.5-4B-NVFP4A16

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Quantized

(324)

this model