Instructions to use Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound")
model = AutoModelForMultimodalLM.from_pretrained("Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound

SGLang

How to use Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound with Docker Model Runner:
```
docker model run hf.co/Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound
```

Qwen3.5-9B Mixed-INT4 AutoRound

This repository contains a Mixed-INT4 quantized version of Qwen/Qwen3.5-9B, produced using Intel AutoRound.

The model weights were quantized to reduce memory requirements while preserving the original Qwen3.5 architecture, tokenizer, configuration, and multimodal capabilities.

Model Information

Base model: Qwen/Qwen3.5-9B
Model type: Multimodal causal language model with vision encoder
Quantization: Mixed INT4
Quantization framework: Intel AutoRound
Weight format: Safetensors
Native context length: 262,144 tokens
License: Apache 2.0

Important Notice

This repository is an unofficial community quantization of Qwen3.5-9B.

The model architecture and original model behavior are provided by the Qwen team. Quantization may cause small differences in output quality, numerical precision, generation consistency, and benchmark performance compared with the original model.

No independent benchmark results are currently provided for this quantized version.

evaluation

Evaluation was performed using AutoRound’s evaluation CLI, powered by LM Evaluation Harness.

Benchmark	Metric	Qwen3.5-9B	Qwen3.5-9B-MixedInt4-AutoRound	Difference	Recovery Rate
MMLU	acc	78.66%	77.62%	-1.04%p	98.68%
ARC-Challenge	acc_norm	55.80%	55.03%	-0.77%p	98.62%
BoolQ	acc	89.17%	86.91%	-2.26%p	97.47%
HellaSwag	acc_norm	78.15%	77.45%	-0.70%p	99.10%
PIQA	acc_norm	80.03%	80.20%	+0.17%p	100.21%
WinoGrande	acc	73.01%	71.82%	-1.19%p	98.37%
Average	—	75.80%	74.84%	-0.97%p	98.73%

MMLU Category	Qwen3.5-9B	Qwen3.5-9B-MixedInt4-AutoRound	Difference	Recovery Rate
Humanities	70.48%	68.93%	-1.55%p	97.80%
Other	83.20%	82.52%	-0.68%p	99.18%
Social Sciences	86.90%	86.55%	-0.35%p	99.60%
STEM	78.34%	77.07%	-1.27%p	98.38%

Serving with vLLM

If the installed vLLM version supports this model architecture and AutoRound quantization format, the model can be served using:

vllm serve YOUR_USERNAME/Qwen3.5-9B-MixedInt4-AutoRound \
  --trust-remote-code

Support for newly released model architectures and quantization formats may require a recent development build of vLLM.

Quantization Details

Method: Mixed-INT4 AutoRound quantization
Source weights: Qwen/Qwen3.5-9B
Fine-tuning: None
Architecture modifications: None intended

Exact quantization settings, calibration dataset, group size, and AutoRound version should be documented here when available.

Limitations

This model inherits the limitations of the original Qwen3.5-9B model.

Additional limitations may result from quantization:

Reduced numerical precision
Small changes in generated responses
Possible degradation on sensitive reasoning or vision-language tasks
Runtime compatibility differences between inference frameworks
Potential differences in long-context behavior

Users should evaluate the model on their own workloads before production use.

Original Model

For complete information about the architecture, supported languages, context length, multimodal usage, benchmarks, intended uses, and limitations, refer to the original model card:

Qwen/Qwen3.5-9B

License

The original Qwen3.5-9B model is distributed under the Apache License 2.0.

This quantized repository follows the license and usage requirements of the original model. Users are responsible for reviewing and complying with the original license terms.

Credits

Original model: Qwen
Quantization framework: Intel AutoRound

Downloads last month: 23

Safetensors

Model size

4B params

Tensor type

I32

BF16

F16

Model tree for Pilcothink/Qwen3.5-9B-MixedInt4-AutoRound

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

Qwen/Qwen3.5-9B

Quantized

(444)

this model