File size: 4,973 Bytes
6b287f0 dc10718 6b287f0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 | ---
license: apache-2.0
base_model: Qwen/Qwen3.5-9B
base_model_relation: quantized
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- qwen
- qwen3.5
- multimodal
- autoround
- quantization
- int4
---
# Qwen3.5-9B Mixed-INT4 AutoRound
This repository contains a Mixed-INT4 quantized version of
[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B), produced using
[Intel AutoRound](https://github.com/intel/auto-round).
The model weights were quantized to reduce memory requirements while preserving
the original Qwen3.5 architecture, tokenizer, configuration, and multimodal
capabilities.
## Model Information
* **Base model:** [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)
* **Model type:** Multimodal causal language model with vision encoder
* **Quantization:** Mixed INT4
* **Quantization framework:** Intel AutoRound
* **Weight format:** Safetensors
* **Native context length:** 262,144 tokens
* **License:** Apache 2.0
## Important Notice
This repository is an unofficial community quantization of Qwen3.5-9B.
The model architecture and original model behavior are provided by the Qwen
team. Quantization may cause small differences in output quality, numerical
precision, generation consistency, and benchmark performance compared with the
original model.
No independent benchmark results are currently provided for this quantized
version.
## evaluation
Evaluation was performed using AutoRound’s evaluation CLI, powered by LM Evaluation Harness.
| Benchmark | Metric | Qwen3.5-9B | Qwen3.5-9B-MixedInt4-AutoRound | Difference | Recovery Rate |
| ------------- | -------: | ---------: | -----------------------------: | ----------: | ------------: |
| MMLU | acc | 78.66% | 77.62% | -1.04%p | 98.68% |
| ARC-Challenge | acc_norm | 55.80% | 55.03% | -0.77%p | 98.62% |
| BoolQ | acc | 89.17% | 86.91% | -2.26%p | 97.47% |
| HellaSwag | acc_norm | 78.15% | 77.45% | -0.70%p | 99.10% |
| PIQA | acc_norm | 80.03% | 80.20% | +0.17%p | 100.21% |
| WinoGrande | acc | 73.01% | 71.82% | -1.19%p | 98.37% |
| **Average** | — | **75.80%** | **74.84%** | **-0.97%p** | **98.73%** |
| MMLU Category | Qwen3.5-9B | Qwen3.5-9B-MixedInt4-AutoRound | Difference | Recovery Rate |
| --------------- | ---------: | -----------------------------: | ---------: | ------------: |
| Humanities | 70.48% | 68.93% | -1.55%p | 97.80% |
| Other | 83.20% | 82.52% | -0.68%p | 99.18% |
| Social Sciences | 86.90% | 86.55% | -0.35%p | 99.60% |
| STEM | 78.34% | 77.07% | -1.27%p | 98.38% |
## Serving with vLLM
If the installed vLLM version supports this model architecture and AutoRound
quantization format, the model can be served using:
```bash
vllm serve YOUR_USERNAME/Qwen3.5-9B-MixedInt4-AutoRound \
--trust-remote-code
```
Support for newly released model architectures and quantization formats may
require a recent development build of vLLM.
## Quantization Details
* **Method:** Mixed-INT4 AutoRound quantization
* **Source weights:** Qwen/Qwen3.5-9B
* **Fine-tuning:** None
* **Architecture modifications:** None intended
Exact quantization settings, calibration dataset, group size, and AutoRound
version should be documented here when available.
## Limitations
This model inherits the limitations of the original Qwen3.5-9B model.
Additional limitations may result from quantization:
* Reduced numerical precision
* Small changes in generated responses
* Possible degradation on sensitive reasoning or vision-language tasks
* Runtime compatibility differences between inference frameworks
* Potential differences in long-context behavior
Users should evaluate the model on their own workloads before production use.
## Original Model
For complete information about the architecture, supported languages, context
length, multimodal usage, benchmarks, intended uses, and limitations, refer to
the original model card:
* [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)
## License
The original Qwen3.5-9B model is distributed under the Apache License 2.0.
This quantized repository follows the license and usage requirements of the
original model. Users are responsible for reviewing and complying with the
original license terms.
## Credits
* Original model: [Qwen](https://huggingface.co/Qwen)
* Quantization framework: [Intel AutoRound](https://github.com/intel/auto-round)
|