| ---
|
| license: apache-2.0
|
| base_model: Qwen/Qwen3.5-9B
|
| base_model_relation: quantized
|
| library_name: transformers
|
| pipeline_tag: image-text-to-text
|
| tags:
|
| - qwen
|
| - qwen3.5
|
| - multimodal
|
| - autoround
|
| - quantization
|
| - int4
|
| ---
|
|
|
| # Qwen3.5-9B Mixed-INT4 AutoRound
|
|
|
| This repository contains a Mixed-INT4 quantized version of
|
| [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B), produced using
|
| [Intel AutoRound](https://github.com/intel/auto-round).
|
|
|
| The model weights were quantized to reduce memory requirements while preserving
|
| the original Qwen3.5 architecture, tokenizer, configuration, and multimodal
|
| capabilities.
|
|
|
| ## Model Information
|
|
|
| * **Base model:** [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)
|
| * **Model type:** Multimodal causal language model with vision encoder
|
| * **Quantization:** Mixed INT4
|
| * **Quantization framework:** Intel AutoRound
|
| * **Weight format:** Safetensors
|
| * **Native context length:** 262,144 tokens
|
| * **License:** Apache 2.0
|
|
|
| ## Important Notice
|
|
|
| This repository is an unofficial community quantization of Qwen3.5-9B.
|
|
|
| The model architecture and original model behavior are provided by the Qwen
|
| team. Quantization may cause small differences in output quality, numerical
|
| precision, generation consistency, and benchmark performance compared with the
|
| original model.
|
|
|
| No independent benchmark results are currently provided for this quantized
|
| version.
|
|
|
| ## evaluation
|
| Evaluation was performed using AutoRound’s evaluation CLI, powered by LM Evaluation Harness.
|
|
|
| | Benchmark | Metric | Qwen3.5-9B | Qwen3.5-9B-MixedInt4-AutoRound | Difference | Recovery Rate |
|
| | ------------- | -------: | ---------: | -----------------------------: | ----------: | ------------: |
|
| | MMLU | acc | 78.66% | 77.62% | -1.04%p | 98.68% |
|
| | ARC-Challenge | acc_norm | 55.80% | 55.03% | -0.77%p | 98.62% |
|
| | BoolQ | acc | 89.17% | 86.91% | -2.26%p | 97.47% |
|
| | HellaSwag | acc_norm | 78.15% | 77.45% | -0.70%p | 99.10% |
|
| | PIQA | acc_norm | 80.03% | 80.20% | +0.17%p | 100.21% |
|
| | WinoGrande | acc | 73.01% | 71.82% | -1.19%p | 98.37% |
|
| | **Average** | — | **75.80%** | **74.84%** | **-0.97%p** | **98.73%** |
|
|
|
| | MMLU Category | Qwen3.5-9B | Qwen3.5-9B-MixedInt4-AutoRound | Difference | Recovery Rate |
|
| | --------------- | ---------: | -----------------------------: | ---------: | ------------: |
|
| | Humanities | 70.48% | 68.93% | -1.55%p | 97.80% |
|
| | Other | 83.20% | 82.52% | -0.68%p | 99.18% |
|
| | Social Sciences | 86.90% | 86.55% | -0.35%p | 99.60% |
|
| | STEM | 78.34% | 77.07% | -1.27%p | 98.38% |
|
|
|
|
|
|
|
| ## Serving with vLLM
|
|
|
| If the installed vLLM version supports this model architecture and AutoRound
|
| quantization format, the model can be served using:
|
|
|
| ```bash
|
| vllm serve YOUR_USERNAME/Qwen3.5-9B-MixedInt4-AutoRound \
|
| --trust-remote-code
|
| ```
|
|
|
| Support for newly released model architectures and quantization formats may
|
| require a recent development build of vLLM.
|
|
|
| ## Quantization Details
|
|
|
| * **Method:** Mixed-INT4 AutoRound quantization
|
| * **Source weights:** Qwen/Qwen3.5-9B
|
| * **Fine-tuning:** None
|
| * **Architecture modifications:** None intended
|
|
|
| Exact quantization settings, calibration dataset, group size, and AutoRound
|
| version should be documented here when available.
|
|
|
| ## Limitations
|
|
|
| This model inherits the limitations of the original Qwen3.5-9B model.
|
|
|
| Additional limitations may result from quantization:
|
|
|
| * Reduced numerical precision
|
| * Small changes in generated responses
|
| * Possible degradation on sensitive reasoning or vision-language tasks
|
| * Runtime compatibility differences between inference frameworks
|
| * Potential differences in long-context behavior
|
|
|
| Users should evaluate the model on their own workloads before production use.
|
|
|
| ## Original Model
|
|
|
| For complete information about the architecture, supported languages, context
|
| length, multimodal usage, benchmarks, intended uses, and limitations, refer to
|
| the original model card:
|
|
|
| * [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)
|
|
|
| ## License
|
|
|
| The original Qwen3.5-9B model is distributed under the Apache License 2.0.
|
|
|
| This quantized repository follows the license and usage requirements of the
|
| original model. Users are responsible for reviewing and complying with the
|
| original license terms.
|
|
|
| ## Credits
|
|
|
| * Original model: [Qwen](https://huggingface.co/Qwen)
|
| * Quantization framework: [Intel AutoRound](https://github.com/intel/auto-round)
|
| |
