---
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
tags:
  - quantization
  - int8
  - bitsandbytes
  - qwen
  - qwen3.5
language:
  - en
pipeline_tag: text-generation
library_name: transformers
---

# Qwen3.5-4B-INT8

**Author:** Prashant Takale

## Model Description

This is an **INT8 quantized** version of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) using **bitsandbytes LLM.int8()** quantization.

### What I Did

- Quantized the original Qwen3.5-4B model from BF16 to INT8 precision using bitsandbytes
- Reduced memory footprint by **42.4%** while maintaining model quality
- Benchmarked on WikiText-2 (perplexity) and GSM8K (math reasoning)

### Why I Did This

1. **Memory Efficiency:** Reduce GPU memory requirements to run on smaller GPUs
2. **Faster Inference:** INT8 operations can be faster on compatible hardware
3. **Accessibility:** Enable deployment on consumer-grade hardware with limited VRAM

## Benchmark Results

| Metric | Baseline (BF16) | INT8 | Change |
|--------|-----------------|------|--------|
| **Memory** | 8.41 GB | 4.84 GB | **-42.4%** |
| **Perplexity** (WikiText-2) | 12.59 | 12.75 | +1.30% |
| **GSM8K Accuracy** | 86.00% | ~86% | Minimal degradation |

### Key Findings

- **1.74x memory reduction** with minimal quality loss
- Only **+1.30% perplexity increase** on WikiText-2
- Math reasoning capabilities preserved on GSM8K benchmark

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AxisQuant/Qwen3.5-4B-INT8",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("prashantcp8/Qwen3.5-4B-INT8")

messages = [{"role": "user", "content": "What is machine learning?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)
```

## Technical Details

- **Quantization Method:** bitsandbytes LLM.int8()
- **Base Model:** Qwen/Qwen3.5-4B
- **Original Precision:** BF16
- **Quantized Precision:** INT8
- **Framework:** Transformers + bitsandbytes


## Limitations

- Requires `bitsandbytes` library for optimal performance
- Some operations may cast to FP16 during inference
- Best suited for NVIDIA GPUs with INT8 tensor core support

## Citation

If you use this model, please cite the original Qwen3.5 model:

```bibtex
@misc{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  year={2025},
  publisher={Alibaba}
}
```

## Author

**Prashant Takale**

---

> **Note:** This model is for testing and experimental purposes only. There are still some improvements required in terms of inference speed and accuracy. I plan to explore better quantization methods like **AWQ** or **GPTQ** in future iterations for optimized performance.