|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: Qwen/Qwen3-0.6B |
|
|
tags: |
|
|
- quantization |
|
|
- neural-compressor |
|
|
- qat |
|
|
- quantization-aware-training |
|
|
- qwen3 |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Qwen3-0.6B Quantized with QAT |
|
|
|
|
|
This model is a quantized version of `Qwen/Qwen3-0.6B` using **Quantization Aware Training (QAT)** with Intel Neural Compressor. |
|
|
|
|
|
## 🚀 Model Details |
|
|
|
|
|
- **Base Model**: Qwen/Qwen3-0.6B |
|
|
- **Quantization Method**: Quantization Aware Training (QAT) |
|
|
- **Framework**: Intel Neural Compressor |
|
|
- **Model Size**: Significantly reduced from original |
|
|
- **Performance**: Maintains quality while improving efficiency |
|
|
|
|
|
## 📊 Benefits |
|
|
|
|
|
✅ **Smaller model size** - Reduced storage requirements |
|
|
✅ **Faster inference** - Optimized for deployment |
|
|
✅ **Lower memory usage** - More efficient resource utilization |
|
|
✅ **Maintained quality** - QAT preserves model performance |
|
|
|
|
|
## 💻 Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
# Load the quantized model |
|
|
model = AutoModelForCausalLM.from_pretrained("Thomaschtl/qwen3-0.6b-qat-test") |
|
|
tokenizer = AutoTokenizer.from_pretrained("Thomaschtl/qwen3-0.6b-qat-test") |
|
|
|
|
|
# Generate text |
|
|
prompt = "The future of AI is" |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_length=100, do_sample=True, temperature=0.7) |
|
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
## ⚙️ Quantization Details |
|
|
|
|
|
- **Training Method**: Quantization Aware Training |
|
|
- **Optimizer**: AdamW |
|
|
- **Learning Rate**: 5e-5 |
|
|
- **Batch Size**: 2 |
|
|
- **Epochs**: 1 (demo configuration) |
|
|
|
|
|
## 🔧 Technical Info |
|
|
|
|
|
This model was quantized using Intel Neural Compressor's QAT approach, which: |
|
|
1. Simulates quantization during training |
|
|
2. Allows model weights to adapt to quantization |
|
|
3. Maintains better accuracy than post-training quantization |
|
|
|
|
|
## 📝 Citation |
|
|
|
|
|
If you use this model, please cite: |
|
|
|
|
|
``` |
|
|
@misc{qwen3-qat, |
|
|
title={Qwen3-0.6B Quantized with QAT}, |
|
|
author={Thomaschtl}, |
|
|
year={2025}, |
|
|
publisher={Hugging Face}, |
|
|
url={https://huggingface.co/Thomaschtl/qwen3-0.6b-qat-test} |
|
|
} |
|
|
``` |
|
|
|
|
|
## ⚖️ License |
|
|
|
|
|
This model follows the same license as the base model (Apache 2.0). |
|
|
|
