This model has been pushed to the Hub using the PytorchModelHubMixin integration.
- Base Model: Qwen/Qwen2.5-7B-Instruct
- Quantization: INT8 (Optimum Quanto)
- Library: Optimum Quanto
- Framework: PyTorch
- Documentation: https://huggingface.co/docs/optimum/quanto
Qwen2.5-7B-Instruct INT8 (Optimum Quanto)
This repository contains an INT8 quantized version of Qwen2.5-7B-Instruct created using Optimum Quanto.
The objective of this project is to reduce model size and memory usage while maintaining inference quality for research and local deployment.
Model Details
- Base Model: Qwen/Qwen2.5-7B-Instruct
- Quantization Method: Optimum Quanto
- Weight Precision: INT8
- Original Model Size: ~15.2 GB
- Quantized Model Size: ~8.18 GB
- Storage Reduction: ~46%
Usage
from optimum.quanto import QuantizedModelForCausalLM
from transformers import AutoTokenizer
model = QuantizedModelForCausalLM.from_pretrained(
"YOUR_USERNAME/Qwen2.5-7B-Instruct-INT8"
)
tokenizer = AutoTokenizer.from_pretrained(
"YOUR_USERNAME/Qwen2.5-7B-Instruct-INT8"
)
Benchmark
Benchmarking is currently in progress.
Planned evaluations include:
- HellaSwag
- ARC Challenge
- Winogrande
- GSM8K
Benchmark results will be added after evaluation.
Hardware
Quantization performed on:
- Windows
- Python 3.11
- PyTorch
- NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM)
Limitations
- Quantization may slightly reduce model accuracy depending on the evaluation task.
- Performance should be validated for production workloads.
- GPU memory requirements depend on sequence length and runtime configuration.
License
This repository contains a quantized derivative of Qwen/Qwen2.5-7B-Instruct.
Please follow the original model's license and usage terms when using or redistributing this model.
This model has been pushed to the Hugging Face Hub using:
Framework: Optimum Quanto
Base Model: Qwen/Qwen2.5-7B-Instruct
Quantization: INT8
Library: huggingface_hub + Optimum Quanto
Documentation: https://huggingface.co/docs/optimum/quanto
Author
**Monishwaran K
GitHub: https://github.com/Monishwaran45
Hugging Face: https://huggingface.co/Helllbos ⭐ If you find this model useful, please consider giving it a Like ❤️ on Hugging Face.
- Downloads last month
- 68