README.md

---
library_name: transformers
tags: []
---

# Qwen-Ar-GEC-4bit

# Qwen-Ar-GEC-4bit

This is a quantized version of **[Qwen-Ar-GEC](https://huggingface.co/CUAIStudents/Qwen-Ar-GEC)**.  
It is smaller in size and optimized for GPU VRAM efficiency.  

For usage examples, please refer to the original **[Qwen-Ar-GEC](https://huggingface.co/CUAIStudents/Qwen-Ar-GEC)** model card.  
Both models are functionally identical, but when loading the 4-bit version you may need to include the following configuration:  

```python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model_name = "Abdo-Alshoki/qwen-ar-gec-v2-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

```
 
⚠️ Note: The model is already quantized. Including the configuration ensures it is loaded correctly and runs as expected.