metadata
library_name: transformers
tags: []
Qwen-Ar-GEC-4bit
Qwen-Ar-GEC-4bit
This is a quantized version of Qwen-Ar-GEC.
It is smaller in size and optimized for GPU VRAM efficiency.
For usage examples, please refer to the original Qwen-Ar-GEC model card.
Both models are functionally identical, but when loading the 4-bit version you may need to include the following configuration:
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
model_name = "Abdo-Alshoki/qwen-ar-gec-v2-4bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
⚠️ Note: The model is already quantized. Including the configuration ensures it is loaded correctly and runs as expected.