metadata
library_name: transformers
license: mit
base_model:
- deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
base_model_relation: quantized
GPTQ 8bit quantized version of DeepSeek-R1-Distill-Qwen-7B
Model Details
See details on the official page of the model: DeepSeek-R1-Distill-Qwen-7B
Quantized using GPTQModel using Allenai/C4 dataset. Quantization config:
bits=8,
group_size=128,
desc_act=False,
How to use
Using transformers library with integrated GPTQ support:
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
model_name = "frankdarkluo/DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int8"
tokenizer = AutoTokenizer.from_pretrained(model_name)
quantized_model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda')
chat = [{"role": "user", "content": "Why is grass green?"},]
question_tokens = tokenizer.apply_chat_template(chat, add_generation_prompt=True, return_tensors="pt").to(quantized_model.device)
answer_tokens = quantized_model.generate(question_tokens, generation_config=GenerationConfig(max_length=2048, ))[0]
print(tokenizer.decode(answer_tokens))