File size: 742 Bytes
35566e4 21b6e66 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | ---
license: mit
base_model:
- deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
---
AWQ 4 bits quantization from DeepSeek-R1-Distill-Qwen-14B commit 123265213609ea67934b1790bbb0203d3c50f54f
```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B"
commit_hash = "123265213609ea67934b1790bbb0203d3c50f54f"
# Download the model and tokenizer at the specific commit hash
model = AutoAWQForCausalLM.from_pretrained(model_name, revision=commit_hash)
tokenizer = AutoTokenizer.from_pretrained(model_name, revision=commit_hash)
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
model.quantize(tokenizer, quant_config=quant_config)
``` |