File size: 1,727 Bytes
54c8299 c4fae7f dec5608 c4fae7f 54c8299 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
---
license: mit
base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
tags:
- deepseek
- r1
- qwen
- 4bit
- bitsandbytes
- reasoning
language:
- en
- zh
pipeline_tag: text-generation
library_name: transformers
---
# DeepSeek-R1-Distill-Qwen-7B-4bit
## Overview
This repository contains a 4-bit quantized version of **[DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)**.
The model is distilled from the original DeepSeek-R1 and uses the Qwen-2.5-7B architecture. It is quantized using `bitsandbytes` (NF4) to run on GPUs with ~5.5GB - 6GB VRAM.
## Model Highlights
- **Reasoning Capabilities:** Distilled from DeepSeek-R1, providing superior logical and mathematical performance for its size.
- **Architecture:** Based on Qwen2.5-7B.
- **Quantization:** 4-bit NormalFloat (NF4) for optimized memory usage.
## Usage
**Install Requirements:**
```bash
pip install -U transformers -U bitsandbytes>=0.46.1
```
**Use the model with transformers:**
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "Pxsoone/DeepSeek-R1-Distill-Qwen-7B-4bit"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype=torch.float16
)
prompt = "Solve this puzzle: If I have 3 apples and you take away 2, how many apples do you have?"
messages = [
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1000)
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |