| | --- |
| | license: mit |
| | library_name: vllm |
| | base_model: |
| | - deepseek-ai/DeepSeek-R1 |
| | pipeline_tag: text-generation |
| | tags: |
| | - deepseek |
| | - neuralmagic |
| | - redhat |
| | - llmcompressor |
| | - quantized |
| | - INT4 |
| | - GPTQ |
| | --- |
| | |
| | # DeepSeek-R1-quantized.w4a16 |
| |
|
| | ## Model Overview |
| | - **Model Architecture:** DeepseekV3ForCausalLM |
| | - **Input:** Text |
| | - **Output:** Text |
| | - **Model Optimizations:** |
| | - **Activation quantization:** None |
| | - **Weight quantization:** INT4 |
| | - **Release Date:** 04/15/2025 |
| | - **Version:** 1.0 |
| | - **Model Developers:** RedHat (Neural Magic) |
| |
|
| |
|
| | ### Model Optimizations |
| |
|
| | This model was obtained by quantizing weights of [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) to INT4 data type. |
| | This optimization reduces the number of bits used to represent weights from 8 to 4, reducing GPU memory requirements (by approximately 50%). |
| | Weight quantization also reduces disk size requirements by approximately 50%. |
| |
|
| |
|
| | ## Deployment |
| |
|
| | This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below. |
| |
|
| | ```python |
| | from vllm import LLM, SamplingParams |
| | from transformers import AutoTokenizer |
| | |
| | model_id = "RedHatAI/DeepSeek-R1-quantized.w4a16" |
| | number_gpus = 8 |
| | |
| | sampling_params = SamplingParams(temperature=0.7, top_p=0.8, max_tokens=256) |
| | |
| | tokenizer = AutoTokenizer.from_pretrained(model_id) |
| | |
| | prompt = "Give me a short introduction to large language model." |
| | |
| | llm = LLM(model=model_id, tensor_parallel_size=number_gpus) |
| | |
| | outputs = llm.generate(prompt, sampling_params) |
| | |
| | generated_text = outputs[0].outputs[0].text |
| | print(generated_text) |
| | ``` |
| |
|
| | vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details. |
| |
|
| |
|
| | ## Evaluation |
| |
|
| | The model was evaluated on the OpenLLM leaderboard task (v1) via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), and on popular reasoning tasks (AIME 2024, MATH-500, GPQA-Diamond) via [LightEval](https://github.com/huggingface/open-r1). |
| | For reasoning evaluations, we estimate pass@1 based on 10 runs with different seeds. |
| |
|
| | <details> |
| | <summary>Evaluation details</summary> |
| |
|
| | **OpenLLM v1** |
| | ``` |
| | lm_eval \ |
| | --model vllm \ |
| | --model_args pretrained="RedHatAI/DeepSeek-R1-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \ |
| | --tasks openllm \ |
| | --batch_size auto |
| | ``` |
| |
|
| | **Reasoning Benchmarks** |
| | ``` |
| | export MODEL_ARGS="pretrained=RedHatAI/DeepSeek-R1-quantized.w4a16,dtype=bfloat16,max_model_length=38768,gpu_memory_utilization=0.8,tensor_parallel_size=8,add_special_tokens=false,generation_parameters={\"max_new_tokens\":32768,\"temperature\":0.6,\"top_p\":0.95,\"seed\":42}" |
| | export VLLM_WORKER_MULTIPROC_METHOD=spawn |
| | lighteval vllm $MODEL_ARGS "custom|aime24|0|0,custom|math_500|0|0,custom|gpqa:diamond|0|0" \ |
| | --custom-tasks src/open_r1/evaluate.py \ |
| | --use-chat-template \ |
| | --output-dir $OUTPUT_DIR |
| | ``` |
| |
|
| | </details> |
| |
|
| | ### Accuracy |
| |
|
| | | | Recovery (%) | deepseek/DeepSeek-R1 | RedHatAI/DeepSeek-R1-quantized.w4a16<br>(this model) | |
| | | --------------------------- | :----------: | :------------------: | :--------------------------------------------------: | |
| | | ARC-Challenge<br>25-shot | 100.00 | 72.53 | 72.53 | |
| | | GSM8k<br>5-shot | 99.76 | 95.91 | 95.68 | |
| | | HellaSwag<br>10-shot | 100.07 | 89.30 | 89.36 | |
| | | MMLU<br>5-shot | 99.74 | 87.22 | 86.99 | |
| | | TruthfulQA<br>0-shot | 100.83 | 59.28 | 59.77 | |
| | | WinoGrande<br>5-shot | 101.65 | 82.00 | 83.35 | |
| | | **OpenLLM v1<br>Average Score** | **100.30** | **81.04** | **81.28** | |
| | | AIME 2024<br>pass@1 | 98.30 | 78.33 | 77.00 | |
| | | MATH-500<br>pass@1 | 99.84 | 97.24 | 97.08 | |
| | | GPQA Diamond<br>pass@1 | 98.01 | 73.38 | 71.92 | |
| | | **Reasoning<br>Average Score** | **98.81** | **82.99** | **82.00** | |
| |
|
