Hugging Face's logo

ISTA-DASLab
/
DeepSeek-R1-GPTQ-4b-128g-experts

Text Generation
Transformers
Safetensors
deepseek_v3
conversational
custom_code
text-generation-inference
compressed-tensors
Model card Files
xet
Community
2
ekurtic commited on
Commit
ce3a5b4
·
verified ·
1 Parent(s): 3d3004e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -14
README.md CHANGED
@@ -29,20 +29,23 @@ Model outputs were generated with the vLLM engine.
29
 
30
  For reasoning tasks we estimate pass@1 based on 10 runs with different seeds and `temperature=0.6`, `top_p=0.95` and `max_new_tokens=32768`.
31
 
32
-
33
- `OpenLLM v1 `
34
- | Model | ArcC | GSM8k | Hellaswag | MMLU | TruthfulQA-mc2 | Winogrande | Average | Recovery |
35
- |-------------------------------|---------------|-------|-----------|------|------------|------------|---------------|----------|
36
- | deepseek-ai/DeepSeek-R1 | 72.53 | 95.91 | 89.83 | 87.22 | 59.28 | 82.00 | 81.04 | 100.00 |
37
- | cognitivecomputations/DeepSeek-R1-AWQ | 73.12 | 95.15 | 89.07 | 86.86| 60.09 | 82.32 | 81.10 | 100.07 |
38
- | ISTA-DASLab/DeepSeek-R1-GPTQ-4b-128g-act_order-mse_scale-experts (this) | 72.53 | 95.68 | 89.36 | 86.99| 59.77 | 83.35 | 81.28 | 100.30 |
39
-
40
- `Reasoning tasks`
41
- | Model | AIME-2024 pass@1 | MATH-500 pass@1 | GPQA-Diamond pass@1 | Average | Recovery |
42
- |-----------------------------------------|------------------|-----------------|---------------------|---------|----------|
43
- | deepseek-ai/DeepSeek-R1 | 78.34 | 97.24 | 73.383 | 82.99 | 100.00 |
44
- | cognitivecomputations/DeepSeek-R1-AWQ | 70.67 | 93.64 | 70.456 | 78.25 | 94.29 |
45
- | ISTA-DASLab/DeepSeek-R1-GPTQ-4b-128g-act_order-mse_scale-experts (this) | 77.00 | 97.08 | 71.92 | 82.00 | 98.81 |
 
 
 
46
 
47
  ## Reproduction
48
 
 
29
 
30
  For reasoning tasks we estimate pass@1 based on 10 runs with different seeds and `temperature=0.6`, `top_p=0.95` and `max_new_tokens=32768`.
31
 
32
+ #### OpenLLM Leaderboard V1 tasks
33
+
34
+ | | Recovery (%) | Average Score | ARC-Challenge<br>acc_norm, 25-shot | GSM8k<br>exact_match, 5-shot | HellaSwag<br>acc_norm, 10-shot | MMLU<br>acc, 5-shot | TruthfulQA<br>mc2, 0-shot | WinoGrande<br>acc, 5-shot |
35
+ | :------------------------------------------: | :----------: | :-----------: | :--------------------------------: | :--------------------------: | :----------------------------: | :-----------------: | :-----------------------: | :-----------------------: |
36
+ | deepseek/DeepSeek-R1 | 100.00 | 81.04 | 72.53 | 95.91 | 89.30 | 87.22 | 59.28 | 82.00 |
37
+ | cognitivecomputations/DeepSeek-R1-AWQ | 100.07 | 81.10 | 73.12 | 95.15 | 89.07 | 86.86 | 60.09 | 82.32 |
38
+ | ISTA-DASLab/DeepSeek-R1-GPTQ-4b-128g | 99.86 | 80.93 | 72.70 | 95.68 | 89.25 | 86.83 | 58.77 | 82.32 |
39
+ | ISTA-DASLab/DeepSeek-R1-GPTQ-4b-128g-experts <br> **(this model)**| 100.30 | 81.28 | 72.53 | 95.68 | 89.36 | 86.99 | 59.77 | 83.35 |
40
+
41
+ #### Reasoning tasks (AIME-24, GPQA-Diamond, MATH-500)
42
+
43
+ | | Recovery (%) | Average Score | AIME 2024<br>pass@1 | MATH-500<br>pass@1 | GPQA Diamond<br>pass@1 |
44
+ | -------------------------------------------- | :----------: | :-----------: | :-----------------: | :----------------: | :--------------------: |
45
+ | deepseek/DeepSeek-R1 | 100.00 | 82.99 | 78.33 | 97.24 | 73.38 |
46
+ | cognitivecomputations/DeepSeek-R1-AWQ | 94.29 | 78.25 | 70.67 | 93.64 | 70.46 |
47
+ | ISTA-DASLab/DeepSeek-R1-GPTQ-4b-128g | 96.52 | 80.10 | 72.96 | 97.09 | 70.26 |
48
+ | ISTA-DASLab/DeepSeek-R1-GPTQ-4b-128g-experts <br> **(this model)** | 98.81 | 82.00 | 77.00 | 97.08 | 71.92 |
49
 
50
  ## Reproduction
51