|
|
--- |
|
|
pipeline_tag: text-generation |
|
|
base_model: |
|
|
- deepseek-ai/DeepSeek-R1 |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
Results from running `vllm serve RedHatAI/DeepSeek-R1-NVFP4-FP8-BLOCK --tensor-parallel-size=4` on 4 B200s, with `python vllm/tests/evals/gsm8k/gsm8k_eval.py --port 8000`: |
|
|
|
|
|
``` |
|
|
Running GSM8K evaluation: 1319 questions, 5-shot |
|
|
Evaluating: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [01:49<00:00, 12.09it/s] |
|
|
|
|
|
Results: |
|
|
Accuracy: 0.952 |
|
|
Invalid responses: 0.000 |
|
|
Total latency: 109.097 s |
|
|
Questions per second: 12.090 |
|
|
Total output tokens: 124914 |
|
|
Output tokens per second: 1144.985 |
|
|
``` |
|
|
|
|
|
Compare to results with `nvidia/DeepSeek-R1-NVFP4` |
|
|
``` |
|
|
Running GSM8K evaluation: 1319 questions, 5-shot |
|
|
Evaluating: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [01:52<00:00, 11.74it/s] |
|
|
|
|
|
Results: |
|
|
Accuracy: 0.954 |
|
|
Invalid responses: 0.000 |
|
|
Total latency: 112.357 s |
|
|
Questions per second: 11.739 |
|
|
Total output tokens: 128126 |
|
|
Output tokens per second: 1140.344 |
|
|
``` |
