leyili6666 commited on
Commit
47b4732
·
verified ·
1 Parent(s): 7ab5849

Simplify published comparisons to complete benchmark configurations

Browse files

Use six compact benchmark rows with twelve similarly sized published-model references. Remove item-count annotations and retain comparisons backed by complete benchmark configurations. Preserve the ScienceAccelBench section unchanged.

Files changed (1) hide show
  1. README.md +1 -5
README.md CHANGED
@@ -54,18 +54,14 @@ Scores (%), grouped by benchmark and model size. Each reference entry gives its
54
 
55
  | PhAI-IDE | Benchmark | Score | Reference models: score (difference) |
56
  | --- | --- | ---: | --- |
57
- | 4B | GSM8K | **89.06†** | [Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct#instruction-tuned-models) (3B): 77.7 (**+11.36**); [Phi-3-mini](https://arxiv.org/html/2404.14219v1) (3.8B): 82.5 (**+6.56**); [Qwen2.5-3B-Instruct](https://arxiv.org/html/2412.15115v2#S5.SS2.SSS1) (3B): 86.7 (**+2.36**); [WizardMath-Llama-RL (Llama 3.2)](https://arxiv.org/html/2308.09583v2) (3B): 85.5 (**+3.56**) |
58
  | 4B | BBH multistep-arithmetic-two | **97.60** | [Llama-3.2-3B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.21B): 53.2 (**+44.40**); [Phi-3.5-mini-8k-instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.82B): 95.6 (**+2.00**) |
59
- | 4B | MATH-500 | **87.50†** | [Llama-3.2-3B-Instruct](https://huggingface.co/microsoft/Phi-4-mini-reasoning/blob/661e72224c678e7360429603fd898712ee0c74b9/README.md) (3B): 44.4 (**+43.10**); [Phi-4-Mini](https://huggingface.co/microsoft/Phi-4-mini-reasoning/blob/661e72224c678e7360429603fd898712ee0c74b9/README.md) (3.8B): 71.8 (**+15.70**) |
60
- | 9B | GSM8K | **92.97†** | [Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct#instruction-tuned-models) (8B): 84.5 (**+8.47**); [Qwen2.5-7B-Instruct](https://arxiv.org/html/2412.15115v2#S5.SS2.SSS1) (7B): 91.6 (**+1.37**); [Gemma-2-9B-IT](https://arxiv.org/html/2412.15115v2#S5.SS2.SSS1) (9B): 76.7 (**+16.27**); [SciTulu-7B](https://arxiv.org/html/2406.07835v2#A3) (7B): 29.5 (**+63.47**) |
61
  | 9B | BBH word-sorting | **60.40** | [Llama-3.1-8B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (8.03B): 51.2 (**+9.20**); [Qwen2.5-7B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (7.62B): 15.6 (**+44.80**) |
62
  | 9B | MATH-500 | **92.20** | [InternLM3-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 83 (**+9.20**); [Qwen2.5-7B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (7B): 72.4 (**+19.80**); [Llama-3.1-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 48.4 (**+43.80**) |
63
- | 72B | GSM8K | **93.75†** | [Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct#instruction-tuned-models) (70B): 93 (**+0.75**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM#3-evaluation-results) (67B): 84.1 (**+9.65**); [Qwen2-72B-Instruct](https://arxiv.org/html/2412.15115v2#S5.SS2.SSS1) (72B): 93.2 (**+0.55**); [SciTulu-70B](https://arxiv.org/html/2406.07835v2#A3) (70B): 67.5 (**+26.25**); [WizardMath-Llama-RL (Llama 2)](https://arxiv.org/html/2308.09583v2) (70B): 92.8 (**+0.95**) |
64
  | 72B | AQuA-RAT | **77.56** | [Llama-2-70B-Chat](https://openreview.net/pdf?id=FvfhHucpLd) (70B): 31.32 (**+46.24**) |
65
  | 72B | ARC-Easy | **84.64** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 76.5 (**+8.14**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 81.6 (**+3.04**) |
66
  | 72B | ARC-Challenge | **64.42** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 59.5 (**+4.92**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 64.1 (**+0.32**) |
67
 
68
- † PhAI-IDE score on a 128-item subset. Reference scores come from the linked publications, model cards, and independent evaluation reports; evaluation settings and sample counts vary by source. Differences describe reported scores across evaluations, rather than matched-protocol head-to-head gains. BBH entries refer to the named tasks.
69
 
70
  ## Quick start
71
 
 
54
 
55
  | PhAI-IDE | Benchmark | Score | Reference models: score (difference) |
56
  | --- | --- | ---: | --- |
 
57
  | 4B | BBH multistep-arithmetic-two | **97.60** | [Llama-3.2-3B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.21B): 53.2 (**+44.40**); [Phi-3.5-mini-8k-instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.82B): 95.6 (**+2.00**) |
 
 
58
  | 9B | BBH word-sorting | **60.40** | [Llama-3.1-8B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (8.03B): 51.2 (**+9.20**); [Qwen2.5-7B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (7.62B): 15.6 (**+44.80**) |
59
  | 9B | MATH-500 | **92.20** | [InternLM3-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 83 (**+9.20**); [Qwen2.5-7B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (7B): 72.4 (**+19.80**); [Llama-3.1-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 48.4 (**+43.80**) |
 
60
  | 72B | AQuA-RAT | **77.56** | [Llama-2-70B-Chat](https://openreview.net/pdf?id=FvfhHucpLd) (70B): 31.32 (**+46.24**) |
61
  | 72B | ARC-Easy | **84.64** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 76.5 (**+8.14**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 81.6 (**+3.04**) |
62
  | 72B | ARC-Challenge | **64.42** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 59.5 (**+4.92**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 64.1 (**+0.32**) |
63
 
64
+ Reference scores come from the linked publications, model cards, and independent evaluation reports; evaluation settings and sample counts vary by source. Differences describe reported scores across evaluations, rather than matched-protocol head-to-head gains. BBH entries refer to the named tasks.
65
 
66
  ## Quick start
67