Update README.md
Browse files
README.md
CHANGED
|
@@ -238,6 +238,6 @@ This project may contain trademarks or logos for projects, products, or services
|
|
| 238 |
We include a brief word on methodology here - and in particular, how we think about optimizing prompts. In an ideal world, we would never change any prompts in our benchmarks to ensure it is always an apples-to-apples comparison when comparing different models. Indeed, this is our default approach, and is the case in the vast majority of models we have run to date. For all benchmarks, we consider using the same generation configuration such as max sequence length (32768), the same temperature for the fair comparison.
|
| 239 |
Benchmark datasets
|
| 240 |
We evaluate the model with three of the most popular math benchmarks where the strongest reasoning models are competing together. Specifically:
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
|
|
|
| 238 |
We include a brief word on methodology here - and in particular, how we think about optimizing prompts. In an ideal world, we would never change any prompts in our benchmarks to ensure it is always an apples-to-apples comparison when comparing different models. Indeed, this is our default approach, and is the case in the vast majority of models we have run to date. For all benchmarks, we consider using the same generation configuration such as max sequence length (32768), the same temperature for the fair comparison.
|
| 239 |
Benchmark datasets
|
| 240 |
We evaluate the model with three of the most popular math benchmarks where the strongest reasoning models are competing together. Specifically:
|
| 241 |
+
+ Math-500: This benchmark consists of 500 challenging math problems designed to test the model's ability to perform complex mathematical reasoning and problem-solving.
|
| 242 |
+
+ AIME 2024/AIME 2025: The American Invitational Mathematics Examination (AIME) is a highly regarded math competition that features a series of difficult problems aimed at assessing advanced mathematical skills and logical reasoning. We evaluate the models on the problems from both the year 2024 and the latest year 2025 examinations.
|
| 243 |
+
+ GPQA Diamond: The Graduate-Level Google-Proof Q&A (GPQA) Diamond benchmark focuses on evaluating the model's ability to understand and solve a wide range of mathematical questions, including both straightforward calculations and more intricate problem-solving tasks.
|