work

README.md

@@ -32,20 +32,17 @@ Reinforcement learning aims to bridge the gap between competence and excellence
 
 |                                  | GLM-5                  | GLM-4.7   | DeepSeek-V3.2 | Kimi K2.5 | Claude Opus 4.5 | Gemini 3 Pro | GPT-5.2 (xhigh) |
 | -------------------------------- | ---------------------- | --------- | ------------- | --------- | --------------- | ------------ | --------------- |
-| Reasoning                        |                        |           |               |           |                 |              |                 |
 | HLE                              | 30.5                   | 24.8      | 25.1          | 31.5      | 28.4            | 37.2         | 35.4            |
 | HLE (w/ Tools)                   | 50.4                   | 42.8      | 40.8          | 51.8      | 43.4*           | 45.8*        | 45.5*           |
 | AIME 2026 I                      | 92.7                   | 92.9      | 92.7          | 92.5      | 93.3            | 90.6         | -               |
 | HMMT Nov. 2025                   | 96.9                   | 93.5      | 90.2          | 91.1      | 91.7            | 93.0         | 97.1            |
 | IMOAnswerBench                   | 82.5                   | 82.0      | 78.3          | 81.8      | 78.5            | 83.3         | 86.3            |
 | GPQA-Diamond                     | 86.0                   | 85.7      | 82.4          | 87.6      | 87.0            | 91.9         | 92.4            |
-| Coding                           |                        |           |               |           |                 |              |                 |
 | SWE-bench Verified               | 77.8                   | 73.8      | 73.1          | 76.8      | 80.9            | 76.2         | 80.0            |
 | SWE-bench Multilingual           | 73.3                   | 66.7      | 70.2          | 73.0      | 77.5            | 65.0         | 72.0            |
-| Terminal-Bench 2.0 (Terminus 2)  | 56.2 / 60.7 (verified) | 41.0      | 39.3          | 50.8      | 59.3            | 54.2         | 54.0            |
-| Terminal-Bench 2.0 (Claude Code) | 56.2 / 61.1 (verified)  | 32.8      | 46.4          | -         | 57.9            | -            | -               |
+| Terminal-Bench 2.0 (Terminus 2)  | 56.2 / 60.7 † | 41.0      | 39.3          | 50.8      | 59.3            | 54.2         | 54.0            |
+| Terminal-Bench 2.0 (Claude Code) | 56.2 / 61.1 †  | 32.8      | 46.4          | -         | 57.9            | -            | -               |
 | CyberGym                         | 43.2                   | 23.5      | 17.3          | 41.3      | 50.6            | 39.9         | -               |
-| Agentic                          |                        |           |               |           |                 |              |                 |
 | BrowseComp                       | 62.0                   | 52.0      | 51.4          | 52 / 60.6 | 37.0            | 37.8         | -               |
 | BrowseComp (w/ Context Manage)   | 75.9                   | 67.5      | 67.6          | 74.9      | 57.8            | 59.2         | 65.8            |
 | BrowseComp-Zh                    | 72.7                   | 66.6      | 65.0          | 62.3      | 62.4            | 66.8         | 76.1            |
@@ -54,7 +51,12 @@ Reinforcement learning aims to bridge the gap between competence and excellence
 | Tool-Decathlon                   | 38.0                   | 23.8      | 35.2          | 27.8      | 43.5            | 36.4         | 46.3            |
 | Vending Bench 2                  | $4,432.12              | $2,376.82 | $1,034.00     | $1,198.46 | $4,967.06       | $5,478.16    | $3,591.33       |
 
+> *: refers to their scores of full set.
+> †: A verified version of Terminal-Bench 2.0 that fixes some ambiguous instructions.
+See footnote for more evaluation details.
+
 ### Footnote
+
 * **Humanity’s Last Exam (HLE) & other reasoning tasks**: We evaluate with a maximum generation length of 131,072 tokens (`temperature=1.0, top_p=0.95, max_new_tokens=131072`). By default, we report the text-only subset; results marked with * are from the full set. We use GPT-5.2 (medium) as the judge model. For HLE-with-tools, we use a maximum context length of 202,752 tokens.
 * **SWE-bench & SWE-bench Multilingual**: We run the SWE-bench suite with OpenHands using a tailored instruction prompt. Settings: `temperature=0.7, top_p=0.95, max_new_tokens=16384`, with a 200K context window.
 * **BrowserComp**: Without context management, we retain details from the most recent 5 turns. With context management, we use the same discard-all strategy as DeepSeek-v3.2 and Kimi K2.5.