Update README.md
Browse files
README.md
CHANGED
|
@@ -50,7 +50,7 @@ We show a brief overview of each stage below:
|
|
| 50 |
|
| 51 |
<div align="center">
|
| 52 |
|
| 53 |
-
| Stage | Tokens | Step | Seqlen | Batch |
|
| 54 |
| ----- | ------ | ------ | ---- | ----- |
|
| 55 |
| Pretrain | 12T | 877,500 | 8K | 1408 -> 3072 |
|
| 56 |
| Midtrain | 200B | 8,000 | 64K | 384 |
|
|
@@ -61,10 +61,6 @@ We show a brief overview of each stage below:
|
|
| 61 |
|
| 62 |
</div>
|
| 63 |
|
| 64 |
-
## Ripple-65B Base Preview
|
| 65 |
-
|
| 66 |
-
This repo contains the unquantized base model checkpoint directly after pretraining.
|
| 67 |
-
|
| 68 |
## Instruct Model Benchmarks
|
| 69 |
|
| 70 |
We compare Ripple-65B Preview against currently-available US open-weight frontier models.
|
|
@@ -94,7 +90,7 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
|
|
| 94 |
| **Knowledge** | | | | | |
|
| 95 |
| MMLU | *89.60* | *92.0* | *91.3* | **92.6** | *92.4* |
|
| 96 |
| MMLU-Pro | *84.15* | 86.8 | *87.3* | **87.95** | *76.5* |
|
| 97 |
-
| GPQA Diamond | *85.19* | 87.0 | 87.2
|
| 98 |
| **Math** | | | | | |
|
| 99 |
| AIME 2024 | *97.08* | *94.17* | *97.5* | **97.92** | *96.67* |
|
| 100 |
| AIME 2025 | *98.33* | *94.17* | *91.67* | **100.0** | *99.58* |
|
|
@@ -103,12 +99,12 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
|
|
| 103 |
| **Coding & Agentic** | | | | | |
|
| 104 |
| Terminal-Bench 2.1 (reported) | **80.52** | 56.4 | 63.8 | 76.4 | *80.1* |
|
| 105 |
| Terminal-Bench 2.1 (Terminus 2) | **80.52** | *51.31* | *55.43* | *79.78* | *67.79* |
|
| 106 |
-
| SWE-bench Verified | β | 71.9 | 77.6
|
| 107 |
-
| SWE-bench Pro v1.0 | β | *46.92* | 54.3
|
| 108 |
| SWE-bench Pro v2.0 Hard | β | *30.72* | *1.96* | *52.29* | 76.5 |
|
| 109 |
| DeepSWE 1.1 | β | *0.22* | *1.96* | **60**[^j] | 43.3 |
|
| 110 |
| SciCode | *39.40* | 44.6 | *39.18* | **58.5** | 49.2 |
|
| 111 |
-
| ΟΒ³-bench banking | β| 22.6 | 23.7
|
| 112 |
| AutomationBench (Public) | β | *8.39* | *14.56* | *42.44* | 39.3 |
|
| 113 |
| MCP Atlas | β | *62.2* | 76.0[^i] | **89.2** | 80.8 |
|
| 114 |
| **Safety** | | | | | |
|
|
|
|
| 50 |
|
| 51 |
<div align="center">
|
| 52 |
|
| 53 |
+
| Stage | Tokens | Step | Seqlen | Batch Size |
|
| 54 |
| ----- | ------ | ------ | ---- | ----- |
|
| 55 |
| Pretrain | 12T | 877,500 | 8K | 1408 -> 3072 |
|
| 56 |
| Midtrain | 200B | 8,000 | 64K | 384 |
|
|
|
|
| 61 |
|
| 62 |
</div>
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
## Instruct Model Benchmarks
|
| 65 |
|
| 66 |
We compare Ripple-65B Preview against currently-available US open-weight frontier models.
|
|
|
|
| 90 |
| **Knowledge** | | | | | |
|
| 91 |
| MMLU | *89.60* | *92.0* | *91.3* | **92.6** | *92.4* |
|
| 92 |
| MMLU-Pro | *84.15* | 86.8 | *87.3* | **87.95** | *76.5* |
|
| 93 |
+
| GPQA Diamond | *85.19* | 87.0 | 87.2 | 89.0 | **90.6** |
|
| 94 |
| **Math** | | | | | |
|
| 95 |
| AIME 2024 | *97.08* | *94.17* | *97.5* | **97.92** | *96.67* |
|
| 96 |
| AIME 2025 | *98.33* | *94.17* | *91.67* | **100.0** | *99.58* |
|
|
|
|
| 99 |
| **Coding & Agentic** | | | | | |
|
| 100 |
| Terminal-Bench 2.1 (reported) | **80.52** | 56.4 | 63.8 | 76.4 | *80.1* |
|
| 101 |
| Terminal-Bench 2.1 (Terminus 2) | **80.52** | *51.31* | *55.43* | *79.78* | *67.79* |
|
| 102 |
+
| SWE-bench Verified | β | 71.9 | 77.6 | *78.67* | **81.6**[^n] |
|
| 103 |
+
| SWE-bench Pro v1.0 | β | *46.92* | 54.3 | β | **66.1** |
|
| 104 |
| SWE-bench Pro v2.0 Hard | β | *30.72* | *1.96* | *52.29* | 76.5 |
|
| 105 |
| DeepSWE 1.1 | β | *0.22* | *1.96* | **60**[^j] | 43.3 |
|
| 106 |
| SciCode | *39.40* | 44.6 | *39.18* | **58.5** | 49.2 |
|
| 107 |
+
| ΟΒ³-bench banking | β| 22.6 | 23.7 | *14.43* | **36.1** |
|
| 108 |
| AutomationBench (Public) | β | *8.39* | *14.56* | *42.44* | 39.3 |
|
| 109 |
| MCP Atlas | β | *62.2* | 76.0[^i] | **89.2** | 80.8 |
|
| 110 |
| **Safety** | | | | | |
|