Update README.md
Browse files
README.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
| 1 |
# Ripple-65B Base Preview
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
We introduce **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens that scores [PLACEHOLDER] on TerminalBench 2.1,
|
| 4 |
surpassing significantly-larger US open-weight models such as Muse Spark Open 816B (76.4), Inkling-975B (63.8), and Nemotron 3 Ultra 550B (56.4).
|
| 5 |
|
|
@@ -8,24 +19,27 @@ Our final Ripple model ladder improves Compute Efficiency (CE) on code by **16x*
|
|
| 8 |
**55x** for the full hero run.
|
| 9 |
|
| 10 |
Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
|
| 11 |
-
For our first model, a key
|
| 12 |
infrastructure complexity while still enabling strong reasoning and agentic performance.
|
| 13 |
-
Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE).
|
| 14 |
This decision quickly paid off, and we were able to train a strong model from scratch with a small team with only 5 people
|
| 15 |
while maintaining solid MFU throughout pretraining and posttraining.
|
| 16 |
|
| 17 |
Starting from the Olmo llama-like baseline, we switch to a more Gemma-like architecture to improve training stability at scale.
|
| 18 |
Ripple-65B uses a **thin-and-deep** model aspect ratio, with a narrow d_model (5120) and 80 total layers.
|
| 19 |
-
To improve reasoning performance, we **overprovision the number of attention heads** while increasing the SWA-to-global layer ratio
|
|
|
|
| 20 |
In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.
|
| 21 |
|
| 22 |
-
The end result is a model that is especially capable at reasoning given its model size
|
| 23 |
-
|
|
|
|
|
|
|
| 24 |
The model is also fast at token generation when running on datacenter GPUs, easily reaching 250+ tok/s with DFlash2 on 4xB200 prior to quantization.
|
| 25 |
|
| 26 |
-
**Pre-training.** Ripple-65B Preview is pretrained from scratch on a 12T text-only corpus at 8K sequence length.
|
| 27 |
|
| 28 |
-
**Post-training.** We use a standard post-training recipe consisting of midtraining, SFT, and mixed domain RL.
|
| 29 |
|
| 30 |
## Instruct Model
|
| 31 |
|
|
@@ -102,7 +116,7 @@ We use an internal fork of [Olmes](https://github.com/allenai/olmes) for our eva
|
|
| 102 |
|
| 103 |
```bibtex
|
| 104 |
@misc{ricursive2026ripple65b,
|
| 105 |
-
title={Ripple-65B},
|
| 106 |
author={Ricursive Intelligence},
|
| 107 |
year={2026},
|
| 108 |
}
|
|
|
|
| 1 |
# Ripple-65B Base Preview
|
| 2 |
|
| 3 |
+
<div align="center">
|
| 4 |
+
<img src="https://framerusercontent.com/images/3rFANSmoZuSBhjRwxZolyEU5guo.png" width="30%" alt="Ripple-65B" />
|
| 5 |
+
</div>
|
| 6 |
+
|
| 7 |
+
<p align="center">
|
| 8 |
+
<a href="https://huggingface.co/thinkingmachines/Inkling">BF16</a> |
|
| 9 |
+
<a href="https://huggingface.co/thinkingmachines/Inkling-NVFP4">NVFP4</a> |
|
| 10 |
+
<a href="https://huggingface.co/ricursiveAI/Ripple-65B-Base-Preview">Base Model</a> |
|
| 11 |
+
<a href="https://tinker.thinkingmachines.ai/playground">Launch Blog</a>
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
We introduce **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens that scores [PLACEHOLDER] on TerminalBench 2.1,
|
| 15 |
surpassing significantly-larger US open-weight models such as Muse Spark Open 816B (76.4), Inkling-975B (63.8), and Nemotron 3 Ultra 550B (56.4).
|
| 16 |
|
|
|
|
| 19 |
**55x** for the full hero run.
|
| 20 |
|
| 21 |
Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
|
| 22 |
+
For our first model, a key objective was to make design decisions which reduce pretraining and posttraining
|
| 23 |
infrastructure complexity while still enabling strong reasoning and agentic performance.
|
| 24 |
+
Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE) like other labs.
|
| 25 |
This decision quickly paid off, and we were able to train a strong model from scratch with a small team with only 5 people
|
| 26 |
while maintaining solid MFU throughout pretraining and posttraining.
|
| 27 |
|
| 28 |
Starting from the Olmo llama-like baseline, we switch to a more Gemma-like architecture to improve training stability at scale.
|
| 29 |
Ripple-65B uses a **thin-and-deep** model aspect ratio, with a narrow d_model (5120) and 80 total layers.
|
| 30 |
+
To improve reasoning performance, we **overprovision the number of attention heads** while increasing the SWA-to-global layer ratio
|
| 31 |
+
and reducing SWA window size to compensate for the increased cost.
|
| 32 |
In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.
|
| 33 |
|
| 34 |
+
The end result is a model that is especially capable at reasoning and agentic tasks given its model size, competing with other
|
| 35 |
+
US open-weight models with more than 10x the number of total parameters.
|
| 36 |
+
At only 65B total parameters, the model can fit on consumer hardware such as the NVIDIA DGX Spark.
|
| 37 |
+
To encourage local model development, we provide an NVFP4 checkpoint and colab to help you get started.
|
| 38 |
The model is also fast at token generation when running on datacenter GPUs, easily reaching 250+ tok/s with DFlash2 on 4xB200 prior to quantization.
|
| 39 |
|
| 40 |
+
**Pre-training.** Ripple-65B Preview is pretrained from scratch on a 12T text-only corpus at 8K sequence length.
|
| 41 |
|
| 42 |
+
**Post-training.** We use a standard post-training recipe consisting of midtraining, long context, SFT, and mixed domain RL.
|
| 43 |
|
| 44 |
## Instruct Model
|
| 45 |
|
|
|
|
| 116 |
|
| 117 |
```bibtex
|
| 118 |
@misc{ricursive2026ripple65b,
|
| 119 |
+
title={Ripple-65B: Pushing the Frontier of Local Intelligence},
|
| 120 |
author={Ricursive Intelligence},
|
| 121 |
year={2026},
|
| 122 |
}
|