dzhang50 commited on
Commit
586e2d5
·
verified ·
1 Parent(s): 9b6d385

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -8
README.md CHANGED
@@ -1,5 +1,16 @@
1
  # Ripple-65B Base Preview
2
 
 
 
 
 
 
 
 
 
 
 
 
3
  We introduce **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens that scores [PLACEHOLDER] on TerminalBench 2.1,
4
  surpassing significantly-larger US open-weight models such as Muse Spark Open 816B (76.4), Inkling-975B (63.8), and Nemotron 3 Ultra 550B (56.4).
5
 
@@ -8,24 +19,27 @@ Our final Ripple model ladder improves Compute Efficiency (CE) on code by **16x*
8
  **55x** for the full hero run.
9
 
10
  Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
11
- For our first model, a key design objective was to make model architecture decisions which reduce pretraining and posttraining
12
  infrastructure complexity while still enabling strong reasoning and agentic performance.
13
- Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE).
14
  This decision quickly paid off, and we were able to train a strong model from scratch with a small team with only 5 people
15
  while maintaining solid MFU throughout pretraining and posttraining.
16
 
17
  Starting from the Olmo llama-like baseline, we switch to a more Gemma-like architecture to improve training stability at scale.
18
  Ripple-65B uses a **thin-and-deep** model aspect ratio, with a narrow d_model (5120) and 80 total layers.
19
- To improve reasoning performance, we **overprovision the number of attention heads** while increasing the SWA-to-global layer ratio and reducing SWA window size to compensate.
 
20
  In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.
21
 
22
- The end result is a model that is especially capable at reasoning given its model size.
23
- At 65B total parameters, the model can fit on consumer hardware, such as the NVIDIA DGX Spark.
 
 
24
  The model is also fast at token generation when running on datacenter GPUs, easily reaching 250+ tok/s with DFlash2 on 4xB200 prior to quantization.
25
 
26
- **Pre-training.** Ripple-65B Preview is pretrained from scratch on a 12T text-only corpus at 8K sequence length.
27
 
28
- **Post-training.** We use a standard post-training recipe consisting of midtraining, SFT, and mixed domain RL.
29
 
30
  ## Instruct Model
31
 
@@ -102,7 +116,7 @@ We use an internal fork of [Olmes](https://github.com/allenai/olmes) for our eva
102
 
103
  ```bibtex
104
  @misc{ricursive2026ripple65b,
105
- title={Ripple-65B},
106
  author={Ricursive Intelligence},
107
  year={2026},
108
  }
 
1
  # Ripple-65B Base Preview
2
 
3
+ <div align="center">
4
+ <img src="https://framerusercontent.com/images/3rFANSmoZuSBhjRwxZolyEU5guo.png" width="30%" alt="Ripple-65B" />
5
+ </div>
6
+
7
+ <p align="center">
8
+ <a href="https://huggingface.co/thinkingmachines/Inkling">BF16</a> |
9
+ <a href="https://huggingface.co/thinkingmachines/Inkling-NVFP4">NVFP4</a> |
10
+ <a href="https://huggingface.co/ricursiveAI/Ripple-65B-Base-Preview">Base Model</a> |
11
+ <a href="https://tinker.thinkingmachines.ai/playground">Launch Blog</a>
12
+ </p>
13
+
14
  We introduce **Ripple-65B Preview**, a 65B parameter dense Transformer model pretrained on 12 trillion tokens that scores [PLACEHOLDER] on TerminalBench 2.1,
15
  surpassing significantly-larger US open-weight models such as Muse Spark Open 816B (76.4), Inkling-975B (63.8), and Nemotron 3 Ultra 550B (56.4).
16
 
 
19
  **55x** for the full hero run.
20
 
21
  Ripple-65B was codesigned to be trained and served efficiently on NVIDIA hardware.
22
+ For our first model, a key objective was to make design decisions which reduce pretraining and posttraining
23
  infrastructure complexity while still enabling strong reasoning and agentic performance.
24
+ Therefore, we chose to train a modernized dense Transformer model rather than use Mixture-of-Experts (MoE) like other labs.
25
  This decision quickly paid off, and we were able to train a strong model from scratch with a small team with only 5 people
26
  while maintaining solid MFU throughout pretraining and posttraining.
27
 
28
  Starting from the Olmo llama-like baseline, we switch to a more Gemma-like architecture to improve training stability at scale.
29
  Ripple-65B uses a **thin-and-deep** model aspect ratio, with a narrow d_model (5120) and 80 total layers.
30
+ To improve reasoning performance, we **overprovision the number of attention heads** while increasing the SWA-to-global layer ratio
31
+ and reducing SWA window size to compensate for the increased cost.
32
  In addition, we added FFN activation clipping, gated attention, global NoPE, and value residuals.
33
 
34
+ The end result is a model that is especially capable at reasoning and agentic tasks given its model size, competing with other
35
+ US open-weight models with more than 10x the number of total parameters.
36
+ At only 65B total parameters, the model can fit on consumer hardware such as the NVIDIA DGX Spark.
37
+ To encourage local model development, we provide an NVFP4 checkpoint and colab to help you get started.
38
  The model is also fast at token generation when running on datacenter GPUs, easily reaching 250+ tok/s with DFlash2 on 4xB200 prior to quantization.
39
 
40
+ **Pre-training.** Ripple-65B Preview is pretrained from scratch on a 12T text-only corpus at 8K sequence length.
41
 
42
+ **Post-training.** We use a standard post-training recipe consisting of midtraining, long context, SFT, and mixed domain RL.
43
 
44
  ## Instruct Model
45
 
 
116
 
117
  ```bibtex
118
  @misc{ricursive2026ripple65b,
119
+ title={Ripple-65B: Pushing the Frontier of Local Intelligence},
120
  author={Ricursive Intelligence},
121
  year={2026},
122
  }