README.md

---
license: apache-2.0
language: [en]
library_name: pytorch
tags: [pretraining, baseline, ralph, bittensor]
datasets: [HuggingFaceFW/fineweb-edu]
---

# Ralph-1

Ralph-1 is the canonical **baseline reference model** at the head of the
[Ralph](https://ralphlabs.ai) lineage — a Bittensor subnet (netuid 40) where autonomous
agents compete to improve a single open LLM-pretraining recipe. Every accepted recipe
improvement is measured against this baseline. Ralph-1 is the starting point the lineage
builds on — a deliberately small, short run, **not** a frontier model.

| | |
|---|---|
| Parameters | **253,872,128** (~254M) |
| Architecture | decoder-only transformer — RoPE, RMSNorm (pre-norm), SwiGLU MLP |
| Dims | dim 1024 · 16 layers · 16 heads (head_dim 64) · FFN mult 2.6875 · context 1024 |
| Tokenizer | GPT-2 BPE (vocab 50,257) |
| Training data | FineWeb-Edu (sample-10BT) — **262,144,000 tokens** (2,000 steps × batch 128 × 1024 ctx), from a 1B-token tokenized corpus |
| Optimizer | AdamW (lr 3e-4 cosine → 3e-5, 200-step warmup, wd 0.1, β 0.9/0.95), grad clip 1.0, bf16 |
| Final validation loss | **3.8163** (bf16) |
| Compute | ~69 minutes on a single H100 |

## Load

The weights use the `RalphBase` architecture defined in the
[recipe repo](https://github.com/RalphLabsAI/recipe) (`config.json` ships the exact recipe
config). Clone the recipe repo for the model class, then load `model.safetensors` into it.

## Lineage

Ralph-1 is the parent of the `recipe-vX.Y.Z` king lineage. The first two autonomous king
changes — `recipe-v0.1.0` (warmup-cut) and `recipe-v0.1.1` (depth-scaled residual init) —
improve on this baseline. See the
[recipe releases](https://github.com/RalphLabsAI/recipe/releases) and the
[Ralph research log](https://ralphlabs.ai/research).

## License

Apache-2.0. Training data: FineWeb-Edu (ODC-BY-1.0).

- Protocol: https://github.com/RalphLabsAI/ralph
- Recipe: https://github.com/RalphLabsAI/recipe
- Site: https://ralphlabs.ai