metadata
language:
- en
license: apache-2.0
tags:
- llm
- pytorch
- causal-lm
- rune-r1
- from-scratch
datasets:
- HuggingFaceFW/fineweb-edu
metrics:
- perplexity
pipeline_tag: text-generation
Rune Base (351M)
A ~351M parameter decoder-only transformer, pretrained from scratch on general web text. This is stage 1 of the Rune-R1 pipeline (Pretrain → SFT → GRPO) and serves as the foundation for Rune-R1-SFT and Rune-R1 (the final GRPO reasoning model). On its own, it is a general-purpose next-token predictor with no instruction-following or chat behavior.
Model Description
| Developed by | samueljayasingh |
| Model type | Causal language model (text-only), trained from scratch (not a fine-tune of an existing base model) |
| Architecture | Decoder-only Transformer — RoPE, RMSNorm (with QK-norm), SwiGLU feed-forward, Grouped-Query Attention |
| Parameters | ~351M |
| Training framework | PyTorch (custom training loop, scripts/train.py), single AMD MI300X GPU |
| Dataset | HuggingFaceFW/fineweb-edu (sample-10BT config), streamed |
| Language | English |
| Tokenizer | GPT-2 (tiktoken) |
| License | Apache 2.0 |
Architecture Details
| Parameter | Value |
|---|---|
| Layers | 22 |
| Embedding dimension | 1024 |
| Attention heads / KV groups | 16 / 4 (GQA) |
| Feed-forward hidden dim | 2816 (SwiGLU) |
| Context length | 1024 tokens |
| Position embeddings | RoPE (base 10,000) |
| Normalization | RMSNorm, with QK normalization |
| Vocab size | 50,257 (GPT-2 / tiktoken) |
Intended Uses & Limitations
Intended Use
- Research and educational reference for training small language models from scratch.
- Base checkpoint for further fine-tuning (SFT, RLHF/RLVR) — see the SFT and GRPO variants in this same family for a math-reasoning-tuned example.
- Studying pretraining dynamics (loss curves, scaling behavior) at small parameter counts.
Limitations
- Not instruction-tuned — it will continue text rather than follow chat-style prompts or answer questions directly.
- Trained on only ~5B tokens, far below the data budget of production-grade LLMs; expect weaker factuality, coherence over long spans, and world knowledge than larger, more heavily trained models.
- Context length is limited to 1024 tokens.
- No safety/RLHF alignment has been applied at this stage — treat outputs like any unaligned base LM (do not deploy directly in user-facing applications).
How to Use
import torch
import tiktoken
from rune.model import CONFIG_350M, RuneModel
ckpt = torch.load("pytorch_model.bin", map_location="cpu")
model = RuneModel(CONFIG_350M)
model.load_state_dict(ckpt)
model.eval()
enc = tiktoken.get_encoding("gpt2")
prompt = "The key to machine learning is"
tokens = torch.tensor([enc.encode(prompt)], dtype=torch.long)
# See rune/generate.py in the source repo for full sampling / KV-cache generation code.
The rune package (model definition + generation utilities) is available at the Rune-R1 GitHub repository.
Hardware
Trained on a single rented GPU instance:
| Component | Spec |
|---|---|
| GPU | 1x AMD MI300X |
| VRAM | 192 GB |
| vCPU | 20 |
| RAM | 240 GB |
| Boot disk | 720 GB NVMe SSD |
| Scratch disk | 5 TB NVMe SSD |
| Rate | $1.99/hr |
Training & Evaluation
Training Procedure
| Parameter | Value |
|---|---|
| Dataset | FineWeb-Edu, sample-10BT config, streamed |
| Objective | Next-token prediction (causal LM), cross-entropy |
| Tokens seen | ~5.05B |
| Batch size (per step) | 8 sequences |
| Block size | 1024 tokens |
| Gradient accumulation | 64 steps (effective ~524k tokens/optimizer step) |
| Learning rate | 3e-4, cosine schedule |
| Warmup steps | 2,000 |
| Precision | fp32 master weights, bf16 autocast compute |
| Hardware | 1x AMD MI300X GPU |
| Optimizer steps completed | 9,624 |
Evaluation Results
| Metric | Value |
|---|---|
| Final training loss (cross-entropy) | ~2.999 |
| Min. observed training loss | 2.68 |
| Avg. training loss, last 300 steps | ~3.12 |
Loss is the streaming training-set cross-entropy (no held-out pretraining validation split was used); downstream task performance is evaluated after the SFT and GRPO stages instead — see those model cards for MATH-500 results.
Citation
@misc{RuneR1Base2026,
author = {Samuel Jayasingh},
title = {Rune-R1 Base: A 351M Transformer Pretrained from Scratch on FineWeb-Edu},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/samueljayasingh/Rune-R1-base}}
}
Acknowledgements
- HuggingFaceFW/fineweb-edu — pretraining corpus.
- Architecture adapted from rasbt/LLMs-from-scratch (Apache 2.0), trimmed to a dense ~350M config with a GPT-2 vocabulary.