README.md

---
language:
- en
license: apache-2.0
tags:
- llm
- pytorch
- causal-lm
- rune-r1
- from-scratch
datasets:
- HuggingFaceFW/fineweb-edu
metrics:
- perplexity
pipeline_tag: text-generation
---

# Rune Base (351M)

A ~351M parameter decoder-only transformer, pretrained from scratch on general web text. This is stage 1 of the Rune-R1 pipeline (**Pretrain → SFT → GRPO**) and serves as the foundation for [Rune-R1-SFT](https://huggingface.co/samueljayasingh/Rune-R1-sft) and [Rune-R1](https://huggingface.co/samueljayasingh/Rune-R1) (the final GRPO reasoning model). On its own, it is a general-purpose next-token predictor with no instruction-following or chat behavior.

## Model Description

| | |
|---|---|
| **Developed by** | samueljayasingh |
| **Model type** | Causal language model (text-only), trained from scratch (not a fine-tune of an existing base model) |
| **Architecture** | Decoder-only Transformer — RoPE, RMSNorm (with QK-norm), SwiGLU feed-forward, Grouped-Query Attention |
| **Parameters** | ~351M |
| **Training framework** | PyTorch (custom training loop, `scripts/train.py`), single AMD MI300X GPU |
| **Dataset** | [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (`sample-10BT` config), streamed |
| **Language** | English |
| **Tokenizer** | GPT-2 (`tiktoken`) |
| **License** | Apache 2.0 |

### Architecture Details

| Parameter | Value |
|---|---|
| Layers | 22 |
| Embedding dimension | 1024 |
| Attention heads / KV groups | 16 / 4 (GQA) |
| Feed-forward hidden dim | 2816 (SwiGLU) |
| Context length | 1024 tokens |
| Position embeddings | RoPE (base 10,000) |
| Normalization | RMSNorm, with QK normalization |
| Vocab size | 50,257 (GPT-2 / tiktoken) |

## Intended Uses & Limitations

### Intended Use
- Research and educational reference for training small language models from scratch.
- Base checkpoint for further fine-tuning (SFT, RLHF/RLVR) — see the SFT and GRPO variants in this same family for a math-reasoning-tuned example.
- Studying pretraining dynamics (loss curves, scaling behavior) at small parameter counts.

### Limitations
- Not instruction-tuned — it will continue text rather than follow chat-style prompts or answer questions directly.
- Trained on only ~5B tokens, far below the data budget of production-grade LLMs; expect weaker factuality, coherence over long spans, and world knowledge than larger, more heavily trained models.
- Context length is limited to 1024 tokens.
- No safety/RLHF alignment has been applied at this stage — treat outputs like any unaligned base LM (do not deploy directly in user-facing applications).

## How to Use

```python
import torch
import tiktoken
from rune.model import CONFIG_350M, RuneModel

ckpt = torch.load("pytorch_model.bin", map_location="cpu")
model = RuneModel(CONFIG_350M)
model.load_state_dict(ckpt)
model.eval()

enc = tiktoken.get_encoding("gpt2")
prompt = "The key to machine learning is"
tokens = torch.tensor([enc.encode(prompt)], dtype=torch.long)

# See rune/generate.py in the source repo for full sampling / KV-cache generation code.
```

The `rune` package (model definition + generation utilities) is available at the [Rune-R1 GitHub repository](https://github.com/samueljayasingh/Rune-R1).

## Hardware

Trained on a single rented GPU instance:

| Component | Spec |
|---|---|
| GPU | 1x AMD MI300X |
| VRAM | 192 GB |
| vCPU | 20 |
| RAM | 240 GB |
| Boot disk | 720 GB NVMe SSD |
| Scratch disk | 5 TB NVMe SSD |
| Rate | $1.99/hr |

## Training & Evaluation

### Training Procedure

| Parameter | Value |
|---|---|
| Dataset | FineWeb-Edu, `sample-10BT` config, streamed |
| Objective | Next-token prediction (causal LM), cross-entropy |
| Tokens seen | ~5.05B |
| Batch size (per step) | 8 sequences |
| Block size | 1024 tokens |
| Gradient accumulation | 64 steps (effective ~524k tokens/optimizer step) |
| Learning rate | 3e-4, cosine schedule |
| Warmup steps | 2,000 |
| Precision | fp32 master weights, bf16 autocast compute |
| Hardware | 1x AMD MI300X GPU |
| Optimizer steps completed | 9,624 |

### Evaluation Results

| Metric | Value |
|---|---|
| Final training loss (cross-entropy) | ~2.999 |
| Min. observed training loss | 2.68 |
| Avg. training loss, last 300 steps | ~3.12 |

Loss is the streaming training-set cross-entropy (no held-out pretraining validation split was used); downstream task performance is evaluated after the SFT and GRPO stages instead — see those model cards for MATH-500 results.

## Citation

```bibtex
@misc{RuneR1Base2026,
  author = {Samuel Jayasingh},
  title = {Rune-R1 Base: A 351M Transformer Pretrained from Scratch on FineWeb-Edu},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/samueljayasingh/Rune-R1-base}}
}
```

## Acknowledgements

- [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) — pretraining corpus.
- Architecture adapted from [rasbt/LLMs-from-scratch](https://github.com/rasbt/LLMs-from-scratch) (Apache 2.0), trimmed to a dense ~350M config with a GPT-2 vocabulary.