---
license: mit
datasets:
- roneneldan/TinyStories
language:
- en
pipeline_tag: text-generation
tags:
- gpt
- from-scratch
- tinystories
- educational
---

# mpu-30m-base

**mpu-30m** is a 30.3M-parameter GPT-style language model trained from scratch
on 400M tokens of [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories).
It is the first model in the **mpu** series and the proof-of-concept run for an
elastic, checkpoint-driven training platform that migrates a single training
job across free and disposable GPU sessions (Colab/Kaggle) without losing
progress.

It writes short, simple children's stories. It is small on purpose: the goal
of this run was to validate the training infrastructure end to end, not to
compete on quality.

## Model details

| | |
|---|---|
| Architecture | Decoder-only transformer (GPT-2 style, pre-LN, tied embeddings) |
| Parameters | 30.3M total (10.6M non-embedding) |
| Layers / heads / width | 6 / 6 / 384 |
| Context length | 1024 |
| Vocabulary | GPT-2 BPE, 50,304 (padded) |
| Precision (training) | fp16 + loss scaling (Tesla T4) |
| Format | safetensors |

## Training

| | |
|---|---|
| Data | TinyStories, ~400M tokens (GPT-2 tokenizer, uint16 memmap shards) |
| Steps | 6,104 (65,536 tokens/step: batch 8 × 1024 ctx × grad-accum 8) |
| Optimizer | AdamW (β=0.9/0.95, wd 0.1), lr 6e-4, cosine to 10%, 300 warmup steps |
| Hardware | 1× NVIDIA T4 (free Colab), ~2 hours, ~49k tokens/sec |
| Checkpointing | Atomic checkpoints pushed to this repo every 300 steps |

### The interesting part: how it was trained

This model was trained by an elastic training system in which **compute is
disposable and checkpoints are persistent**. Every ~20 minutes the trainer
writes a full checkpoint (weights, optimizer, scaler, RNG state) and promotes
it atomically to this repo. A session can be killed at any time; any other
GPU session resumes from the latest promoted checkpoint and — because data
batches are a pure function of (seed, step) — continues **bit-identically**,
verified by test. This run survived a mid-training kill-and-resume across
sessions.

## Usage

The checkpoint uses a custom (nanoGPT-style) architecture and is **not**
loadable via `transformers.AutoModel`. Weights are standard safetensors
(see `config.json` in the checkpoint folder for the architecture: 6 layers,
6 heads, width 384, GPT-2 BPE tokenizer). The training and inference code
is not yet public; it will be released alongside a later model in the
series.

## Sample output

> Once upon a time there was a little robot. He was very happy and liked to
> roll with his friends. But one day, he rolled too fast and fell into a big
> puddle. He tried to roll out of his wet puddle, but he couldn't. He was
> stuck and couldn't get out. Luckily, a kind little girl saw the robot and
> knew just what to do. [...] From then on, the robot was extra careful.

## Limitations

- Trained only on synthetic children's stories: tiny vocabulary in practice,
  simple grammar, no factual knowledge, English only.
- At this scale the model loses track of characters and pronouns, and
  occasionally substitutes a wrong noun mid-story.
- No instruction tuning, no safety tuning, no formal evaluation. Not for any
  production use — this is an educational artifact.