| --- |
| license: mit |
| datasets: |
| - roneneldan/TinyStories |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - gpt |
| - from-scratch |
| - tinystories |
| - educational |
| --- |
| |
| # mpu-30m-base |
|
|
| **mpu-30m** is a 30.3M-parameter GPT-style language model trained from scratch |
| on 400M tokens of [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). |
| It is the first model in the **mpu** series and the proof-of-concept run for an |
| elastic, checkpoint-driven training platform that migrates a single training |
| job across free and disposable GPU sessions (Colab/Kaggle) without losing |
| progress. |
|
|
| It writes short, simple children's stories. It is small on purpose: the goal |
| of this run was to validate the training infrastructure end to end, not to |
| compete on quality. |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Architecture | Decoder-only transformer (GPT-2 style, pre-LN, tied embeddings) | |
| | Parameters | 30.3M total (10.6M non-embedding) | |
| | Layers / heads / width | 6 / 6 / 384 | |
| | Context length | 1024 | |
| | Vocabulary | GPT-2 BPE, 50,304 (padded) | |
| | Precision (training) | fp16 + loss scaling (Tesla T4) | |
| | Format | safetensors | |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Data | TinyStories, ~400M tokens (GPT-2 tokenizer, uint16 memmap shards) | |
| | Steps | 6,104 (65,536 tokens/step: batch 8 × 1024 ctx × grad-accum 8) | |
| | Optimizer | AdamW (β=0.9/0.95, wd 0.1), lr 6e-4, cosine to 10%, 300 warmup steps | |
| | Hardware | 1× NVIDIA T4 (free Colab), ~2 hours, ~49k tokens/sec | |
| | Checkpointing | Atomic checkpoints pushed to this repo every 300 steps | |
|
|
| ### The interesting part: how it was trained |
|
|
| This model was trained by an elastic training system in which **compute is |
| disposable and checkpoints are persistent**. Every ~20 minutes the trainer |
| writes a full checkpoint (weights, optimizer, scaler, RNG state) and promotes |
| it atomically to this repo. A session can be killed at any time; any other |
| GPU session resumes from the latest promoted checkpoint and — because data |
| batches are a pure function of (seed, step) — continues **bit-identically**, |
| verified by test. This run survived a mid-training kill-and-resume across |
| sessions. |
|
|
| ## Usage |
|
|
| The checkpoint uses a custom (nanoGPT-style) architecture and is **not** |
| loadable via `transformers.AutoModel`. Weights are standard safetensors |
| (see `config.json` in the checkpoint folder for the architecture: 6 layers, |
| 6 heads, width 384, GPT-2 BPE tokenizer). The training and inference code |
| is not yet public; it will be released alongside a later model in the |
| series. |
|
|
| ## Sample output |
|
|
| > Once upon a time there was a little robot. He was very happy and liked to |
| > roll with his friends. But one day, he rolled too fast and fell into a big |
| > puddle. He tried to roll out of his wet puddle, but he couldn't. He was |
| > stuck and couldn't get out. Luckily, a kind little girl saw the robot and |
| > knew just what to do. [...] From then on, the robot was extra careful. |
|
|
| ## Limitations |
|
|
| - Trained only on synthetic children's stories: tiny vocabulary in practice, |
| simple grammar, no factual knowledge, English only. |
| - At this scale the model loses track of characters and pronouns, and |
| occasionally substitutes a wrong noun mid-story. |
| - No instruction tuning, no safety tuning, no formal evaluation. Not for any |
| production use — this is an educational artifact. |
