TinyStories Llama 64M

A 64.4M-parameter Llama-style language model trained from random weights on TinyStories. It writes short children's stories in simple English. This is a course assignment on scaling laws, not a general assistant.

The full training script is in this repo: train.py. This run used TRAIN_MINUTES=90.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "yzdvai/tinystories-llama-64m"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

inputs = tokenizer("Once upon a time, a little robot found a book.", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.8, top_k=40,
                        pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))

No BOS token is used. Each story ends with <eos>, so generation stops by itself.

Architecture

Total parameters 64,365,440
Layers 13
Hidden size 640
Attention heads 10 (head size 64)
KV heads 5 (grouped-query attention, 2 query heads per KV head)
Intermediate size 1728 (SwiGLU)
Vocabulary 8,192 (byte-level BPE)
Context length 512 tokens
Other RoPE positions, RMSNorm (pre-norm), no biases, tied input and output embeddings

Where the parameters are: SwiGLU 43,130,880 (67%), attention 15,974,400 (25%), embedding 5,242,880 (8%).

Why these choices

  • Llama-style block. RoPE, RMSNorm, SwiGLU and grouped-query attention are what most open models use today. It is a plain LlamaConfig, so it loads with AutoModelForCausalLM and needs no custom code.
  • Size near the 60M floor. The assignment asks for 60-100M. My real compute was 90 minutes on 2 x Tesla T4, and for that compute the scaling rule says a much smaller model is optimal (see below). So I stayed close to the floor instead of going bigger.
  • Wider and shallower (640 wide, 13 layers) than the lab's 512 wide, 24 layers. Fewer, wider layers run faster on a GPU, which means more tokens in the same time.
  • Small vocabulary (8,192). TinyStories uses a few thousand distinct words. A small vocabulary keeps the embedding at 8% of the model, so the parameters go to the layers.
  • Context 512. A story is about 210 tokens, so a whole story fits with room to spare.

Dataset

Source roneneldan/TinyStories, train split (Eldan and Li, 2023)
Language English
Licence CDLA-Sharing-1.0
Stories sampled 900,000 (shuffled, seed 42)
Stories kept 791,092

Why TinyStories. With about 9% of the planned token budget, a broad corpus such as Wikipedia would give word salad. TinyStories was built so that small models can learn to write coherent text, so it is the corpus where a short run still shows real learning.

Preprocessing

  1. Straighten curly quotes and dashes, collapse whitespace.
  2. Drop stories under 100 characters (88), stories with non-ASCII characters, which in this dataset are mostly encoding damage (55,164), and exact duplicates (53,656).
  3. Split by whole story, before tokenizing, so no story is in two splits.
  4. Train a byte-level BPE tokenizer on training stories only (first 200,000).
  5. Tokenize, add <eos> after each story, join, and cut into blocks of 512 tokens with no overlap.
Split Stories Tokens Blocks of 512
Train 775,270 163,954,618 320,223
Validation 7,911 1,677,349 3,276
Test 7,911 1,665,592 3,253

Scaling-law calculations

Planning rule from the lab (a simplified Chinchilla): D = 20 N and C = 6 N D.

Parameters N 64,365,440
Planned tokens D = 20 N 1,287,308,800
Planned compute C = 6 N D 4.97e+17 FLOPs
Tokens actually processed 113,737,728 (8.8% of plan)
Scored next-token targets 113,515,584
Actual compute, 6 N D 4.39e+16 FLOPs
Passes over the training tokens 0.69 (no block was seen twice)
Full plan at the measured speed about 17 hours on the same hardware

Read the other way: for the compute I actually used, the same rule gives an optimal size of N = sqrt(C / 120) = 19M parameters. So this model is larger than its training budget justifies.

What I saw. The loss was still falling at the end, but slowly: validation went from 1.93 at 30M tokens to 1.65 at 62M and 1.51 at 114M. Each doubling of data bought less. The 20:1 ratio was measured on broad web text. TinyStories is much simpler and more repetitive, so the model runs out of new things to learn from it sooner. The rule sets the plan, but how many tokens are worth processing also depends on how hard the data is. On this corpus I expect the remaining 91% of the plan to bring a modest gain, not a large one. I did not test that.

Training setup

Hardware 2 x Tesla T4 (Kaggle), mixed precision (float16)
Optimizer AdamW, betas (0.9, 0.95), weight decay 0.1 on weight matrices only
Learning rate 0.0006 peak, 100 warmup updates, cosine decay to 6e-05
Effective batch 64 sequences x 512 tokens = 32,768 tokens per update (16 per GPU x 2 GPUs x 2 accumulation)
Gradient clipping 1.0
Training steps 3,471 optimizer updates
Training duration 90 minutes (21,062 tokens per second)
Seed 42

Training was limited by time, not by steps: the cosine schedule follows the clock, so the learning rate reaches its floor exactly when the time budget ends.

Evaluation

Held-out next-token cross-entropy (nats) and perplexity, on stories the model and the tokenizer never saw.

Split Loss Perplexity Scored tokens
Validation (full) 1.521 4.57 1,674,036
Test (full) 1.526 4.60 1,662,283

Before training, the loss on the validation subset was 9.14 (a uniform guess is ln 8192 = 9.01). After training it was 1.51. Perplexity only compares with models that use this same tokenizer and data.

Loss figure

Training and validation loss

Training loss is from one batch every 10 updates (the dark line is a moving average). Validation loss is from a fixed subset of 640 blocks every 100 updates. Every training batch is new data, so the two curves sit close together.

Example outputs

Sampling with temperature 0.8, top-k 40, fixed seeds. Not cherry-picked.

Prompt: Once upon a time, a little robot found a book.

Once upon a time, a little robot found a book. The book was very tall, with lots of pages. The robot was very happy. It wanted to keep the book and read it all day. The robot tried to open the book, but it was too hard. The robot was sad. One day, the robot saw a big, red ball. The robot wanted to help the ball. The robot had an idea. It called the red ball, "Hello, red ball!" The red ball opened the book. The robot was very happy. The robot and the red ball became good friends. They played together in the park every day. The robot and the red ball became best friends. They were very happy.

Prompt: Lily and her dog went to the park. Suddenly,

Lily and her dog went to the park. Suddenly, Lily saw a little boy crying. She asked him what was wrong. He told her about his dog and his ice-cream. Lily was compassionate and kind. She said, "Don't worry, we can share the ice-cream with your dog." The little boy smiled and they shared the ice-cream. It was a great day at the park.

Prompt: The old tree in the garden had a secret.

The old tree in the garden had a secret. A secret that no one could find. The people in the garden laughed and said they would never find the secret again. They kept the secret and the old tree stayed there forever.

Limitations

  • Undertrained. It saw 8.8% of the planned token budget.
  • One narrow domain. It only knows simple children's stories. It cannot answer questions, follow instructions, or write about anything else.
  • Synthetic data. TinyStories was written by GPT-3.5 and GPT-4, so the stories repeat the same names, plots and morals, and the model copies that.
  • Weak logic. The grammar is good but the events often make no sense. In the samples above the book is "very tall", a ball opens the book, and the tree's secret is never told.
  • Repetition and thin plots. It repeats itself ("became good friends" and then "became best friends") and some stories end after two or three sentences.
  • English, ASCII only. Other languages and symbols split into bytes and give poor results.
  • Single run. One seed, no hyperparameter search. One run does not prove a scaling law.

Credits

Yazeed Alhifzi. The training code was written with Claude. Based on the course lab "From a compute budget to a 75M-parameter language model". References: Hoffmann et al. 2022 (Chinchilla), Eldan and Li 2023 (TinyStories).

Downloads last month
61
Safetensors
Model size
64.4M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Dataset used to train yzdvai/tinystories-llama-64m