sraivante/tinystories-35m

TinyStories 35M (base + chat)

A small GPT (decoder-only transformer) trained from scratch on TinyStories on a single Apple M5 laptop, plus a chat fine-tune ("Tiny", a story bot for kids).

Variant	Folder	What it does
base	base/	Continues a story from any prompt.
chat	chat/	Chats in User: / Assistant: turns: small talk, tells stories on request, answers questions about a story you paste, politely declines out-of-scope questions.
unique-v4	unique-v4/	Same architecture and data as base, but trained with a "phase 1" pre-pass (each unique token as its own length-1 context, 8 epochs) before normal training. Its apparent val-loss win was later shown to be eval noise — see the 2026-10-20 revision below.
unique-v7	unique-v7/	Like unique-v4, but weights initialized from pi's decimal digits (each digit × 0.003 × random sign, weight rms ≈ 0.016) instead of gaussian noise — trains exactly as well as gaussian init: only the init's scale matters, not its value source.
unique-v9	unique-v9/	Like unique-v7 but pi digits × 0.001 (weight rms ≈ 0.0053, ~4× smaller than standard). Val loss 1.3517 — under-scaled init costs ~+0.025.
unique-v10	unique-v10/	Pyramid architecture: each block's attention/MLP dims shrink ~21% per layer (ratio 0.79), snapped to hardware-friendly sizes (head dim ×8, MLP hidden ×64), total params matched to the uniform model exactly. Trains at identical speed to uniform; val loss 1.3456. Needs code/model_pyramid_v10.py to load (not the standard GPT class).
unique-v11	unique-v11/	Weights initialized from the training set's own unique token ids (each weight = a token id's mantissa × 0.001 × random sign, rms ≈ 0.0004, ~50× too small). Val loss 1.4772 — the strongest demonstration that under-scaled init hurts.

Model

Architecture	GPT-2 style: 9 layers, 6 heads, 384 dim, pre-LayerNorm, GELU MLP, learned positions, tied input/output embeddings, no biases
Parameters	35.65M (19.3M of them are the tied token embedding)
Context	1024 tokens
Tokenizer	GPT-2 BPE via tiktoken (gpt2), vocab padded to 50,304; `<|endoftext
Weights	safetensors, fp32. lm_head.weight is not stored (tied to wte.weight)

Training

Base (code/train.py, log: logs/train.log)

• Data: 2,119,719 TinyStories training stories (~472M tokens), each followed by <|endoftext|>
• 10,682 steps × 65,536 tokens/step = 700M tokens (~1.5 epochs)
• AdamW (β 0.9/0.95, wd 0.1), lr 1e-3 → 1e-4 cosine, 300 warmup steps, grad clip 1.0, bf16 autocast
• Apple M5 (MPS), ~10.5k tokens/s, ~19 hours
• Final loss: train 1.302, validation 1.340

Chat (code/finetune_chat.py, data builder code/chat_data.py, log: logs/finetune_chat.log)

• Starts from the base model; 3,000 steps, lr 2e-4 → 2e-5
• Synthetic conversations: small talk, story requests (answered with real TinyStories stories), passage QA, simple kid "why" questions, out-of-scope refusals
• Loss only on Assistant turns; every few steps trains on plain TinyStories text to keep storytelling
• Final: chat validation loss 1.048, story loss 1.664 (base: 1.340). The chat model is noticeably weaker as a plain story continuer, so use base for raw story generation.

📄 Technical report

The full write-up of this experiment series (including the revision below) is published on Zenodo: Initialization Scale, Not Value Source — DOI 10.5281/zenodo.23489983 (report.pdf also in this repo).

⚠️ 2026-10-20 revision — multi-seed replication overturned the phase-1 "gain"

Earlier versions of this card reported the single-token pre-pass (v4/v7) as a real −0.013 val-loss improvement, "confirmed" by a 2-seed control, and v7 as significantly preferred in generation. Proper replication overturned this:

• Each run's reported "final val" came from 50 randomly sampled eval batches — a protocol that turns out to carry ±≈0.01 measurement noise (e.g. the seed-777 baseline scored 1.3478 on its own eval but is the best of six models, 1.3260, on a fixed eval set).
• Re-scoring all checkpoints on the identical 400 validation batches (3.28M tokens) and training 3 baseline seeds:

Seed	base	v4 (pre-pass)	v7 (pi-init + pre-pass)
1337	1.3301	1.3338	1.3324
2024	1.3324	—	—
777	1.3260	1.3294	—

Baseline seed-SD ≈ 0.003; the pre-pass is consistently ~+0.0035 worse than its same-seed baseline. The pre-pass gives no gain. (Missing cells: runs lost to a cloud-account ban before download.)

• A 6-way 300-prompt cross-judge among the seed models came out 48/42/53/52/52/53 — all within chance; v7's earlier generation "wins" did not recur against seed-matched baselines.
• What stands: the init-scale U-curve (effects 10–400× the noise), the harm of vocabulary streams, the pyramid results, and the 125M finding below. Details and the full post-mortem are in the Zenodo report.

New: 125M-scale init comparison (12 layers / 12 heads / 768 dim, same budget)

Two runs differing only in init, scored on the same fixed 400 batches: default normal(0, 0.02) → 1.2325; pi ×0.003 (rms ≈ 0.016) → 1.2029. The smaller init wins by 0.030 (~10× the 35M seed-SD) — consistent with the optimal init scale shrinking as width grows. Preliminary (single runs, 125M baseline re-run pending).

unique-v7 — pi-digit initialization + single-token pre-pass

unique-v7 remains the series' most interesting artifact: a model in which none of the weights start from random numbers — every initial weight comes from the decimal digits of pi — that trains exactly as well as its gaussian-initialized siblings.

What exactly v7 is

Component	Exactly what was used
Architecture	Identical to base: GPT-2 style, 9 layers / 6 heads / 384 dim, 35.65M params, context 1024, tied embeddings, no biases
Weight init	No gaussian/random init at all. All 35,635,200 weights (every 2-D matrix, incl. the tied token embedding) filled sequentially from the first 35.6M decimal digits of pi: w_i = digit_i × 0.003 × random sign. The ×0.003 scale puts the weight rms at ≈0.016, deliberately close to the standard init's 0.02
Phase 1 (pre-pass)	Before seeing any real text, the model trains on each of the 25,944 unique GPT-2 token ids of TinyStories as its own complete context: a length-1 sequence where input = the token and target = the same token. All ids in first-appearance order, 8 epochs ≈ 56 optimizer steps (seconds of compute), lr warmup to 1e-3
Phase 2 (main training)	Byte-identical to base: 10,682 iters × 65,536 tokens/step = 700M TinyStories tokens, AdamW, lr 1e-3→1e-4 cosine, fp16 on Kaggle 2×T4 (DDP)
Total budget	Exactly the same as every other run — no extra data, no extra steps

Why it is meaningful

1. Initialization needs scale, not randomness. A model initialized from pi — a fixed, public, maximally "non-random" sequence — trains exactly as well as one initialized from gaussian noise. Together with its siblings (v6: same digits ×0.07 → destroyed, loss 2.46; v9: ×0.001 → 1.3517; v11: token-id values, rms ≈0.0004 → 1.4772) it maps a clean U-curve with the optimum at weight rms ≈ 0.02 (at width 384): the source of the initial values never mattered, only their magnitude.
2. Fully reproducible init. Since pi's digits are public and the sign RNG is seeded (1337), v7's starting weights are bit-reproducible by anyone — a property gaussian inits don't naturally give you.
3. Its earlier val-loss and generation "wins" are retracted per the revision above: on fixed-batch eval v7 is indistinguishable from (same-seed) baseline, and its +3 SD generation win did not replicate against seed-matched baselines.

Load it from unique-v7/ (same class/config as base).

Init / phase-1 experiments

Thirteen controlled runs on Kaggle T4x2 asking: does a different weight initialization, a phase-1 pre-pass over the corpus's unique tokens, or a pyramid architecture beat plain training? One variable changes per run; the budget is identical everywhere (10,682 iters × 65,536 tokens/step = 700M tokens). unique-v4 and unique-v7 tie as winners (see the headline section above); all variants named unique-* are downloadable from this repo.

Full scoreboard (best → worst)

Note (2026-10-20): the "final val loss" column uses each run's own 50-random-batch eval, which carries ±≈0.01 measurement noise — differences under ~0.02 in this table are not meaningful (see the revision section above for the fixed-batch numbers). The large effects (streams, v6/v8/v9/v11) stand.

#	Run	Init	Phase 1	Final val loss	vs base
1	v7	pi digits ×0.003 ± (rms ≈0.016)	1 sweep	1.3264	−0.013
2	v4	normal 0.02 (standard)	1 sweep	1.3267	−0.013
3	v12	normal 0.02	21 sweeps, interleaved every 508 iters	1.3330	−0.007
4	base	normal 0.02	—	1.3396	—
—	base (seed 2024)	normal 0.02, different seed	—	1.3382	seed-noise control: ±0.0014
5	v10	pi ×0.003 ± + pyramid 0.79, hardware-friendly dims	1 sweep	1.3456	+0.006
6	v5	uniform ±0.07	—	1.3465	+0.007
7	v3	uniform ±0.07	1 sweep	1.3474	+0.008
8	v9	pi digits ×0.001 ± (rms ≈0.0053)	1 sweep	1.3517	+0.012
9	v8	pi ×0.003 ± + pyramid 2/3, odd dims	1 sweep	1.3598	+0.020
10	stream	normal 0.02	unique-word stream, 100 iters	1.3647	+0.025
11	stream+EOT	normal 0.02	stream with <|endoftext|>	1.4237	+0.084
12	v11	token-id mantissa ×0.001 ± (rms ≈0.0004)	1 sweep	1.4772	+0.138
13	v6	pi digits ×0.07 ± (rms ≈0.37)	1 sweep	2.4568	broken

("1 sweep" = the single-token pre-pass: each of the 25,944 unique token ids as its own length-1 context, 8 epochs ≈ 56 steps, before training.)

Finding 1 — initialization needs scale, not randomness

The same training recipe with the init's magnitude swept across four orders of structure sources gives a clean U-curve with its optimum at weight rms ≈ 0.02. The source of the values (pi digits, the tokens' own ids, gaussian noise) never mattered — only their size:

Weight rms	Source	Run	Val loss
0.37 (18× too large)	pi digits ×0.07	v6	2.4568 — destroyed
0.04 (2× too large)	uniform ±0.07	v3 / v5	1.3474 / 1.3465
0.02 / 0.016 (optimum)	gaussian / pi digits ×0.003	v4 / v7	1.3267 / 1.3264
0.0053 (4× too small)	pi digits ×0.001	v9	1.3517
0.0004 (50× too small)	token-id values ×0.001	v11	1.4772

Finding 2 — the single-token pre-pass gives no gain (REVISED 2026-10-20)

The original single-seed ordering (v4 1.3267 < v12 1.3330 < base 1.3396) and its "seed-noise control" were artifacts of the noisy per-run eval protocol. On the fixed 400-batch eval across 3 seeds (table in the revision section above), the pre-pass is consistently ~+0.0035 worse than its same-seed baseline, and the apparent v4/v12/base ordering dissolves into the noise band. What survives: the pre-pass only doesn't hurt in its length-1-context form, while unique-word streams actively hurt (+0.025 to +0.084 — effects far above the noise).

Finding 3 — pyramid blocks are (at best) free, never better

Shrinking each block's attention/MLP dims with depth, at a total parameter count matched to the uniform model exactly:

Variant	Per-layer shrink	Dims	Training speed (T4)	Val loss
v8	×2/3	odd (head dims 197, 131, 88, …)	2.05 s/it — 2.2× slower (odd shapes miss fused kernels)	1.3598
v10	×0.79	snapped: head dim ×8, MLP hidden ×64	0.91 s/it — identical to uniform	1.3456

Hardware-friendly dims completely fix the speed penalty, and the gentler ratio shrinks the quality gap to +0.006 — nearly free, but the "deep layers need less capacity" idea never actually beat the uniform model.

Generation: 300-prompt cross-judged comparisons

Protocol: every model continues the same 300 validation-story openings (fixed seed per prompt); each story is scored by every other model's mean log-prob per token; highest average wins the prompt. Four fields were run:

Field (chance wins each)	Results (wins / 300)
base, v4, v7, v8 (75)	v7 85, v8 83, base 70, v4 62 — tie
base, v4, v7, v9, v10, v11 (50)	v7 70 (+3 SD), v10 58, v9 54, v4 49, base 47, v11 22 (−4 SD)
base, v4, v7, v12 (75)	v7 87, v12 74, v4 71, base 68 — tie
base×3 seeds, v4×2 seeds, v7 (50)	48 / 42 / 53 / 52 / 52 / 53 — all within chance

v7 nominally led the first three fields (once at +3 SD), but in the decisive fourth field — against seed-matched baselines instead of weakened variants — everything is a dead tie, so we treat the earlier +3 SD as a field/protocol artifact, not a model property. Robust separation appears only at the bottom: v11 (−4 SD) and v6 (0/300 in an earlier 3-way). Val-loss differences of ±0.02 are invisible to the judges. Caveat: wins are relative to the field and judge panel, not an absolute quality measure.

HellaSwag: a negative control that stayed negative

All 13 runs — and roneneldan's reference models — sit in a 26.2–26.8% band, within noise of the 25% chance floor and of each other (SE ≈ ±0.44pp; per-2000-question blocks swing 0.23–0.32 for every model in lockstep):

v12	base s2024	v8	v9	v4 s777	v10	base s777	v4	v11	v7	base	random weights
.2680	.2669	.2662	.2660	.2656	.2652	.2653	.2640	.2634	.2624	.2620	.2546

No init, pre-pass, or architecture changed it — expected, since HellaSwag tests world knowledge that TinyStories simply doesn't contain. Included as a sanity check, not a scoreboard.

Results

All outputs are in reports/, produced by run_reports.sh. Generations are sampled, so a re-run gives different text.

Test	Model	Result
Passage QA, 20 hand-written questions (chat_eval.txt)	chat	15/20 (base model with prompt tricks: 12/20)
Passage QA, 150 held-out generated questions (chat_eval.txt)	chat	135/150 = 90%
HellaSwag val, 10,042 items, acc / acc_norm	random weights	0.255 / 0.250
	base	0.262 / 0.265
	chat	0.263 / 0.259
Grammar correction, 8 sentences (grammar_check_base.txt)	base	7 fixed or unchanged correctly, 1 changed the meaning (She → They)

HellaSwag is close to chance (25%). That is expected: it tests general world knowledge, while this model has only read children's stories. It is included as a sanity check, not a strength.

Comparison with the original TinyStories models

Benchmarked against roneneldan's original TinyStories models (the paper's reference checkpoints), evaluated locally with identical scoring code.

HellaSwag (full validation set, 10,042 items; code/hellaswag_hf.py for the HF models):

Model	acc	acc_norm
this model (base, 35M)	0.262	0.265
roneneldan/TinyStories-1M	0.257	0.252
roneneldan/TinyStories-8M	0.268	0.263
roneneldan/TinyStories-2Layers-33M	0.270	0.271
chance	0.250	0.250

The original models score best here: TinyStories-2Layers-33M is the strongest on both metrics, and TinyStories-8M also edges this model on raw accuracy. That said, all TinyStories-only models sit within ~2 points of chance regardless of size — the corpus simply contains no adult commonsense knowledge, so none of these differences mean much.

300-prompt cross-judged story generation (code/compare_300_hf4way.py): each model continues the same 300 validation-story openings (fixed seed per prompt); every story is scored by the other three models' mean log-prob per token; highest average judge score wins the prompt.

Model	Wins / 300
this model (base, 35M)	124
roneneldan/TinyStories-8M	114
roneneldan/TinyStories-2Layers-33M	61
roneneldan/TinyStories-1M	1

This model is statistically tied with the paper's 8M reference model and clearly ahead of the 2-layer 33M one — at this scale, depth matters more than width.

Example: chat

You : hi
Tiny: Hey! I am Tiny. I can tell stories and answer questions about them.
You : Why do we brush our teeth?
Tiny: We brush our teeth to keep them clean and strong, so they do not hurt.
You : What is the capital of France?
Tiny: I'm sorry, I don't know about that. I know about kids, animals and stories. Shall I tell you a story?
You : I am sad today
Tiny: Oh no, I am sorry you feel sad. Do you want a happy story to cheer you up?

Example: base story (prompt "Once upon a time")

Once upon a time, there was a little girl named Lily. She loved to play in her room with her toys.
One day, she saw her mom cleaning and said, "Mommy, can I help you?" Her mom said, "Sure, Lily.
You can help me wipe the floor."

Example: grammar check (base model scores candidate edits)

IN : the girl want to go hoospital
OUT: the girl wanted to go to the hospital
IN : Tom wanted play with his freind.
OUT: Tom wanted to play with his friends.

All reports

File	Contents
reports/stories_base.txt	5 stories from "Once upon a time"
reports/stories_base_prompts.txt	4 stories from custom opening lines
reports/chat_eval.txt	4 chat conversations, 20 hand-written QA, 150 held-out QA
reports/ask_base.txt	Base model answering questions framed as story dialogue
reports/passage_qa_base.txt	Base model answering questions about a passage
reports/grammar_check_base.txt	8 grammar/spelling corrections
reports/hellaswag_{random,base,chat}.txt	HellaSwag benchmark runs

Usage

pip install torch tiktoken safetensors huggingface_hub
# get the code folder, then from inside it:
python from_hub.py --repo sraivante/tinystories-35m           # writes out/final.pt (base) and out/chat.pt (chat)
python sample.py --ckpt out/final.pt --prompt "Once upon a time"
python chat.py                              # interactive chat with Tiny
python grammar_check.py "He have a big red ball."
./start.sh                                  # web UI at http://127.0.0.1:8765

Or load directly in Python (with code/model.py on the path):

import torch, tiktoken
from from_hub import load
model, cfg = load("sraivante/tinystories-35m", "base")        # or "chat"
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode_ordinary("Once upon a time")])
out = model.generate(ids, 200, temperature=0.8, top_k=200, stop_token=enc.eot_token)
print(enc.decode(out[0].tolist()))

Chat prompt format: User: <message>\nAssistant:; stop when the model writes \nUser:.

Limitations

• Knows only the simple world of children's stories; no facts, maths or general knowledge.
• Stories can lose track of characters or contradict themselves, especially past ~300 tokens.
• Passage QA misses questions that need reasoning across sentences (e.g. "What broke the sandcastle?").
• Not safety-tuned beyond the chat model's out-of-scope refusals; not intended for production use.

Files

config.json  configs of both variants (read by from_hub.py)
base/  model.safetensors, config.json
chat/  model.safetensors, config.json
code/  model, training, fine-tuning, evaluation, inference, web UI, from_hub.py
logs/  train.log, finetune_chat.log
reports/  evaluation outputs (see above)
run_reports.sh  regenerates reports/