Instructions to use yzdvai/tinystories-llama-64m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use yzdvai/tinystories-llama-64m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="yzdvai/tinystories-llama-64m")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("yzdvai/tinystories-llama-64m") model = AutoModelForCausalLM.from_pretrained("yzdvai/tinystories-llama-64m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use yzdvai/tinystories-llama-64m with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "yzdvai/tinystories-llama-64m" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "yzdvai/tinystories-llama-64m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/yzdvai/tinystories-llama-64m
- SGLang
How to use yzdvai/tinystories-llama-64m with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "yzdvai/tinystories-llama-64m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "yzdvai/tinystories-llama-64m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "yzdvai/tinystories-llama-64m" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "yzdvai/tinystories-llama-64m", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use yzdvai/tinystories-llama-64m with Docker Model Runner:
docker model run hf.co/yzdvai/tinystories-llama-64m
TinyStories Llama 64M
A 64.4M-parameter Llama-style language model trained from random weights on TinyStories. It writes short children's stories in simple English. This is a course assignment on scaling laws, not a general assistant.
The full training script is in this repo: train.py. This run used TRAIN_MINUTES=90.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "yzdvai/tinystories-llama-64m"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
inputs = tokenizer("Once upon a time, a little robot found a book.", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.8, top_k=40,
pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))
No BOS token is used. Each story ends with <eos>, so generation stops by itself.
Architecture
| Total parameters | 64,365,440 |
| Layers | 13 |
| Hidden size | 640 |
| Attention heads | 10 (head size 64) |
| KV heads | 5 (grouped-query attention, 2 query heads per KV head) |
| Intermediate size | 1728 (SwiGLU) |
| Vocabulary | 8,192 (byte-level BPE) |
| Context length | 512 tokens |
| Other | RoPE positions, RMSNorm (pre-norm), no biases, tied input and output embeddings |
Where the parameters are: SwiGLU 43,130,880 (67%), attention 15,974,400 (25%), embedding 5,242,880 (8%).
Why these choices
- Llama-style block. RoPE, RMSNorm, SwiGLU and grouped-query attention are what most open
models use today. It is a plain
LlamaConfig, so it loads withAutoModelForCausalLMand needs no custom code. - Size near the 60M floor. The assignment asks for 60-100M. My real compute was 90 minutes on 2 x Tesla T4, and for that compute the scaling rule says a much smaller model is optimal (see below). So I stayed close to the floor instead of going bigger.
- Wider and shallower (640 wide, 13 layers) than the lab's 512 wide, 24 layers. Fewer, wider layers run faster on a GPU, which means more tokens in the same time.
- Small vocabulary (8,192). TinyStories uses a few thousand distinct words. A small vocabulary keeps the embedding at 8% of the model, so the parameters go to the layers.
- Context 512. A story is about 210 tokens, so a whole story fits with room to spare.
Dataset
| Source | roneneldan/TinyStories, train split (Eldan and Li, 2023) |
| Language | English |
| Licence | CDLA-Sharing-1.0 |
| Stories sampled | 900,000 (shuffled, seed 42) |
| Stories kept | 791,092 |
Why TinyStories. With about 9% of the planned token budget, a broad corpus such as Wikipedia would give word salad. TinyStories was built so that small models can learn to write coherent text, so it is the corpus where a short run still shows real learning.
Preprocessing
- Straighten curly quotes and dashes, collapse whitespace.
- Drop stories under 100 characters (88), stories with non-ASCII characters, which in this dataset are mostly encoding damage (55,164), and exact duplicates (53,656).
- Split by whole story, before tokenizing, so no story is in two splits.
- Train a byte-level BPE tokenizer on training stories only (first 200,000).
- Tokenize, add
<eos>after each story, join, and cut into blocks of 512 tokens with no overlap.
| Split | Stories | Tokens | Blocks of 512 |
|---|---|---|---|
| Train | 775,270 | 163,954,618 | 320,223 |
| Validation | 7,911 | 1,677,349 | 3,276 |
| Test | 7,911 | 1,665,592 | 3,253 |
Scaling-law calculations
Planning rule from the lab (a simplified Chinchilla): D = 20 N and C = 6 N D.
| Parameters N | 64,365,440 |
| Planned tokens D = 20 N | 1,287,308,800 |
| Planned compute C = 6 N D | 4.97e+17 FLOPs |
| Tokens actually processed | 113,737,728 (8.8% of plan) |
| Scored next-token targets | 113,515,584 |
| Actual compute, 6 N D | 4.39e+16 FLOPs |
| Passes over the training tokens | 0.69 (no block was seen twice) |
| Full plan at the measured speed | about 17 hours on the same hardware |
Read the other way: for the compute I actually used, the same rule gives an optimal size of N = sqrt(C / 120) = 19M parameters. So this model is larger than its training budget justifies.
What I saw. The loss was still falling at the end, but slowly: validation went from 1.93 at 30M tokens to 1.65 at 62M and 1.51 at 114M. Each doubling of data bought less. The 20:1 ratio was measured on broad web text. TinyStories is much simpler and more repetitive, so the model runs out of new things to learn from it sooner. The rule sets the plan, but how many tokens are worth processing also depends on how hard the data is. On this corpus I expect the remaining 91% of the plan to bring a modest gain, not a large one. I did not test that.
Training setup
| Hardware | 2 x Tesla T4 (Kaggle), mixed precision (float16) |
| Optimizer | AdamW, betas (0.9, 0.95), weight decay 0.1 on weight matrices only |
| Learning rate | 0.0006 peak, 100 warmup updates, cosine decay to 6e-05 |
| Effective batch | 64 sequences x 512 tokens = 32,768 tokens per update (16 per GPU x 2 GPUs x 2 accumulation) |
| Gradient clipping | 1.0 |
| Training steps | 3,471 optimizer updates |
| Training duration | 90 minutes (21,062 tokens per second) |
| Seed | 42 |
Training was limited by time, not by steps: the cosine schedule follows the clock, so the learning rate reaches its floor exactly when the time budget ends.
Evaluation
Held-out next-token cross-entropy (nats) and perplexity, on stories the model and the tokenizer never saw.
| Split | Loss | Perplexity | Scored tokens |
|---|---|---|---|
| Validation (full) | 1.521 | 4.57 | 1,674,036 |
| Test (full) | 1.526 | 4.60 | 1,662,283 |
Before training, the loss on the validation subset was 9.14 (a uniform guess is ln 8192 = 9.01). After training it was 1.51. Perplexity only compares with models that use this same tokenizer and data.
Loss figure
Training loss is from one batch every 10 updates (the dark line is a moving average). Validation loss is from a fixed subset of 640 blocks every 100 updates. Every training batch is new data, so the two curves sit close together.
Example outputs
Sampling with temperature 0.8, top-k 40, fixed seeds. Not cherry-picked.
Prompt: Once upon a time, a little robot found a book.
Once upon a time, a little robot found a book. The book was very tall, with lots of pages. The robot was very happy. It wanted to keep the book and read it all day. The robot tried to open the book, but it was too hard. The robot was sad. One day, the robot saw a big, red ball. The robot wanted to help the ball. The robot had an idea. It called the red ball, "Hello, red ball!" The red ball opened the book. The robot was very happy. The robot and the red ball became good friends. They played together in the park every day. The robot and the red ball became best friends. They were very happy.
Prompt: Lily and her dog went to the park. Suddenly,
Lily and her dog went to the park. Suddenly, Lily saw a little boy crying. She asked him what was wrong. He told her about his dog and his ice-cream. Lily was compassionate and kind. She said, "Don't worry, we can share the ice-cream with your dog." The little boy smiled and they shared the ice-cream. It was a great day at the park.
Prompt: The old tree in the garden had a secret.
The old tree in the garden had a secret. A secret that no one could find. The people in the garden laughed and said they would never find the secret again. They kept the secret and the old tree stayed there forever.
Limitations
- Undertrained. It saw 8.8% of the planned token budget.
- One narrow domain. It only knows simple children's stories. It cannot answer questions, follow instructions, or write about anything else.
- Synthetic data. TinyStories was written by GPT-3.5 and GPT-4, so the stories repeat the same names, plots and morals, and the model copies that.
- Weak logic. The grammar is good but the events often make no sense. In the samples above the book is "very tall", a ball opens the book, and the tree's secret is never told.
- Repetition and thin plots. It repeats itself ("became good friends" and then "became best friends") and some stories end after two or three sentences.
- English, ASCII only. Other languages and symbols split into bytes and give poor results.
- Single run. One seed, no hyperparameter search. One run does not prove a scaling law.
Credits
Yazeed Alhifzi. The training code was written with Claude. Based on the course lab "From a compute budget to a 75M-parameter language model". References: Hoffmann et al. 2022 (Chinchilla), Eldan and Li 2023 (TinyStories).
- Downloads last month
- 61
