--- license: mit language: - en library_name: transformers pipeline_tag: text-generation tags: - tinyllm - transformer - causal-lm - tinystories - storytelling - pytorch - small-language-model - from-scratch datasets: - roneneldan/TinyStories --- # TinyStories-17M **TinyStories-17M** is a **17.2 million parameter** decoder-only Transformer trained **entirely from scratch** for children's story generation. The project explores how far a modern Small Language Model (SLM) can be pushed using high-quality synthetic data, an efficient Transformer architecture, and careful training. Despite its compact size, the model is capable of generating coherent, grammatically correct, multi-paragraph children's stories while following a wide variety of prompts. --- ## References This project is inspired by the TinyStories paper: **TinyStories: How Small Can Language Models Be and Still Speak Coherent English?** Ronen Eldan, Yuanzhi Li (2023) Paper: https://arxiv.org/abs/2305.07759 Dataset: https://huggingface.co/datasets/roneneldan/TinyStories # Highlights - Trained **from scratch** - **17.2M parameters** - Trained on **2.1 million** TinyStories-style synthetic stories - Modern LLaMA-style architecture - Rotary Position Embeddings (RoPE) - RMSNorm - SwiGLU Feed Forward Network - PyTorch Scaled Dot Product Attention (Flash Attention compatible) - Tied Input & Output Embeddings - Fully compatible with Hugging Face Transformers --- # Model Architecture | Property | Value | |-----------|------:| | Architecture | Decoder-only Transformer | | Parameters | **17,234,304** | | Layers | 8 | | Hidden Size | 384 | | Attention Heads | 6 | | Feed Forward Size | 1024 | | Context Length | 512 | | Vocabulary Size | 8000 | | Positional Encoding | Rotary Embeddings (RoPE) | | Activation | SwiGLU | | Normalization | RMSNorm | | Weight Tying | Yes | --- # Training The model was trained **from scratch** using next-token prediction on a synthetic TinyStories-style dataset. ### Training Configuration | Setting | Value | |---------|------:| | Optimizer | AdamW | | Precision | BF16 Mixed Precision | | Learning Rate | 3e-4 | | Weight Decay | 0.1 | | Epochs | 8 | | Batch Size | 70 | | Context Length | 512 | | Vocabulary | 8000 | Training was performed using PyTorch with Automatic Mixed Precision (AMP) and a cosine learning rate schedule. --- # Dataset The model was trained on approximately **2.1 million** synthetic English children's stories inspired by the TinyStories dataset. The dataset focuses on: - Simple English - Short coherent narratives - Moral lessons - Everyday situations - Dialogues - Fantasy adventures - Animal stories - Child-friendly vocabulary --- # Intended Use TinyStories-17M is intended for: - Story generation - Small Language Model research - Educational projects - Transformer architecture experiments - Learning how decoder-only LLMs work - Efficient inference research --- # Evaluation The model was evaluated using a combination of: - Validation loss during training - Manual inspection of generated stories - LLM-assisted qualitative evaluation using ChatGPT-5 Evaluation focused on: - Grammar - Story coherence - Prompt following - Creativity - Character consistency - Repetition - Overall story quality ## Qualitative Assessment | Category | Score | |-----------|------:| | Grammar | **9.7 / 10** | | Story Coherence | **8.7 / 10** | | Character Consistency | **8.6 / 10** | | Creativity | **8.1 / 10** | | Repetition Resistance | **8.0 / 10** | | Overall Story Quality | **8.62 / 10** | > **Note:** These scores are qualitative assessments based on manual inspection and LLM-assisted evaluation. They are intended to provide an approximate indication of the model's strengths and limitations rather than standardized benchmark results. --- # Generation Examples The following examples are **actual generations** produced by TinyStories-17M. --- ## Example 1 ### Prompt ```text Once upon a time ``` ### Generation ```text there was a little girl who was very curious. One day she decided to explore the woods and take a look around. As she walked, the little girl heard some strange noises coming from behind some trees.She stopped and looked around but couldn't see anything. Suddenly, a voice said, "Help me!" The little girl was scared but she asked, "Who are you?" The voice replied, "I am a giant. I need to tell you that you must leave these woods and not come back." The little girl knew she had to do something... ``` --- ## Example 2 ### Prompt ```text Ben discovered an old compass hidden inside his grandfather's attic. ``` ### Generation ```text He decided to explore the attic, but it was very dark and scary. He was scared, but he was determined to find out what was behind the boxes. He opened one of the boxes and found a heavy chest. He tried to lift it, but it was too heavy. He wondered what could be inside. Suddenly, he heard a mysterious voice introducing itself as the spirit of the attic, encouraging Ben to continue exploring and discover new adventures... ``` --- ## Example 3 ### Prompt ```text A lonely dragon met a tiny mouse for the first time. ``` ### Generation ```text The dragon wanted to play with the mouse, but the mouse was scared and ran away. Hoping to become friends, the dragon searched for the tiny mouse throughout the forest. Soon a misunderstanding turned into an exciting chase, and both of them learned that first impressions are not always true... ``` --- > **Note:** The examples above are intentionally truncated for brevity. The model continues generating complete children's stories beyond these excerpts. --- # Usage ```python from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained( "kaushik-harsh-99/TinyStories-17M", trust_remote_code=True, ) model = AutoModelForCausalLM.from_pretrained( "kaushik-harsh-99/TinyStories-17M", trust_remote_code=True, ) prompt = "Once upon a time" inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate( **inputs, max_new_tokens=200, temperature=0.8, top_p=0.95, do_sample=True, ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` --- # Repository Structure ```text model.safetensors config.json generation_config.json configuration_tiny.py modeling_tiny.py __init__.py tokenizer.json tokenizer.model tokenizer_config.json special_tokens_map.json README.md LICENSE ``` --- # Known Limitations As a **17M parameter** language model, TinyStories-17M has several limitations: - May become repetitive during long generations. - Limited world knowledge. - Occasionally produces repetitive dialogue. - Sometimes struggles with uncommon names or complex entities. - Optimized specifically for children's storytelling rather than general-purpose language tasks. --- # Future Work The next version of this project focuses on improving both the dataset and training methodology. Planned improvements include: - Training on approximately **3.7 million** curated stories. - Greater diversity of character names and story settings. - Continued pretraining from the current checkpoint. - Larger effective batch size using gradient accumulation. - Longer training schedules. - Improved tokenizer. - Architecture and efficiency experiments. --- # Acknowledgements This project was inspired by: - TinyStories Special thanks to the open-source AI community for making efficient language model research accessible.