README.md

---
license: mit
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- tinyllm
- transformer
- causal-lm
- tinystories
- storytelling
- pytorch
- small-language-model
- from-scratch
datasets:
- roneneldan/TinyStories
---

# TinyStories-17M

**TinyStories-17M** is a **17.2 million parameter** decoder-only Transformer trained **entirely from scratch** for children's story generation.

The project explores how far a modern Small Language Model (SLM) can be pushed using high-quality synthetic data, an efficient Transformer architecture, and careful training. Despite its compact size, the model is capable of generating coherent, grammatically correct, multi-paragraph children's stories while following a wide variety of prompts.

---
## References

This project is inspired by the TinyStories paper:

**TinyStories: How Small Can Language Models Be and Still Speak Coherent English?**

Ronen Eldan, Yuanzhi Li (2023)

Paper: https://arxiv.org/abs/2305.07759

Dataset: https://huggingface.co/datasets/roneneldan/TinyStories

# Highlights

-  Trained **from scratch**
-  **17.2M parameters**
-  Trained on **2.1 million** TinyStories-style synthetic stories
-  Modern LLaMA-style architecture
-  Rotary Position Embeddings (RoPE)
-  RMSNorm
-  SwiGLU Feed Forward Network
-  PyTorch Scaled Dot Product Attention (Flash Attention compatible)
-  Tied Input & Output Embeddings
-  Fully compatible with Hugging Face Transformers

---

# Model Architecture

| Property | Value |
|-----------|------:|
| Architecture | Decoder-only Transformer |
| Parameters | **17,234,304** |
| Layers | 8 |
| Hidden Size | 384 |
| Attention Heads | 6 |
| Feed Forward Size | 1024 |
| Context Length | 512 |
| Vocabulary Size | 8000 |
| Positional Encoding | Rotary Embeddings (RoPE) |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Weight Tying | Yes |

---

# Training

The model was trained **from scratch** using next-token prediction on a synthetic TinyStories-style dataset.

### Training Configuration

| Setting | Value |
|---------|------:|
| Optimizer | AdamW |
| Precision | BF16 Mixed Precision |
| Learning Rate | 3e-4 |
| Weight Decay | 0.1 |
| Epochs | 8 |
| Batch Size | 70 |
| Context Length | 512 |
| Vocabulary | 8000 |

Training was performed using PyTorch with Automatic Mixed Precision (AMP) and a cosine learning rate schedule.

---

# Dataset

The model was trained on approximately **2.1 million** synthetic English children's stories inspired by the TinyStories dataset.

The dataset focuses on:

- Simple English
- Short coherent narratives
- Moral lessons
- Everyday situations
- Dialogues
- Fantasy adventures
- Animal stories
- Child-friendly vocabulary

---

# Intended Use

TinyStories-17M is intended for:

- Story generation
- Small Language Model research
- Educational projects
- Transformer architecture experiments
- Learning how decoder-only LLMs work
- Efficient inference research

---

# Evaluation

The model was evaluated using a combination of:

- Validation loss during training
- Manual inspection of generated stories
- LLM-assisted qualitative evaluation using ChatGPT-5

Evaluation focused on:

- Grammar
- Story coherence
- Prompt following
- Creativity
- Character consistency
- Repetition
- Overall story quality

## Qualitative Assessment

| Category | Score |
|-----------|------:|
| Grammar | **9.7 / 10** |
| Story Coherence | **8.7 / 10** |
| Character Consistency | **8.6 / 10** |
| Creativity | **8.1 / 10** |
| Repetition Resistance | **8.0 / 10** |
| Overall Story Quality | **8.62 / 10** |

> **Note:** These scores are qualitative assessments based on manual inspection and LLM-assisted evaluation. They are intended to provide an approximate indication of the model's strengths and limitations rather than standardized benchmark results.

---

# Generation Examples

The following examples are **actual generations** produced by TinyStories-17M.

---

## Example 1

### Prompt

```text
Once upon a time
```

### Generation

```text
there was a little girl who was very curious. One day she decided to
explore the woods and take a look around. As she walked, the little
girl heard some strange noises coming from behind some trees.She
stopped and looked around but couldn't see anything. Suddenly, a
voice said, "Help me!" The little girl was scared but she asked,
"Who are you?" The voice replied, "I am a giant. I need to tell you
that you must leave these woods and not come back." The little girl
knew she had to do something...
```

---

## Example 2

### Prompt

```text
Ben discovered an old compass hidden inside his grandfather's attic.
```

### Generation

```text
 He decided to explore the attic, but it was very dark and scary. He was
 scared, but he was determined to find out what was behind the boxes. He
 opened one of the boxes and found a heavy chest. He tried to lift it,
 but it was too heavy. He wondered what could be inside. Suddenly, he
 heard a mysterious voice introducing itself as the spirit of the attic,
 encouraging Ben to continue exploring and discover new adventures...
```

---

## Example 3

### Prompt

```text
A lonely dragon met a tiny mouse for the first time.
```

### Generation

```text
The dragon wanted to play with the mouse, but the mouse was scared and ran
away. Hoping to become friends, the dragon searched for the tiny mouse
throughout the forest. Soon a misunderstanding turned into an exciting
chase, and both of them learned that first impressions are not always true...
```

---

> **Note:** The examples above are intentionally truncated for brevity. The model continues generating complete children's stories beyond these excerpts.

---

# Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained(
    "kaushik-harsh-99/TinyStories-17M",
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "kaushik-harsh-99/TinyStories-17M",
    trust_remote_code=True,
)

prompt = "Once upon a time"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.8,
    top_p=0.95,
    do_sample=True,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

# Repository Structure

```text
model.safetensors
config.json
generation_config.json

configuration_tiny.py
modeling_tiny.py
__init__.py

tokenizer.json
tokenizer.model
tokenizer_config.json
special_tokens_map.json

README.md
LICENSE
```

---

# Known Limitations

As a **17M parameter** language model, TinyStories-17M has several limitations:

- May become repetitive during long generations.
- Limited world knowledge.
- Occasionally produces repetitive dialogue.
- Sometimes struggles with uncommon names or complex entities.
- Optimized specifically for children's storytelling rather than general-purpose language tasks.

---

# Future Work

The next version of this project focuses on improving both the dataset and training methodology.

Planned improvements include:

-  Training on approximately **3.7 million** curated stories.
-  Greater diversity of character names and story settings.
-  Continued pretraining from the current checkpoint.
-  Larger effective batch size using gradient accumulation.
-  Longer training schedules.
-  Improved tokenizer.
-  Architecture and efficiency experiments.

---

# Acknowledgements

This project was inspired by:

- TinyStories

Special thanks to the open-source AI community for making efficient language model research accessible.