Llama-TinyStories

This model is for educational purposes only. If you are looking for the actual TinyStories model to use, refer to the TinyStories-33M by Eldan & Li.

Reproduction of the TinyStories (Eldan & Li, 2023) model using a LLaMA-style architecture, trained on ~1B tokens with nanotron.

Read the accompanying blog post for a detailed walkthrough of the training process.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model = AutoModelForCausalLM.from_pretrained('vinothkumarn/Llama-TinyStories')
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-125M")
prompt = "Once upon a time there was"
input_ids = tokenizer.encode(prompt, return_tensors="pt")

# Generate completion
output = model.generate(input_ids, max_length = 256, num_beams=1)

# Decode the completion
output_text = tokenizer.decode(output[0], skip_special_tokens=True)

# Print the generated text
print(output_text)

Model Details

Config	Value
Architecture	LLaMA (SwiGLU, RoPE, RMSNorm, no bias)
Parameters	~66.9M total / ~28.3M non-embedding
Layers / Heads / Hidden	4 / 16 / 768
Intermediate size	2,048
Context length	512 tokens
Training tokens	~1B
Tokenizer	EleutherAI/gpt-neo-125M (vocab 50,257)
Framework	nanotron

Limitations

• English only, children's story domain
• Educational/research use — not for production
• Can be repetitive on long generations (only 4 layers)

License

MIT. Training data licensed under CDLA-Sharing-1.0.

Citation

@article{eldan2023tinystories,
  title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
  author={Eldan, Ronen and Li, Yuanzhi},
  journal={arXiv preprint arXiv:2305.07759},
  year={2023}
}