---
library_name: transformers
license: mit
datasets:
- roneneldan/TinyStories
language:
- en
---

# Model Card for Model ID


## Model Details

This is a reproduction of a 3.6 million parameter language model from scratch by following the paper [**TinyStories: How Small Can Language Models Be and Still Speak
Coherent English?**](https://arxiv.org/pdf/2305.07759). The goal of this project is to demostrate that a very small transformer model, when trained on a simpliefied synthetic dataset, can generate fluent, grammatically correct and consistent short stories.  

### Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub.

- **Developed by:** Saurav Prateek
- **Model type:** Text Generationg (Transformer - Decoder model)
- **Parameters:** 3.65 Million
- **Attention Layers:** 8
- **Hidden Dimension:** 64
- **Attention Heads per Layer:** 16
- **Context Window:** 512 tokens
- **Vocab Size:** ~50K (GPT-Neo Tokenizer)
- **Learning Rate:** 5e-4
- **Language(s) (NLP):** English
- **License:** MIT

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/SauravP97/tiny-stories-hf
- **Paper [optional]:** https://arxiv.org/pdf/2305.07759


## Training Details

### Training Data

The model was trained on the TinyStories dataset, which consist of synthetic short stories generated by GPT-3.5/4. The stories use a restricted vocabulary typical of a 3-year-old child.

  - Source: [Hugging Face Datasets (roneneldan/TinyStories)](https://huggingface.co/datasets/roneneldan/TinyStories)
  - Size: ~2GB text data

### Training Procedure

The model was trained from scratch on a **NVIDIA T4** GPU for around 3 hours to achieve a loss of `2.17`. The model was trained for `0.22` epochs estimating around `55K` steps.
We used **EleutherAI/gpt-neo-125M** tokenizer model training and inference.

#### Training Hyperparameters

- **Training regime:**
  - Epochs: 0.22
  - Loss: 2.17
  - GPU: NVIDIA T4
  - Training Steps: 55,000
  - Training Time: ~3 hours


## Citation [optional]

 - https://arxiv.org/abs/2305.07759