|
|
--- |
|
|
license: mit |
|
|
tags: |
|
|
- gpt2 |
|
|
- tinystories |
|
|
- language-model |
|
|
--- |
|
|
|
|
|
# TinyStories-GPT |
|
|
|
|
|
This is a small GPT-like model trained from scratch on the [TinyStories dataset](https://huggingface.co/datasets/roneneldan/TinyStories). |
|
|
It was implemented using a NanoGPT-style training loop in PyTorch. |
|
|
|
|
|
## Model Details |
|
|
- **Architecture:** 6 layers, 6 heads, 384 hidden size |
|
|
- **Context length:** 128 tokens |
|
|
- **Vocab size:** 50257 (GPT-2 tokenizer) |
|
|
- **Dataset:** TinyStories |
|
|
- **Training:** ~20k steps, AdamW, cosine LR decay |
|
|
|
|
|
## Example Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("Here2Disrupt/tiny-stories-gpt") |
|
|
model = AutoModelForCausalLM.from_pretrained("Here2Disrupt/tiny-stories-gpt") |
|
|
|
|
|
prompt = "Once upon a time" |
|
|
inputs = tokenizer(prompt, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=50) |
|
|
print(tokenizer.decode(outputs[0])) |
|
|
|
