|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- text-generation |
|
|
- gpt2 |
|
|
- stories |
|
|
- children-stories |
|
|
- tinystories |
|
|
datasets: |
|
|
- roneneldan/TinyStories |
|
|
widget: |
|
|
- text: "Once upon a time" |
|
|
example_title: "Story Beginning" |
|
|
- text: "The little girl loved to" |
|
|
example_title: "Character Story" |
|
|
- text: "In a magical forest," |
|
|
example_title: "Fantasy Setting" |
|
|
--- |
|
|
|
|
|
# RonMicro-LLM-Story (Phase 1) |
|
|
|
|
|
A small GPT-2 style language model trained on TinyStories dataset for generating children's stories. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Model Type:** GPT-2 Causal Language Model |
|
|
- **Parameters:** ~40M |
|
|
- **Training Data:** TinyStories (5% subset, ~105K stories) |
|
|
- **Vocabulary Size:** 25,913 tokens |
|
|
- **Context Length:** 512 tokens |
|
|
- **Training Epochs:** 3 |
|
|
- **Language:** English |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- **Framework:** Transformers (Hugging Face) |
|
|
- **Tokenizer:** Custom BPE trained on TinyStories |
|
|
- **Architecture:** |
|
|
- 6 transformer layers |
|
|
- 384 embedding dimensions |
|
|
- 6 attention heads |
|
|
- 1536 FFN dimensions |
|
|
|
|
|
## Usage |
|
|
```python |
|
|
from transformers import pipeline |
|
|
|
|
|
# Load the model |
|
|
generator = pipeline("text-generation", model="endurasolution/ronmicro-llm-story") |
|
|
|
|
|
# Generate a story |
|
|
story = generator( |
|
|
"Once upon a time", |
|
|
max_new_tokens=150, |
|
|
temperature=0.7, |
|
|
repetition_penalty=1.3, |
|
|
no_repeat_ngram_size=3, |
|
|
do_sample=True |
|
|
) |
|
|
|
|
|
print(story[0]["generated_text"]) |
|
|
``` |
|
|
|
|
|
## Example Outputs |
|
|
|
|
|
**Prompt:** "Once upon a time" |
|
|
**Output:** "Once upon a time, there was a little boy named Timmy. He loved to play with his toy cars and trucks all day long..." |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- Trained on only 5% of TinyStories (Phase 1) |
|
|
- May generate repetitive text occasionally |
|
|
- Best for short children's stories (100-200 words) |
|
|
- Limited to simple vocabulary and grammar |
|
|
|
|
|
## Next Steps |
|
|
|
|
|
Phase 2 training in progress with 20% data and 5 epochs for improved quality. |
|
|
|
|
|
## Citation |
|
|
|
|
|
Built using TinyStories dataset: |
|
|
``` |
|
|
@article{eldan2023tinystories, |
|
|
title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?}, |
|
|
author={Eldan, Ronen and Li, Yuanzhi}, |
|
|
journal={arXiv preprint arXiv:2305.07759}, |
|
|
year={2023} |
|
|
} |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
MIT License |
|
|
|
