|
|
--- |
|
|
license: apache-2.0 |
|
|
language: en |
|
|
tags: |
|
|
- causal-lm |
|
|
- from-scratch |
|
|
- transformer |
|
|
- tiny-stories |
|
|
- pytorch |
|
|
- custom-architecture |
|
|
- text-generation |
|
|
--- |
|
|
|
|
|
# TinyWay 1.0.0 |
|
|
|
|
|
**TinyWay 1.0.0** is a **52.94M parameter GPT-style causal language model** trained **from scratch** on the **TinyStories** dataset. |
|
|
The model is designed for **lightweight story generation, research, and educational exploration** of decoder-only Transformer architectures. |
|
|
|
|
|
Unlike fine-tuned models, TinyWay was **implemented, trained, serialized, and released end-to-end**, including a **custom Hugging Face-compatible architecture**. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🔍 Model Overview |
|
|
|
|
|
| Attribute | Value | |
|
|
|---------|------| |
|
|
| Architecture | Decoder-only Transformer (GPT-style) | |
|
|
| Parameters | **52.94M** | |
|
|
| Layers | 8 | |
|
|
| Hidden size | 384 | |
|
|
| Attention heads | 8 | |
|
|
| Context length | 256 tokens | |
|
|
| Tokenizer | GPT-2 BPE | |
|
|
| Framework | PyTorch | |
|
|
| Precision | FP16 (AMP during training) | |
|
|
|
|
|
--- |
|
|
|
|
|
## 📚 Training Details |
|
|
|
|
|
- **Dataset**: TinyStories (text file, streamed) |
|
|
- **Training strategy**: Streaming token dataset |
|
|
- **Epochs**: 1 |
|
|
- **Effective batch size**: 64 |
|
|
- **Optimizer**: AdamW |
|
|
- **Learning rate**: 3e-4 |
|
|
- **Dropout**: 0.1 |
|
|
- **Hardware**: NVIDIA Tesla P100 (16GB) |
|
|
- **Environment**: Kaggle |
|
|
|
|
|
The model was trained using **causal language modeling**, predicting the next token given previous tokens. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🎯 Intended Use |
|
|
|
|
|
TinyWay is suitable for: |
|
|
|
|
|
- Short story generation |
|
|
- Educational demonstrations of Transformer internals |
|
|
- Research on small-scale language models |
|
|
- Understanding end-to-end LLM construction |
|
|
|
|
|
--- |
|
|
|
|
|
## ⚠️ Limitations |
|
|
|
|
|
- Trained only on narrative-style data (TinyStories) |
|
|
- Not instruction-tuned |
|
|
- Not suitable for factual QA or reasoning-heavy tasks |
|
|
- Limited context window (256 tokens) |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 Usage |
|
|
|
|
|
### Load and generate text |
|
|
|
|
|
```python |
|
|
from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
model_id = "shivamsharma120120/TinyWay-1.0.0" |
|
|
|
|
|
config = AutoConfig.from_pretrained( |
|
|
model_id, |
|
|
trust_remote_code=True |
|
|
) |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained( |
|
|
model_id, |
|
|
trust_remote_code=True |
|
|
) |
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_id, |
|
|
config=config, |
|
|
trust_remote_code=True |
|
|
) |
|
|
|
|
|
inputs = tokenizer("Once upon a time", return_tensors="pt") |
|
|
|
|
|
output = model.generate( |
|
|
**inputs, |
|
|
max_new_tokens=100, |
|
|
temperature=0.8, |
|
|
top_p=0.95, |
|
|
do_sample=True |
|
|
) |
|
|
|
|
|
print(tokenizer.decode(output[0], skip_special_tokens=True)) |
|
|
|
