|
|
--- |
|
|
language: en |
|
|
license: mit |
|
|
tags: |
|
|
- pytorch |
|
|
- language-model |
|
|
- transformer |
|
|
- tiny-shakespeare |
|
|
library_name: transformers |
|
|
model_name: mini-language-model |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Mini Language Model |
|
|
|
|
|
## 🧠 Model Description |
|
|
This is a toy decoder-only language model based on a TransformerDecoder architecture. It was trained from scratch on the [Tiny Shakespeare dataset](https://huggingface.co/datasets/tiny_shakespeare) using PyTorch. |
|
|
|
|
|
The goal was to explore autoregressive language modeling using minimal resources and libraries like torch.nn and transformers. |
|
|
|
|
|
## 🏋️ Training Details |
|
|
- **Architecture**: TransformerDecoder |
|
|
- **Tokenizer**: GPT2Tokenizer from Hugging Face |
|
|
- **Vocabulary Size**: 50257 (from GPT-2) |
|
|
- **Sequence Length**: 64 tokens |
|
|
- **Batch Size**: 8 |
|
|
- **Epochs**: 5 |
|
|
- **Learning Rate**: 1e-3 |
|
|
- **Number of Parameters**: ~900k |
|
|
- **Hardware**: Trained on CPU (Google Colab) |
|
|
|
|
|
## 📊 Evaluation |
|
|
The model was evaluated on a 10% validation split. It shows consistent training and validation loss decrease, though it is not expected to produce coherent long text due to the small training size. |
|
|
|
|
|
## 📂 Intended Use |
|
|
This model is intended for educational purposes only. It is **not suitable for production use**. |
|
|
|
|
|
## 🚫 Limitations |
|
|
- Only trained on a tiny dataset |
|
|
- Small architecture, limited capacity |
|
|
- Limited ability to generalize or generate meaningful long text |
|
|
|
|
|
## 💬 Example Usage (Python) |
|
|
python |
|
|
from transformers import GPT2Tokenizer |
|
|
from model import MiniDecoderModel # Assuming you restore the class |
|
|
|
|
|
tokenizer = GPT2Tokenizer.from_pretrained("Pavloria/mini-language-model") |
|
|
model = MiniDecoderModel(...) # Load your config |
|
|
model.load_state_dict(torch.load("pytorch_model.bin")) |
|
|
|
