|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- tiny |
|
|
- from-scratch |
|
|
- educational |
|
|
- causal-lm |
|
|
- personal-llm |
|
|
model-index: |
|
|
- name: tiny-llm-54m |
|
|
results: [] |
|
|
--- |
|
|
|
|
|
# Tiny-LLM 54M |
|
|
|
|
|
A small transformer language model (~54.93M parameters) trained from scratch for educational and experimental purposes. |
|
|
|
|
|
## Model Description |
|
|
|
|
|
This is a decoder-only transformer trained from scratch on Wikipedia text. It demonstrates that meaningful language models can be trained on consumer hardware with modest compute budgets. |
|
|
|
|
|
### Architecture |
|
|
|
|
|
| Component | Value | |
|
|
|-----------|-------| |
|
|
| Parameters | **54.93M** | |
|
|
| Layers | 12 | |
|
|
| Hidden Size | 512 | |
|
|
| Attention Heads | 8 | |
|
|
| Intermediate (FFN) | 1408 | |
|
|
| Vocab Size | 32,000 | |
|
|
| Max Sequence Length | 512 | |
|
|
| Position Encoding | RoPE | |
|
|
| Normalization | RMSNorm | |
|
|
| Activation | SwiGLU | |
|
|
| Weight Tying | Yes | |
|
|
|
|
|
### Training Details |
|
|
|
|
|
| Parameter | Value | |
|
|
|-----------|-------| |
|
|
| Training Steps | 50,000 | |
|
|
| Tokens | ~100M | |
|
|
| Batch Size | 32 | |
|
|
| Learning Rate | 3e-4 | |
|
|
| Warmup Steps | 2,000 | |
|
|
| Weight Decay | 0.1 | |
|
|
| Hardware | NVIDIA RTX 5090 (32GB) | |
|
|
| Training Time | ~3 hours | |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import AutoTokenizer |
|
|
|
|
|
# Load tokenizer (uses standard GPT-2 style tokenizer) |
|
|
tokenizer = AutoTokenizer.from_pretrained("jonmabe/tiny-llm-54m") |
|
|
|
|
|
# For custom model loading, see the model files |
|
|
# This model uses a custom architecture - see scripts/ for inference code |
|
|
``` |
|
|
|
|
|
### Generation Example |
|
|
|
|
|
```python |
|
|
# Note: This model uses a custom architecture |
|
|
# Full inference code available in the repository |
|
|
|
|
|
prompt = "The history of artificial intelligence" |
|
|
# Model generates continuation based on learned Wikipedia patterns |
|
|
``` |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
- **Educational**: Understanding transformer training from scratch |
|
|
- **Experimental**: Testing fine-tuning approaches on small models |
|
|
- **Personal LLM**: Base for personal voice/style fine-tuning |
|
|
- **Research**: Lightweight model for NLP experiments |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- Small model size limits knowledge and capabilities |
|
|
- Trained only on Wikipedia - limited domain coverage |
|
|
- Not suitable for production use cases requiring high quality |
|
|
- May generate factually incorrect information |
|
|
- No RLHF or instruction tuning |
|
|
|
|
|
## Training Data |
|
|
|
|
|
- **Source**: Wikipedia (English) |
|
|
- **Processing**: Tokenized with 32K vocabulary SentencePiece tokenizer |
|
|
- **Format**: Standard causal language modeling (next token prediction) |
|
|
|
|
|
## Future Work |
|
|
|
|
|
This model is intended as a base for: |
|
|
1. **Personal Fine-tuning**: Adapt to individual writing style using personal data |
|
|
2. **Domain Adaptation**: Specialize for specific topics or tasks |
|
|
3. **Instruction Tuning**: Add instruction-following capabilities |
|
|
|
|
|
## Hardware Requirements |
|
|
|
|
|
- **Inference**: ~300MB GPU memory, runs on any modern GPU or Apple Silicon |
|
|
- **Fine-tuning**: ~2GB GPU memory recommended |
|
|
|
|
|
## Related Work |
|
|
|
|
|
Inspired by: |
|
|
- Andrej Karpathy's nanoGPT |
|
|
- Geddy Duke's small LLM experiments |
|
|
- LLaMA architecture design choices |
|
|
|
|
|
## Citation |
|
|
|
|
|
```bibtex |
|
|
@misc{tiny-llm-54m, |
|
|
author = {jonmabe}, |
|
|
title = {Tiny-LLM: A 54M Parameter Language Model}, |
|
|
year = {2026}, |
|
|
publisher = {Hugging Face}, |
|
|
url = {https://huggingface.co/jonmabe/tiny-llm-54m} |
|
|
} |
|
|
``` |
|
|
|
