MiniLLM: A Modern Transformer Language Model Built From Scratch
A small decoder-only language model trained from scratch on the TinyStories dataset. Built as a hands-on learning project implementing modern architectural techniques (RoPE, GQA, SwiGLU, RMSNorm) from first principles.
Code + training script: GitHub repo
Model Details
| Parameters | 18,536,832 (~18.5M) |
| Model size (fp32) | 74.15 MB |
| Model size (fp16/bf16) | 37.07 MB |
| Layers | 8 |
| d_model | 384 |
| Attention heads | 8 (query) / 2 (key-value), GQA |
| Context length | 256 tokens |
| Vocabulary | 16,000 (custom byte-level BPE, trained from scratch) |
| Training data | 1,000,000 TinyStories (~220M tokens) |
| Training steps | 13,000 (≈1 epoch) |
| Final val loss / perplexity | 1.63 / 5.10 |
Architecture
- RMSNorm (pre-norm)
- Rotary Positional Embeddings (RoPE)
- Grouped Query Attention (GQA)
- SwiGLU feed-forward network
- Weight-tied embedding/output layers
Usage
This model requires the custom MiniLLM architecture class to load (not a standard transformers architecture). See the GitHub repo for the full model code and a ready-to-run inference.py.
git clone https://github.com/IbrahimKhan7208/minillm-tinystories
cd minillm-tinystories
pip install -r requirements.txt
# download minillm_final.pt and tokenizer.json from this Hugging Face repo into weights/
python inference.py --prompt "Once upon a time" --temperature 0.8
Sample Output
Prompt: "Once upon a time"
Once upon a time, there was a little girl named Lily. She loved to play outside and explore. One day, she went on a walk and saw a big bright sun in the sky...
Intended Use & Limitations
Trained on a small, simple-vocabulary children's-story dataset for educational purposes. Not suitable for general-purpose or production text generation. No safety filtering has been applied — treat generations accordingly for a children's-story-style dataset.
Training Details
Optimizer: AdamW, LR 3e-4 (constant), gradient clipping at 1.0, batch size 64, ~13,000 steps (≈1 epoch over 1M TinyStories). Trained on a single Colab T4 GPU, ~3.5-4 hours total (run across multiple sessions using manual checkpoint save/resume).