File size: 2,127 Bytes
32d0f47 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | # 🚀 MiniGPT from Scratch
A decoder-only GPT implementation built completely from scratch using PyTorch.
This project was created as part of my journey to understand how modern Large Language Models work internally instead of simply using existing libraries.
---
# Features
- ✅ Character-level Tokenizer
- ✅ Token Embeddings
- ✅ Positional Embeddings
- ✅ Multi-Head Self Attention
- ✅ FlashAttention (`scaled_dot_product_attention`)
- ✅ Feed Forward Network (MLP)
- ✅ Residual Connections
- ✅ Layer Normalization
- ✅ Decoder-only GPT Architecture
- ✅ Autoregressive Text Generation
- ✅ Top-K Sampling
- ✅ Top-P (Nucleus) Sampling
- ✅ Temperature Sampling
---
# Architecture
```
Input Tokens
│
▼
Token Embedding
│
▼
Position Embedding
│
▼
Transformer Block × 4
│
▼
LayerNorm
│
▼
Linear Head
│
▼
Vocabulary Logits
│
▼
Next Token
```
---
# Project Structure
```
MiniGPT-from-Scratch/
├── model.py
├── train.py
├── generate.py
├── utils.py
├── config.py
├── requirements.txt
├── notebook/
├── checkpoints/
├── outputs/
├── images/
└── data/
```
---
# Dataset
Tiny Shakespeare
https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
---
# Training
```bash
python train.py
```
---
# Generate Text
```bash
python generate.py
```
---
# Sample Output
```
FANH:
A Prusjortus:
Weath the too,
eyes glorls cobsuess fewn.
```
---
# Future Work
- KV Cache
- Mixed Precision
- LoRA Fine-tuning
- Beam Search
- HuggingFace Integration
- Distributed Training
- Triton Kernels
---
# Author
Built with ❤️ while learning PyTorch Internals and Transformer Architecture.
|