README.md

# 🚀 MiniGPT from Scratch

![Python](https://img.shields.io/badge/Python-3.10-blue)
![PyTorch](https://img.shields.io/badge/PyTorch-2.x-red)
![License](https://img.shields.io/badge/License-MIT-green)

A decoder-only GPT implementation built completely from scratch using PyTorch.

This project was created as part of my journey to understand how modern Large Language Models work internally instead of simply using existing libraries.

---

# Features

- ✅ Character-level Tokenizer
- ✅ Token Embeddings
- ✅ Positional Embeddings
- ✅ Multi-Head Self Attention
- ✅ FlashAttention (`scaled_dot_product_attention`)
- ✅ Feed Forward Network (MLP)
- ✅ Residual Connections
- ✅ Layer Normalization
- ✅ Decoder-only GPT Architecture
- ✅ Autoregressive Text Generation
- ✅ Top-K Sampling
- ✅ Top-P (Nucleus) Sampling
- ✅ Temperature Sampling

---

# Architecture

```
Input Tokens
      │
      ▼
Token Embedding
      │
      ▼
Position Embedding
      │
      ▼
Transformer Block × 4
      │
      ▼
LayerNorm
      │
      ▼
Linear Head
      │
      ▼
Vocabulary Logits
      │
      ▼
Next Token
```

---

# Project Structure

```
MiniGPT-from-Scratch/

├── model.py
├── train.py
├── generate.py
├── utils.py
├── config.py
├── requirements.txt
├── notebook/
├── checkpoints/
├── outputs/
├── images/
└── data/
```

---

# Dataset

Tiny Shakespeare

https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

---

# Training

```bash
python train.py
```

---

# Generate Text

```bash
python generate.py
```

---

# Sample Output

```
FANH:
A Prusjortus:
Weath the too,
eyes glorls cobsuess fewn.
```

---

# Future Work

- KV Cache
- Mixed Precision
- LoRA Fine-tuning
- Beam Search
- HuggingFace Integration
- Distributed Training
- Triton Kernels

---

# Author

Built with ❤️ while learning PyTorch Internals and Transformer Architecture.