---
language:
- en
license: mit
tags:
- gpt2
- language-model
- causal-lm
- tinystories
- from-scratch
- pytorch
- apple-silicon
datasets:
- roneneldan/TinyStories
pipeline_tag: text-generation
---

# GPT-2 (163M) — Trained from Scratch on TinyStories

Mô hình GPT-2 architecture (163M parameters) được **train từ đầu** (from scratch) bằng PyTorch thuần, trên dataset [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). Đây là sản phẩm của dự án học tập [LEARN-LLM](https://github.com/Tung003/LEARN-LLM).

---

## 🔢 Model Architecture

| Tham số | Giá trị |
|---------|---------|
| Architecture | GPT-2 (Decoder-only Transformer) |
| Vocab size | 50,257 (GPT-2 tiktoken BPE) |
| Context length | 1,024 tokens |
| Embedding dim | 768 |
| Attention heads | 12 |
| Transformer layers | 12 |
| Feed-forward dim | 3,072 (4×) |
| **Total parameters** | **~163M** |

> **Note**: Không dùng weight-tying giữa `tok_emb` và `out_head` nên 163M thay vì 124M của GPT-2 gốc.

---

## 🏋️ Training Details

| Chi tiết | Giá trị |
|---------|---------|
| Dataset | roneneldan/TinyStories |
| Tokenizer | GPT-2 (tiktoken) |
| Optimizer | AdamW |
| Learning rate | 6e-4 (cosine decay + warmup) |
| Warmup steps | 2,000 |
| Effective batch size | 64 (microbatch=6, accum=16) |
| Context length | 1,024 |
| Hardware | Apple M5 Pro (MPS) |
| Checkpoint step | 49,000 |

---

## 📦 Files

| File | Mô tả |
|------|-------|
| `best_checkpoint.pth` | Checkpoint với val loss thấp nhất |
| `last_checkpoint.pth` | Checkpoint cuối cùng (dùng để resume) |

---

## 🚀 Sử dụng

### Load model

```python
import torch
import tiktoken
import sys

# Clone repo để có model code
# git clone https://github.com/Tung003/LEARN-LLM.git

sys.path.insert(0, "LEARN-LLM/notebooks")
from chapter_3_models.artifacts.gpt_model import GPTModel
from chapter_3_models.artifacts.generate import generate_text_simple

# Download checkpoint từ HF
from huggingface_hub import hf_hub_download

ckpt_path = hf_hub_download(
    repo_id="TungChu/gpt2",
    filename="best_checkpoint.pth"
)

# Load model
checkpoint = torch.load(ckpt_path, map_location="cpu")
model = GPTModel(checkpoint["model_config"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# Generate text
tokenizer = tiktoken.get_encoding("gpt2")
prompt = "Once upon a time"
tokens = tokenizer.encode(prompt)
idx = torch.tensor([tokens])

with torch.no_grad():
    out = generate_text_simple(model, idx, max_new_tokens=100, context_size=1024)

print(tokenizer.decode(out[0].tolist()))
```

---

## 📊 Training Results

| Steps | Val Loss | Ghi chú |
|-------|----------|---------|
| 6,000 | 2.35 | Early checkpoint |
| 49,000 | — | Current checkpoint |

---

## 🔗 Links

- 📂 **Source code**: [github.com/Tung003/LEARN-LLM](https://github.com/Tung003/LEARN-LLM)
- 📖 **Dataset**: [roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)

---

## 📄 License

MIT License — Tự do sử dụng cho mục đích học tập và nghiên cứu.