File size: 3,124 Bytes
ccae8c0 1b3f0ab ccae8c0 1b3f0ab ccae8c0 1b3f0ab | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | ---
language:
- en
license: mit
tags:
- gpt2
- language-model
- causal-lm
- tinystories
- from-scratch
- pytorch
- apple-silicon
datasets:
- roneneldan/TinyStories
pipeline_tag: text-generation
---
# GPT-2 (163M) — Trained from Scratch on TinyStories
Mô hình GPT-2 architecture (163M parameters) được **train từ đầu** (from scratch) bằng PyTorch thuần, trên dataset [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). Đây là sản phẩm của dự án học tập [LEARN-LLM](https://github.com/Tung003/LEARN-LLM).
---
## 🔢 Model Architecture
| Tham số | Giá trị |
|---------|---------|
| Architecture | GPT-2 (Decoder-only Transformer) |
| Vocab size | 50,257 (GPT-2 tiktoken BPE) |
| Context length | 1,024 tokens |
| Embedding dim | 768 |
| Attention heads | 12 |
| Transformer layers | 12 |
| Feed-forward dim | 3,072 (4×) |
| **Total parameters** | **~163M** |
> **Note**: Không dùng weight-tying giữa `tok_emb` và `out_head` nên 163M thay vì 124M của GPT-2 gốc.
---
## 🏋️ Training Details
| Chi tiết | Giá trị |
|---------|---------|
| Dataset | roneneldan/TinyStories |
| Tokenizer | GPT-2 (tiktoken) |
| Optimizer | AdamW |
| Learning rate | 6e-4 (cosine decay + warmup) |
| Warmup steps | 2,000 |
| Effective batch size | 64 (microbatch=6, accum=16) |
| Context length | 1,024 |
| Hardware | Apple M5 Pro (MPS) |
| Checkpoint step | 49,000 |
---
## 📦 Files
| File | Mô tả |
|------|-------|
| `best_checkpoint.pth` | Checkpoint với val loss thấp nhất |
| `last_checkpoint.pth` | Checkpoint cuối cùng (dùng để resume) |
---
## 🚀 Sử dụng
### Load model
```python
import torch
import tiktoken
import sys
# Clone repo để có model code
# git clone https://github.com/Tung003/LEARN-LLM.git
sys.path.insert(0, "LEARN-LLM/notebooks")
from chapter_3_models.artifacts.gpt_model import GPTModel
from chapter_3_models.artifacts.generate import generate_text_simple
# Download checkpoint từ HF
from huggingface_hub import hf_hub_download
ckpt_path = hf_hub_download(
repo_id="TungChu/gpt2",
filename="best_checkpoint.pth"
)
# Load model
checkpoint = torch.load(ckpt_path, map_location="cpu")
model = GPTModel(checkpoint["model_config"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()
# Generate text
tokenizer = tiktoken.get_encoding("gpt2")
prompt = "Once upon a time"
tokens = tokenizer.encode(prompt)
idx = torch.tensor([tokens])
with torch.no_grad():
out = generate_text_simple(model, idx, max_new_tokens=100, context_size=1024)
print(tokenizer.decode(out[0].tolist()))
```
---
## 📊 Training Results
| Steps | Val Loss | Ghi chú |
|-------|----------|---------|
| 6,000 | 2.35 | Early checkpoint |
| 49,000 | — | Current checkpoint |
---
## 🔗 Links
- 📂 **Source code**: [github.com/Tung003/LEARN-LLM](https://github.com/Tung003/LEARN-LLM)
- 📖 **Dataset**: [roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)
---
## 📄 License
MIT License — Tự do sử dụng cho mục đích học tập và nghiên cứu.
|