File size: 2,920 Bytes
83feb4e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
language: en
tags:
- text-generation
- gpt2
- from-scratch
- cpu-training
- instruction-tuning
datasets:
- tatsu-lab/alpaca
inference: true
---
# CPU Chat LLM
A GPT-2 family language model trained **entirely from scratch on CPU** using the Alpaca instruction dataset. This project demonstrates training a choppable language model with limited compute resources.
## Training History
| Run | Model | Params | Training Data | Time | Steps | Epochs | Min Loss |
|---|---|---|---|---|---|---|---|
| 1 | 51M Base | 51.1M | Full Alpaca (52k) | 28 min | 848 | ~0.02 | 2.85 |
| 2 | 51M v2 | 51.1M | 500 short examples | 10 min | ~350 | 7 | 2.07 |
| 3 | 16M Chat | 16.1M | 500 short examples | 10 min | 460 | 16 | 1.98 |
| 4 | 51M v3 | 51.1M | 10k filtered Alpaca | 10 min+ | in-progress | - | - |
All models were initialized with random weights and trained from scratch — no pre-training or transfer learning.
## Available Models on Hugging Face
| Model ID | Params | Size | Description |
|---|---|---|---|
| `USAGAMES365/cpu-chat-llm/models/v2-51m` | 51.1M | 204 MB | Retrained 500 short examples, 7 epochs |
| `USAGAMES365/cpu-chat-llm/models/v3-51m` | 51.1M | 204 MB | Continued training on 10k examples (in progress) |
| `USAGAMES365/cpu-chat-llm/models/gemma-270m` | 268M | 1.0 GB | Gemma-3-270m-it for comparison (pre-trained, not ours) |
## Usage
```python
from transformers import GPT2LMHeadModel, AutoTokenizer
# Load our model
model = GPT2LMHeadModel.from_pretrained("USAGAMES365/cpu-chat-llm/models/v2-51m")
tokenizer = AutoTokenizer.from_pretrained("USAGAMES365/cpu-chat-llm/models/v2-51m")
# Format prompt correctly
prompt = "<|User|> What is 2+2?</s><|Assistant|>"
inputs = tokenizer.encode(prompt, return_tensors="pt")
# Generate
outputs = model.generate(inputs, max_new_tokens=80, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## Training Details
- **Architecture:** GPT-2 decoder-only transformer
- **Config (51M):** 8 layers, 8 heads, 512 hidden dim, 2048 FF dim, 192-256 context
- **Config (16M):** 6 layers, 4 heads, 256 hidden dim, 512 FF dim, 192 context
- **Tokenizer:** GPT-2 tokenizer (50,257 vocab)
- **Optimizer:** AdamW (lr=3e-4 to 1e-3, weight_decay=0.01)
- **Batch size:** 4-16
- **Hardware:** CPU only (Replit environment)
- **Software:** PyTorch, Transformers, Datasets
## Run Structure
The repo organizes checkpoints from each training phase:
- `previous-run/` - Initial training runs with step-level checkpoints
- `current-run/` - Continued training with saved checkpoints
- `models/` - Final exportable models
## Limitations
- Small model size (16M-51M params vs. billions in production models)
- Very limited training time (minutes vs. thousands of GPU-hours)
- CPU-only training constrains both speed and model capacity
- Outputs may be incoherent or repetitive
- Not suitable for production deployment
|