| --- |
| language: en |
| tags: |
| - text-generation |
| - gpt2 |
| - from-scratch |
| - cpu-training |
| - instruction-tuning |
| datasets: |
| - tatsu-lab/alpaca |
| inference: true |
| --- |
| |
| # CPU Chat LLM |
|
|
| A GPT-2 family language model trained **entirely from scratch on CPU** using the Alpaca instruction dataset. This project demonstrates training a choppable language model with limited compute resources. |
|
|
| ## Training History |
|
|
| | Run | Model | Params | Training Data | Time | Steps | Epochs | Min Loss | |
| |---|---|---|---|---|---|---|---| |
| | 1 | 51M Base | 51.1M | Full Alpaca (52k) | 28 min | 848 | ~0.02 | 2.85 | |
| | 2 | 51M v2 | 51.1M | 500 short examples | 10 min | ~350 | 7 | 2.07 | |
| | 3 | 16M Chat | 16.1M | 500 short examples | 10 min | 460 | 16 | 1.98 | |
| | 4 | 51M v3 | 51.1M | 10k filtered Alpaca | 10 min+ | in-progress | - | - | |
|
|
| All models were initialized with random weights and trained from scratch — no pre-training or transfer learning. |
|
|
| ## Available Models on Hugging Face |
|
|
| | Model ID | Params | Size | Description | |
| |---|---|---|---| |
| | `USAGAMES365/cpu-chat-llm/models/v2-51m` | 51.1M | 204 MB | Retrained 500 short examples, 7 epochs | |
| | `USAGAMES365/cpu-chat-llm/models/v3-51m` | 51.1M | 204 MB | Continued training on 10k examples (in progress) | |
| | `USAGAMES365/cpu-chat-llm/models/gemma-270m` | 268M | 1.0 GB | Gemma-3-270m-it for comparison (pre-trained, not ours) | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import GPT2LMHeadModel, AutoTokenizer |
| |
| # Load our model |
| model = GPT2LMHeadModel.from_pretrained("USAGAMES365/cpu-chat-llm/models/v2-51m") |
| tokenizer = AutoTokenizer.from_pretrained("USAGAMES365/cpu-chat-llm/models/v2-51m") |
| |
| # Format prompt correctly |
| prompt = "<|User|> What is 2+2?</s><|Assistant|>" |
| inputs = tokenizer.encode(prompt, return_tensors="pt") |
| |
| # Generate |
| outputs = model.generate(inputs, max_new_tokens=80, temperature=0.7, do_sample=True) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Training Details |
|
|
| - **Architecture:** GPT-2 decoder-only transformer |
| - **Config (51M):** 8 layers, 8 heads, 512 hidden dim, 2048 FF dim, 192-256 context |
| - **Config (16M):** 6 layers, 4 heads, 256 hidden dim, 512 FF dim, 192 context |
| - **Tokenizer:** GPT-2 tokenizer (50,257 vocab) |
| - **Optimizer:** AdamW (lr=3e-4 to 1e-3, weight_decay=0.01) |
| - **Batch size:** 4-16 |
| - **Hardware:** CPU only (Replit environment) |
| - **Software:** PyTorch, Transformers, Datasets |
| |
| ## Run Structure |
| |
| The repo organizes checkpoints from each training phase: |
| - `previous-run/` - Initial training runs with step-level checkpoints |
| - `current-run/` - Continued training with saved checkpoints |
| - `models/` - Final exportable models |
| |
| ## Limitations |
| |
| - Small model size (16M-51M params vs. billions in production models) |
| - Very limited training time (minutes vs. thousands of GPU-hours) |
| - CPU-only training constrains both speed and model capacity |
| - Outputs may be incoherent or repetitive |
| - Not suitable for production deployment |
| |
