README.md

metadata

language: en
tags:
  - text-generation
  - gpt2
  - from-scratch
  - cpu-training
  - instruction-tuning
datasets:
  - tatsu-lab/alpaca
inference: true

CPU Chat LLM

A GPT-2 family language model trained entirely from scratch on CPU using the Alpaca instruction dataset. This project demonstrates training a choppable language model with limited compute resources.

Training History

Run	Model	Params	Training Data	Time	Steps	Epochs	Min Loss
1	51M Base	51.1M	Full Alpaca (52k)	28 min	848	~0.02	2.85
2	51M v2	51.1M	500 short examples	10 min	~350	7	2.07
3	16M Chat	16.1M	500 short examples	10 min	460	16	1.98
4	51M v3	51.1M	10k filtered Alpaca	10 min+	in-progress	-	-

All models were initialized with random weights and trained from scratch — no pre-training or transfer learning.

Available Models on Hugging Face

Model ID	Params	Size	Description
USAGAMES365/cpu-chat-llm/models/v2-51m	51.1M	204 MB	Retrained 500 short examples, 7 epochs
USAGAMES365/cpu-chat-llm/models/v3-51m	51.1M	204 MB	Continued training on 10k examples (in progress)
USAGAMES365/cpu-chat-llm/models/gemma-270m	268M	1.0 GB	Gemma-3-270m-it for comparison (pre-trained, not ours)

Usage

from transformers import GPT2LMHeadModel, AutoTokenizer

# Load our model
model = GPT2LMHeadModel.from_pretrained("USAGAMES365/cpu-chat-llm/models/v2-51m")
tokenizer = AutoTokenizer.from_pretrained("USAGAMES365/cpu-chat-llm/models/v2-51m")

# Format prompt correctly
prompt = "<|User|> What is 2+2?</s><|Assistant|>"
inputs = tokenizer.encode(prompt, return_tensors="pt")

# Generate
outputs = model.generate(inputs, max_new_tokens=80, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

• Architecture: GPT-2 decoder-only transformer
• Config (51M): 8 layers, 8 heads, 512 hidden dim, 2048 FF dim, 192-256 context
• Config (16M): 6 layers, 4 heads, 256 hidden dim, 512 FF dim, 192 context
• Tokenizer: GPT-2 tokenizer (50,257 vocab)
• Optimizer: AdamW (lr=3e-4 to 1e-3, weight_decay=0.01)
• Batch size: 4-16
• Hardware: CPU only (Replit environment)
• Software: PyTorch, Transformers, Datasets

Run Structure

The repo organizes checkpoints from each training phase:

• previous-run/ - Initial training runs with step-level checkpoints
• current-run/ - Continued training with saved checkpoints
• models/ - Final exportable models

Limitations

• Small model size (16M-51M params vs. billions in production models)
• Very limited training time (minutes vs. thousands of GPU-hours)
• CPU-only training constrains both speed and model capacity
• Outputs may be incoherent or repetitive
• Not suitable for production deployment