1B GPT-2 — Code & Math Language Model (Trained From Scratch)
A 1.19B-parameter causal language model trained entirely from scratch (random initialization, zero pretrained weights) on a weighted mix of code (70%) and mathematical text (30%). Built on the GPT-2 architecture (MIT license, OpenAI). Final training loss 3.057, best eval loss 3.571 at step 2000.
Why This Model
- Truly from scratch — no pretrained weights, no inherited bias; fully your own model.
- Code + Math focus — trained on clean code corpora and OpenWebMath, so it understands both programming syntax and mathematical reasoning.
- Small and fast — ~1.19B params, hundreds of tokens/sec on a modern GPU; ~3 GB VRAM for inference.
- Production-safe licensing — MIT: free to use, modify, and monetize.
Model Details
| Attribute | Value |
|---|---|
| Architecture | GPT-2 (causal LM, MIT) |
| Parameters | 1.19B |
| Layers / Hidden / Heads | 36 / 1600 / 25 |
| Context length | 1024 tokens |
| Vocab size | 50,257 (GPT-2 BPE) |
| Training steps | 2,000 (from scratch) |
| Total tokens | ~65.5M (32,768 tokens/step) |
| Batch | 4 per device × 8 gradient accumulation = 32 × 1024 seq |
| Optimizer | AdamW · lr 3e-4 cosine · warmup 500 · wd 0.01 |
| Precision | bf16 + gradient checkpointing |
| Final train loss | 3.057 |
| Best eval loss | 3.571 (step 2000) |
| Hardware | NVIDIA Blackwell B6000 (96 GB) · molab |
| License | MIT — free for commercial use |
Training Data — Full Dataset List
| Dataset | Content | Share | Streams | Used |
|---|---|---|---|---|
codeparrot/codeparrot-clean-subset |
Deduplicated clean code (Python, Java, JS, C++, Go…) | 70% | Steps 0–1800 | Yes |
open-web-math/open-web-math |
Mathematical text from the web | 30% | Steps 0–2000 | Yes |
- Streamed with 10,000-sample shuffle buffer, seed 42.
- Tokenized on the fly (GPT-2 BPE, truncation to 1024), into a 50M-token RAM buffer.
- Chunked into 1024-token blocks; the last 500 blocks held out for evaluation (never seen during training).
- Note: the Codeparrot code stream became unreachable on the HF Hub partway through the project, so the final 200 steps (1800→2000) continued on OpenWebMath alone.
Training Loss Curve
Eval loss by step
| Step | Eval loss |
|---|---|
| 250 | 5.797 |
| 500 | 5.163 |
| 750 | 4.674 |
| 1000 | 4.257 |
| 1250 | 3.868 |
| 1500 | 3.678 |
| 1750 | 3.582 |
| 2000 | 3.571 |
Quickstart
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "pinkelephantlimited/1b-gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_id, subfolder="final")
model = AutoModelForCausalLM.from_pretrained(
model_id, subfolder="final", torch_dtype=torch.bfloat16
).cuda().eval()
prompt = "def is_prime(n):\n "
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(
**inputs, max_new_tokens=80, do_sample=True, temperature=0.7,
top_k=50, pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
Tip:
from_pretrained(..., device_map="auto")works too — only ~3 GB VRAM for inference.
Capabilities
- Code completion and function bodies (Python and other C-family languages)
- Mathematical and reasoning-style text generation
- General English text generation
Limitations
- Text only — cannot process images, PDFs, or Word documents directly; extract text first.
- Small model — not competitive with frontier LLMs on complex reasoning.
- Not instruction-tuned — use a prompting style, or fine-tune for chat.
- 1024-token context window.
- Trained on only ~65.5M tokens — a research-scale dataset; fine-tuning is recommended for production quality.
Reproducibility
- Seed 42 (data shuffle + training RNG)
- All training checkpoints (steps 200–2000) preserved in
checkpoints/ - Full log history in each checkpoint's
trainer_state.json - Final weights, tokenizer, and config in
final/
Base: GPT-2 (MIT) · Trained: molab (Blackwell B6000) · License: MIT