---
language: en
license: cc-by-sa-4.0
tags:
  - nanogpt
  - gpt2
  - from-scratch
  - pretraining
  - wikipedia
  - project-gutenberg
  - research
pipeline_tag: text-generation
---

# WikipediaGutenberg-0.5B

A **from-scratch** GPT-2-style language model (nanoGPT), **530M params**, trained for **1 epoch** on a combination of **English Wikipedia** ([wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia), 20231101.en) and **Project Gutenberg** ([common-pile/project_gutenberg_filtered](https://huggingface.co/datasets/common-pile/project_gutenberg_filtered)) — ~9.0B tokens, ~17 tokens/parameter (near the [Chinchilla](https://arxiv.org/abs/2203.15556) compute-optimal ratio).

This is a **research model** from a study of how from-scratch pretraining behaves across capacity / data / exposure. It is **not** instruction-tuned, not safety-aligned, and not intended for production use.

## Model details
- **Architecture:** GPT-2 (learned positional embeddings, GeLU MLP, full multi-head attention). **16 layers, n_embd 1536, 24 heads (head_dim 64) — 530.30M params.**
- **Training data:** Wikipedia (4.17B tok) + Project Gutenberg (4.82B tok) = **8.99B train tokens (1 epoch)**; 1.00B val tokens.
- **Optimizer:** AdamW, LR 1.2e-3 → 1.2e-4 cosine, warmup 5000, batch 256 (65,536 tokens/iter), bf16, dropout 0.1.
- **Hardware:** 1× NVIDIA GH200, ~16.7 h.
- **Status:** training in progress. `ckpt.pt` is the **latest** checkpoint (mirrored here every ~30 min); it carries `iter_num` + optimizer state for clean resumption.

## ⚠️ Format: nanoGPT, not HF Transformers
This is a **raw nanoGPT checkpoint** (a torch `state_dict` + config), **not** loadable via `AutoModelForCausalLM.from_pretrained`. Load it with [nanoGPT](https://github.com/karpathy/nanoGPT):
```python
import torch
from model import GPTConfig, GPT
ckpt = torch.load("ckpt.pt", map_location="cuda", weights_only=False)
model = GPT(GPTConfig(**ckpt["model_args"]))
model.load_state_dict(ckpt["model"])
```

## Intended use & limitations
- **Intended:** ML research / education on pretraining dynamics and small-model knowledge elicitation.
- **Not intended:** deployment, factual QA, instruction-following. At 530M it has limited and weakly-elicitable factual knowledge (see the project's elicitation evals).

## Evaluation
The full **elicitation scoreboard** (latent completion-probe recall, multiple-choice, free-form QA, few-shot/CoT) vs the 253M baselines **will be added here on training completion**.

## Data & license
Wikipedia (CC BY-SA 4.0) + Project Gutenberg (public domain). Per Wikipedia's share-alike, model weights are released under **CC BY-SA 4.0**.

## Sources
- nanoGPT: [karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)
- Chinchilla (compute-optimal scaling): [Hoffmann et al., 2022](https://arxiv.org/abs/2203.15556)
- Datasets: [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) · [common-pile/project_gutenberg_filtered](https://huggingface.co/datasets/common-pile/project_gutenberg_filtered)