PyM Studio GPT-1 Project Gutenberg
A 116.5M-parameter decoder-only Transformer created from scratch by Kamron M (Hugging Face: kmehrad, Kaggle: kammehr) at PyM Studio.
The architecture follows the 117M-parameter model from Improving Language Understanding by Generative Pre-Training. It was pretrained with a causal language-model objective on the filtered Project Gutenberg subset of the Common Pile, using the original 40,478-token OpenAI GPT tokenizer.
The implementation, training pipeline, and evaluation notebook are available in the llm-recipes GitHub repository.
Architecture
- 12 decoder blocks with post-residual LayerNorm
- 768-dimensional token and learned position embeddings
- 12 causal self-attention heads
- 3,072-dimensional GELU feed-forward hidden layer
- 512-token context length
- tied token-embedding and language-model-head weights
- 0.1 embedding, attention, residual, and feed-forward dropout
- approximately 116.5M unique trainable parameters
The included modeling_gpt1.py contains the handwritten PyTorch model definitions used for training.
Pretraining
| Property | Value |
|---|---|
| Dataset | Common Pile filtered Project Gutenberg |
| Training documents | 55,178 |
| Available packed training tokens | 4,558,447,104 |
| Tokens consumed by this run | 3,276,800,000 |
| Next-token targets | 3,270,400,000 |
| Context length | 512 |
| Global batch | 64 sequences |
| Optimizer updates | 100,000 |
| Peak learning rate | 2.5e-4 |
| Warmup | 2,000 updates |
| Schedule | Linear warmup and cosine decay |
| Optimizer | AdamW |
| Weight decay | 0.01 on matrix weights |
| Gradient clipping | 1.0 |
| Precision | BF16 |
| Training GPU | NVIDIA GeForce RTX 3090 Ti |
| Training time | approximately 32.2 hours |
Training sequences were deterministically shuffled with seed 42. The run consumed approximately 71.9% of the packed training corpus and therefore did not repeat an epoch. The held-out Project Gutenberg validation loss at update 100,000 was 3.2356, corresponding to perplexity 25.42.
The filtered corpus contains public-domain and openly licensed books, but its dataset card notes that licensing metadata may not be perfect. Review the source dataset card for provenance and limitations.
Comparison with published GPT-1 weights
Both models were evaluated on the same packed WikiText-103 published test split with the OpenAI GPT tokenizer, 512-token blocks, shifted next-token loss, and BF16 inference.
| Model | Test loss | Perplexity |
|---|---|---|
| PyM Studio GPT-1 Project Gutenberg | 4.2182 | 67.91 |
| Published OpenAI GPT-1 | 4.3636 | 78.54 |
The test contains 544 blocks and 277,984 predicted tokens. Under this exact protocol, this checkpoint has 13.54% lower perplexity than the published OpenAI GPT-1 weights. This is an out-of-domain language-modeling comparison, not evidence that the model is universally better: the pretraining corpora differ, and downstream transfer must be measured separately.
Complete evaluation code and qualitative next-token comparisons are in 2_evaluate_gpt.ipynb.
Files
- model.pt: inference-only PyTorch state dictionary, configuration, and metadata
- modeling_gpt1.py: handwritten GPT-1 architecture
- generate.py: sampling and interactive-generation CLI
- tokenizer.json and tokenizer_config.json: OpenAI GPT tokenizer assets
- training_metadata.json: architecture and pretraining configuration
- evaluation_results.json: controlled WikiText-103 comparison
- requirements.txt: minimal runtime dependencies
- LICENSE: Apache License 2.0
Usage
Install the minimal dependencies:
pip install -r requirements.txt
Generate a continuation:
python generate.py "Once upon a time, there was"
Use deterministic greedy decoding:
python generate.py "The scientist discovered that" --greedy
Start an interactive prompt loop:
python generate.py
Use --device cpu or --device cuda to choose a device. Sampling defaults to temperature 0.8, top-k 40, and top-p 0.95.
Intended use and limitations
This is a base causal language model intended for educational study, language-model experiments, representation transfer, and supervised fine-tuning. It is not instruction-tuned or safety-tuned.
- English only
- maximum context length of 512 tokens
- historical book-domain language and biases from Project Gutenberg
- possible memorization, factual errors, harmful text, and repetitive output
- no chat template or end-of-document token
- not evaluated for production or high-stakes use
Attribution and license
Copyright 2026 Kamron M (Kaggle: kammehr), PyM Studio.
Licensed under the Apache License, Version 2.0. See LICENSE.