metadata
language:
- en
license: mit
tags:
- text-generation
- pytorch
- gpt2
- causal-lm
- from-scratch
datasets:
- wikitext
metrics:
- perplexity
himbot-gpt-level2
A GPT-style language model trained from scratch using PyTorch on WikiText-103.
- 44.7M parameters
- 6 layers, 8 heads, 512 embedding dim
- Trained for 20,000 steps
- Final perplexity: 38.3
Usage
from transformers import GPT2LMHeadModel, GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("hranjan043/himbot-gpt-level2")
model = GPT2LMHeadModel.from_pretrained("hranjan043/himbot-gpt-level2")
inputs = tokenizer("The history of science", return_tensors="pt")
output = model.generate(
**inputs,
max_new_tokens=100,
temperature=0.8,
top_k=50,
do_sample=True,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Sample Output
"The history of science was not considered to be the first to be in the United States, but the second to be seen in the first American science fiction book series..."
Training Details
| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer (GPT-style) |
| Parameters | 44.7M |
| Layers | 6 |
| Attention heads | 8 |
| Embedding dim | 512 |
| Context length | 256 tokens |
| Vocabulary | GPT-2 BPE (50,257 tokens) |
| Training steps | 20,000 |
| Val loss | 3.644 |
| Perplexity | 38.3 |
| Dataset | WikiText-103 |
| Optimizer | AdamW (lr=3e-4) |
| Hardware | RTX 4000 Ada 20GB |
License
MIT