---
language:
- en
license: mit
tags:
- text-generation
- pytorch
- gpt2
- causal-lm
- from-scratch
datasets:
- wikitext
metrics:
- perplexity
---

# himbot-gpt-level2

A GPT-style language model trained **from scratch** using PyTorch on WikiText-103.

- 44.7M parameters
- 6 layers, 8 heads, 512 embedding dim
- Trained for 20,000 steps
- Final perplexity: **38.3**

## Usage

```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("hranjan043/himbot-gpt-level2")
model     = GPT2LMHeadModel.from_pretrained("hranjan043/himbot-gpt-level2")

inputs = tokenizer("The history of science", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.8,
    top_k=50,
    do_sample=True,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Sample Output

> *"The history of science was not considered to be the first to be in the
> United States, but the second to be seen in the first American science
> fiction book series..."*

## Training Details

| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer (GPT-style) |
| Parameters | 44.7M |
| Layers | 6 |
| Attention heads | 8 |
| Embedding dim | 512 |
| Context length | 256 tokens |
| Vocabulary | GPT-2 BPE (50,257 tokens) |
| Training steps | 20,000 |
| Val loss | 3.644 |
| Perplexity | 38.3 |
| Dataset | WikiText-103 |
| Optimizer | AdamW (lr=3e-4) |
| Hardware | RTX 4000 Ada 20GB |

## License
MIT