| --- |
| language: |
| - en |
| license: mit |
| tags: |
| - text-generation |
| - pytorch |
| - gpt2 |
| - causal-lm |
| - from-scratch |
| datasets: |
| - wikitext |
| metrics: |
| - perplexity |
| --- |
| |
| # himbot-gpt-level2 |
|
|
| A GPT-style language model trained **from scratch** using PyTorch on WikiText-103. |
|
|
| - 44.7M parameters |
| - 6 layers, 8 heads, 512 embedding dim |
| - Trained for 20,000 steps |
| - Final perplexity: **38.3** |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import GPT2LMHeadModel, GPT2Tokenizer |
| |
| tokenizer = GPT2Tokenizer.from_pretrained("hranjan043/himbot-gpt-level2") |
| model = GPT2LMHeadModel.from_pretrained("hranjan043/himbot-gpt-level2") |
| |
| inputs = tokenizer("The history of science", return_tensors="pt") |
| output = model.generate( |
| **inputs, |
| max_new_tokens=100, |
| temperature=0.8, |
| top_k=50, |
| do_sample=True, |
| ) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Sample Output |
|
|
| > *"The history of science was not considered to be the first to be in the |
| > United States, but the second to be seen in the first American science |
| > fiction book series..."* |
|
|
| ## Training Details |
|
|
| | Property | Value | |
| |---|---| |
| | Architecture | Decoder-only Transformer (GPT-style) | |
| | Parameters | 44.7M | |
| | Layers | 6 | |
| | Attention heads | 8 | |
| | Embedding dim | 512 | |
| | Context length | 256 tokens | |
| | Vocabulary | GPT-2 BPE (50,257 tokens) | |
| | Training steps | 20,000 | |
| | Val loss | 3.644 | |
| | Perplexity | 38.3 | |
| | Dataset | WikiText-103 | |
| | Optimizer | AdamW (lr=3e-4) | |
| | Hardware | RTX 4000 Ada 20GB | |
|
|
| ## License |
| MIT |
|
|
