---
language: en
license: mit
tags:
- gpt2
- language-model
- pytorch
---

# GPT-2 Model - Trained from Scratch

This is a GPT-2 model trained from scratch on [your dataset].

## Model Details

- **Architecture**: GPT-2
- **Parameters**: 124M
- **Training Steps**: 19073
- **Validation Loss**: 4.21
- **HellaSwag Accuracy**: 25.58%

## Training Configuration

- Batch Size: 32
- Sequence Length: 1024
- Learning Rate: 6e-4
- Total Tokens: ~1B tokens

## Usage
```python
import torch

# Load checkpoint
checkpoint = torch.load('model_hf_09000.pt') #current saved checkpoint, to be updated after fully training
config = checkpoint['config']

# Initialize model
from your_model_file import GPT  # Your GPT class
model = GPT(config)
model.load_state_dict(checkpoint['model'])
model.eval()

# Generate text
# ... your generation code ...
```

## Training Details

Trained using custom GPT-2 implementation following Andrej Karpathy's approach.