---
library_name: transformers
tags:
- gpt2
- text-generation
---

# Model Card for harpertoken/harpertokenGPT2

GPT-2 small model trained from scratch on WikiText-2-raw-v1 dataset for text generation.

## Model Details

### Model Description

This is a GPT-2 small model (117M parameters) trained from random initialization on the WikiText-2-raw-v1 dataset. It can generate coherent text continuations.

- **Developed by:** Niladri Das
- **Model type:** GPT-2
- **Language(s) (NLP):** English
- **License:** Apache-2.0

### Model Sources

- **Repository:** https://github.com/bniladridas/models

## Uses

### Direct Use

Use for text generation tasks, such as completing sentences or generating stories.

### Out-of-Scope Use

Not suitable for tasks requiring factual accuracy, safety-critical applications, or languages other than English.

## Bias, Risks, and Limitations

Trained on WikiText, which may contain biases from the source data. Model may generate inappropriate or biased content.

### Recommendations

Use with caution; implement content filters for production use.

## How to Get Started with the Model

```python
from transformers import pipeline

generator = pipeline('text-generation', model='harpertoken/harpertokenGPT2')
print(generator("The quick brown fox"))
```

## Training Details

### Training Data

WikiText-2-raw-v1 dataset, a collection of Wikipedia articles.

### Training Procedure

Trained from scratch using PyTorch and Transformers.

#### Training Hyperparameters

- Epochs: 3
- Batch size: 1
- Learning rate: 5e-5
- Max length: 512

## Evaluation

Basic evaluation via text generation coherence.

### Results

Generates plausible text continuations.

## Environmental Impact

- **Hardware Type:** CPU/MPS
- **Hours used:** ~10 minutes
- **Carbon Emitted:** Minimal (local training)

## Technical Specifications

### Model Architecture and Objective

GPT-2 decoder-only transformer for causal language modeling.

### Compute Infrastructure

- Hardware: Mac with MPS
- Software: PyTorch, Transformers