---
language:
- en
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- finance
- sec-edgar
- gpt2
---

# SEC-EDGAR GPT-2 124M

A GPT-2 (124M) model trained from scratch on SEC-EDGAR filings (10-K, 10-Q, 8-K, etc.) using nanoGPT.

## Model Details

| Parameter | Value |
|-----------|-------|
| Architecture | GPT-2 (GPT2LMHeadModel) |
| Parameters | ~124M |
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 |
| Context length | 1024 |
| Vocab size | 50,257 |
| Precision | float32 |

## Training

- **Framework**: nanoGPT (Karpathy's)
- **Dataset**: SEC-EDGAR filings (financial disclosures, annual/quarterly reports)
- **Tokenizer**: GPT-2 BPE (tiktoken)

## Usage

```python
from transformers import GPT2LMHeadModel, GPT2Tokenizer

model = GPT2LMHeadModel.from_pretrained("lzwjava/sec-edgar-gpt-124m-hf")
tokenizer = GPT2Tokenizer.from_pretrained("lzwjava/sec-edgar-gpt-124m-hf")

prompt = "The company reported total revenue of"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.8, top_k=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Intended Use

This model is trained for research and educational purposes — demonstrating nanoGPT training on domain-specific financial text. It is **not** suitable for production financial analysis or advice.

## Limitations

- Trained on a subset of SEC filings; may not generalize to all financial domains
- No RLHF or instruction tuning — raw language model only
- Generated text may contain factual inaccuracies