# 5M Text Generator

A small 5M parameter text generation model trained on educational/scientific text data.

## Model Details

- **Parameters**: 4,983,808 (4.98M)
- **Architecture**: Transformer Decoder with GQA, RoPE, SwiGLU
- **Training**: 1000 steps on 5000 text samples
- **Vocabulary**: 8000 tokens (BPE)

## Usage

```python
from transformers import PreTrainedTokenizerFast
import torch
import sys
sys.path.insert(0, ".")  # if needed

from model import TextDecoder

# Load model
model = TextDecoder.from_pretrained("CodeDevX/5m-text-generator")
tokenizer = PreTrainedTokenizerFast.from_pretrained("CodeDevX/5m-text-generator")

# Generate text
input_ids = tokenizer("The quick brown fox", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, temperature=0.8)
print(tokenizer.decode(output[0]))
```

## Architecture

- Hidden size: 256
- Layers: 6
- Attention heads: 8 (KV heads: 2)
- Intermediate size: 768
- Max sequence length: 512

## Training Results

| Step | Loss |
|------|------|
| 100 | 12.68 |
| 200 | 2.17 |
| 500 | 0.15 |
| 1000 | 0.13 |

## License

MIT