File size: 1,107 Bytes
9b28709 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | # 5M Text Generator
A small 5M parameter text generation model trained on educational/scientific text data.
## Model Details
- **Parameters**: 4,983,808 (4.98M)
- **Architecture**: Transformer Decoder with GQA, RoPE, SwiGLU
- **Training**: 1000 steps on 5000 text samples
- **Vocabulary**: 8000 tokens (BPE)
## Usage
```python
from transformers import PreTrainedTokenizerFast
import torch
import sys
sys.path.insert(0, ".") # if needed
from model import TextDecoder
# Load model
model = TextDecoder.from_pretrained("CodeDevX/5m-text-generator")
tokenizer = PreTrainedTokenizerFast.from_pretrained("CodeDevX/5m-text-generator")
# Generate text
input_ids = tokenizer("The quick brown fox", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, temperature=0.8)
print(tokenizer.decode(output[0]))
```
## Architecture
- Hidden size: 256
- Layers: 6
- Attention heads: 8 (KV heads: 2)
- Intermediate size: 768
- Max sequence length: 512
## Training Results
| Step | Loss |
|------|------|
| 100 | 12.68 |
| 200 | 2.17 |
| 500 | 0.15 |
| 1000 | 0.13 |
## License
MIT
|