CodeDevX/5m-text-generator

5M Text Generator

A small 5M parameter text generation model trained on educational/scientific text data.

Model Details

• Parameters: 4,983,808 (4.98M)
• Architecture: Transformer Decoder with GQA, RoPE, SwiGLU
• Training: 1000 steps on 5000 text samples
• Vocabulary: 8000 tokens (BPE)

Usage

from transformers import PreTrainedTokenizerFast
import torch
import sys
sys.path.insert(0, ".")  # if needed

from model import TextDecoder

# Load model
model = TextDecoder.from_pretrained("CodeDevX/5m-text-generator")
tokenizer = PreTrainedTokenizerFast.from_pretrained("CodeDevX/5m-text-generator")

# Generate text
input_ids = tokenizer("The quick brown fox", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, temperature=0.8)
print(tokenizer.decode(output[0]))

Architecture

• Hidden size: 256
• Layers: 6
• Attention heads: 8 (KV heads: 2)
• Intermediate size: 768
• Max sequence length: 512

Training Results

Step	Loss
100	12.68
200	2.17
500	0.15
1000	0.13

License

MIT