Ryze-125M

A 125M parameter GPT-2 style language model trained from scratch.

Model Details

Property	Value
Architecture	GPT-2 (decoder-only transformer)
Parameters	~125M
Layers	12
Attention Heads	12
Hidden Dimension	768
Context Length	512 tokens
Vocabulary Size	50,000
Training Steps	23,000
Tokens Seen	376,832,000 (~377M)
Best Val Loss	4.3596

Training Data

• Primary: HuggingFaceFW/fineweb (sample-10BT)
• Validation: HuggingFaceFW/dclm_100BT

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Norman89107/ryze-125m")
tokenizer = AutoTokenizer.from_pretrained("Norman89107/ryze-125m")

input_text = "The future of artificial intelligence"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Configuration

• Micro Batch Size: 8
• Gradient Accumulation Steps: 4
• Effective Batch Size: 32 sequences (16,384 tokens)
• Learning Rate: 0.0003 (with ReduceLROnPlateau scheduler)
• Weight Decay: 0.1
• Max Gradient Norm: 1.0
• Dropout: 0.1
• Bias: Disabled

Custom Tokenizer

This model uses a custom BPE tokenizer trained on 100,000 documents with a vocabulary of 50,000 tokens.