Petal-2-50M

Small language model (49.7M parameters), Qwen3 architecture, trained from scratch.

Architecture

Property	Value
Layers	14
Hidden size	512
Intermediate size	1408
Attention heads	8 (GQA kv=4, head_dim=64)
QK-Norm	True
Sliding window	0/14 layers (disabled)
Max sequence length	1024
Vocab size	16384
Tied embeddings	True
Total parameters	49.693M

Training

• Tokens seen: 15,577,353,990
• Val loss: 1.9078
• Val PPL: 6.74

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("CyanMonkey/Petal-2-50M")
model = AutoModelForCausalLM.from_pretrained("CyanMonkey/Petal-2-50M")
inputs = tokenizer("Hello", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))