↓
Token Embedding 151,936 → 1024
↓ ×28 Transformer Blocks
GQA Attention 16Q / 8KV heads · dim 128
Q proj 1024→2048 · K,V 1024→1024 · O 2048→1024
Q-Norm & K-Norm (RMSNorm per head)
RoPE · θ=1,000,000 · max 40,960 tokens
Post-Attn LayerNorm [1024]
SwiGLU FFN 1024→3072→1024
↓
↓
Language Model Head 1024 → 151,936
↓