Prince-111M

Prince is an attention-free language model. No attention of any kind.

Built by The Entropy Space AI.

Architecture

x → RMSNorm → [EMA ‖ DeltaRule] → concat → Linear(2D→D) → Focus → + x
  → RMSNorm → FFN (SwiGLU) → + x

Component	Role
EMA	Local smoothing (learnable per-channel decay, O(d) params)
DeltaRule	Associative memory — token retrieval via fast weights
Focus	Mamba selective SSM — temporal flow, long-range context
SwiGLU FFN	Channel mixing

Properties

• Training: O(nd·log n) parallel associative scan
• Inference: O(d) per token — fixed state, no KV cache
• Hardware agnostic: pure JAX/Flax NNX

Config

Params	105.9M
Layers	8
d_model	512
d_state	64
Vocab	151665
Tokenizer	Qwen2.5

Training

Dataset: FineWeb-Edu 1B tokens | TPU v5e-8 | 50K steps | AdamW lr=3e-4