src/kernels/normalization/README.md

Normalization Kernels

RMSNorm (Root Mean Square Layer Normalization) — a simplified alternative to LayerNorm that skips mean-centering and only rescales by the root-mean-square of activations. Cheaper to compute and performs comparably for transformer pre-norm architectures.

Two weight conventions exist across model families:

Convention	Formula	Models
Direct	norm(x) * weight	Qwen3, LLaMA, Mistral
Unit offset	norm(x) * (1 + weight)	Gemma, Gemma3

The unit-offset convention initialises weights to zero so the initial scale is 1.0 (identity). Our implementation supports both via the add_unit_offset parameter.

Computation is always done in float32 regardless of input dtype to avoid numerical instability in half-precision, then cast back.

References

Paper	Link
Root Mean Square Layer Normalization (Zhang & Sennrich, 2019)	https://arxiv.org/abs/1910.07467
Layer Normalization (Ba et al., 2016) — the predecessor	https://arxiv.org/abs/1607.06450