license: apache-2.0
base_model: google/gemma-4-E2B
tags:
- flashnorm
- gemma4
- weightless-rmsnorm
gemma-4-E2B-FlashNorm
The FlashNorm variant of google/gemma-4-E2B: the folded norm weight tensors are removed from the state dict entirely, and the model runs WITHOUT the pre-attention RMSNorm in every decoder layer (Proposition 3 of the FlashNorm paper, applicable because Gemma 4 re-normalizes queries, keys, and values per head). The pre-feedforward norm keeps its division but is weightless. Requires trust_remote_code:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('open-machine/gemma-4-E2B-FlashNorm', trust_remote_code=True)
For stock-Transformers loading without remote code, use the compatibility companion gemma-4-E2B-FlashNorm-test (same fold, norm tensors kept as all-ones), whose card also lists the full norm-tensor inventory.
Validation: outputs are bitwise identical (bf16) to the compatibility checkpoint with runtime cancellation applied, verified on the uploaded copy over a 1024-token context and under cached greedy decoding; deviations from the base model are at the bf16 rounding level. In fp32 on this architecture the cancellation measures max logit deviation 6.8e-4 (vs 5.6e-4 for the mathematically exact fold-only control), WikiText-2 perplexity change 0.0006%, greedy generations identical, HellaSwag unchanged. Post-attention, post-feedforward, per-layer-input, q/k, and final norms are unchanged (not foldable, or not yet folded).
Gemma is a trademark of Google; distributed under the same Apache 2.0 license as the base model.