| --- |
| license: apache-2.0 |
| base_model: google/gemma-4-E2B |
| tags: |
| - flashnorm |
| - gemma4 |
| - weightless-rmsnorm |
| --- |
| |
| # gemma-4-E2B-FlashNorm-test |
|
|
| The stock-loading compatibility companion of [gemma-4-E2B-FlashNorm](https://huggingface.co/open-machine/gemma-4-E2B-FlashNorm) (naming follows the FlashNorm convention: the plain -FlashNorm repo is the strict artifact, -test is the companion that loads everywhere). The gains of `input_layernorm` are folded into the q/k/v projections (q only on KV-shared layers) and the gains of `pre_feedforward_layernorm` into the gate/up projections, in float64, with those norm weights kept as all-ones tensors so the checkpoint loads in stock HuggingFace Transformers with no custom code and computes the same function as the base model: |
|
|
| ```python |
| from transformers import AutoModelForCausalLM |
| model = AutoModelForCausalLM.from_pretrained('open-machine/gemma-4-E2B-FlashNorm-test') |
| ``` |
|
|
| ## Which norm tensors remain, and why |
|
|
| | Norm (per layer) | State here | Why | |
| |---|---|---| |
| | `input_layernorm` | all-ones (folded into q/k/v) | removed entirely in the strict repo | |
| | `pre_feedforward_layernorm` | all-ones (folded into gate/up) | weightless in the strict repo | |
| | `q_norm`, `k_norm` | unchanged | per-head gains, foldable into the q/k projections (possible follow-up); the per-head division itself stays | |
| | `post_attention_layernorm`, `post_feedforward_layernorm`, `post_per_layer_input_norm` | unchanged | feed residual additions, not foldable | |
| | final `model.norm` | unchanged | `lm_head` is tied to the embeddings, folding would alter the embedding table | |
| | `v_norm` | no weight tensor | scale-free in Gemma 4 | |
|
|
| ## Validation |
|
|
| Float64 microcheck of the fold identity is exact to ~1e-14. At bf16, logit deviations from the base model are at the bf16 rounding level, consistent with the fp32 fold-only measurements for this architecture (max logit deviation 5.6e-4 at fp32). WikiText-2 (wikitext-2-raw-v1, test) perplexity changes are below 0.001%. Runtime cancellation of the pre-attention norm (Proposition 3) applies directly to this checkpoint via [`flashNorm_cancel.py`](https://github.com/OpenMachine-ai/transformer-tricks/blob/main/flashNorm_cancel.py); the strict repo ships that cancellation built into its modeling code. |
|
|
| Gemma is a trademark of Google; distributed under the same Apache 2.0 license as the base model. |
|
|
