Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers

SaluNet explores replacing feature normalization with learned elementwise geometry, using two complementary components:

• SALU replaces normalization layers without computing batch or token feature statistics.
• SWALU and GALU, inspired by Swish and GELU respectively, provide SALU-based alternatives to conventional activation functions.

Paper: https://arxiv.org/abs/2606.02927
Code, configurations and evaluation scripts: https://github.com/MouradZaied/SaluNet

This model card describes several experiment-specific checkpoints. The SaluNet-T-B/16 ImageNet EMA checkpoint is hosted directly in this repository. The other checkpoint links currently point to external downloads.

Operators

For positive learned coefficients (a,b),

$$\mathrm{SALU}(x;a,b)=\frac{ax}{\sqrt{1+ab{x}^2}}\mathrm{.}\backslash operatorname\{SALU\}(x;a,b)=\backslash frac\{ax\}\{\backslash sqrt\{1+abx^2\}\}.$$

Each SALU module learns two scalars, shared across its input elements. Its geometric interpretation uses saturation amplitude (S) and transition width (W):

$$S=\sqrt{\frac{a}{b}},W=\frac{1}{\sqrt{ab}}\mathrm{.}S=\backslash sqrt\{\backslash frac\{a\}\{b\}\},\; \backslash qquad\; W=\backslash frac\{1\}\{\backslash sqrt\{ab\}\}.$$

The activation gates are

$$\mathrm{SWALU}(x)=\frac{x}{2}[1+\mathrm{SALU}(x)],\backslash operatorname\{SWALU\}(x)=\backslash frac\{x\}\{2\}\; \backslash left[1+\backslash operatorname\{SALU\}(x)\backslash right],$$

$$\mathrm{GALU}(x)=\frac{x}{2}[1+\mathrm{SALU}(\psi (x))],\psi (x)=\sqrt{\frac{2}{\pi }}(x+0.044715x^3)\mathrm{.}\backslash operatorname\{GALU\}(x)=\backslash frac\{x\}\{2\}\; \backslash left[1+\backslash operatorname\{SALU\}(\backslash psi(x))\backslash right],\; \backslash qquad\; \backslash psi(x)=\backslash sqrt\{\backslash frac\{2\}\{\backslash pi\}\}\backslash left(x+0.044715x^3\backslash right).$$

The gate coefficients are learned independently from those of the stabilization modules. Positivity policies and initialization depend on the experiment.

Architecture families

SaluNet-C: convolutional networks

BatchNorm layers are replaced by SALU. In gated configurations, ReLU activations are replaced by SWALU or GALU.

Experiments include ResNet-18 on CIFAR and ResNet-50 on ImageNet-1K. CNN gate parameters are shared across activation calls within each residual block; the stem has its own gate.

SaluNet-T: Vision Transformers

LayerNorm layers are replaced by SALU. MLP activations retain GELU or use SWALU or GALU, depending on the configuration.

The ImageNet ViT-B/16 experiment replaces LayerNorm with SALU and GELU with SWALU.

Selected results

CIFAR-100 / ResNet-18

The SALU + SWALU configuration obtains 83.02 ± 0.29% EMA Top-1 over three independent runs.

An additional deterministic run reaches 83.42% and is reported separately from that three-run aggregate.

In the physical-batch-size sweep, SaluNet-C-18 retains 76.23% Top-1 at batch size 1. The corresponding BN + ReLU result is 1.10%. These batch-size-1 results each represent one completed run and do not isolate the separate contributions of stabilization and gating.

ImageNet-1K

Model	Scheduled training epochs	Physical / effective training batch	Evaluation crop	EMA Top-1 (%)	EMA Top-5 (%)
SaluNet-C-50	150	180 / 900	224 × 224	79.59	94.59
SaluNet-C-50	Same checkpoint	180 / 900	288 × 288	80.16	94.96
SaluNet-T-B/16	300	128 / 128	224 × 224	80.12	94.79
SaluNet-T-B/16	Same checkpoint	128 / 128	288 × 288	80.28	94.74

Both ImageNet models were trained from scratch on one shared NVIDIA A40 GPU with 24 GB available to the training process. Each backbone is represented by one completed training run.

The higher-resolution evaluations reuse the selected EMA checkpoint without additional training or fine-tuning.

ImageNet preprocessing and implementation details

Training images: both models

Images are resized offline to a longest-side length of 320 pixels, preserving aspect ratio with Lanczos interpolation, and saved as JPEG at quality 95 using Pillow's default chroma subsampling. Online augmentations and 224 × 224 random crops are applied afterward.

SaluNet-C-50 evaluation

Evaluation uses the original 50,000 ImageNet-1K validation images:

• Bilinear shorter-side resize to 240, then center crop to 224 × 224.
• Bilinear shorter-side resize to 320, then center crop to 288 × 288.

The reference C-50 implementation clamps SWALU outputs to [-10, 10] during both training and evaluation.

Its sampler shuffles four copies of every training index without truncation. Each scheduled epoch processes 4N training views; 150 epochs therefore process 600N views. Five-step gradient accumulation gives an effective batch size of 900.

SaluNet-T-B/16 evaluation

The 50,000 validation images are preprocessed offline in the same way as the training images, including longest-side resize to 320 and JPEG quality 95.

Online evaluation then applies:

• Bicubic shorter-side resize to 256, then center crop to 224 × 224.
• Bicubic shorter-side resize to 320, then center crop to 288 × 288.

The learned positional embedding is interpolated bicubically for the higher evaluation resolution. SWALU is not output-clamped in this implementation.

The completed run uses an initial learning-rate setting of 0.000125 with 20 warmup epochs, followed by a resumed stage from checkpoint-150 at epoch index 151 with learning-rate setting 0.0001 and no additional warmup. The total horizon remains 300 epochs. EMA decay is 0.999.

Checkpoint downloads

Model	Dataset	Weights	Download
SaluNet-C-18 / SALU + GALU	CIFAR-10	EMA	Checkpoint
SaluNet-C-18 / SALU + SWALU	CIFAR-100	EMA, 83.42% run	Checkpoint
SaluNet-C-50 / clamped SWALU	ImageNet-1K	EMA	Checkpoint
SaluNet-T-B/16 / SALU + SWALU	ImageNet-1K	EMA	Checkpoint

Use the architecture and evaluation script associated with each checkpoint. These custom PyTorch checkpoints are not standard Transformers from_pretrained models.

Usage

Follow the installation instructions in the GitHub repository.

ImageNet implementations have separate setup and evaluation guides:

• SaluNet-C-50
• SaluNet-T-B/16

The ViT-B/16 implementation builds on the Derf codebase. Follow the pinned upstream setup described in its SaluNet guide.

Intended use and limitations

These are research models for studying normalization-free signal control, learnable activations, and sensitivity to physical batch size.

The ImageNet experiments demonstrate scaling feasibility under resource constraints. They do not establish superiority over baselines trained under matched recipes.

Evaluation preprocessing differs between the two ImageNet backbones. Their reported accuracies should not be interpreted as a controlled comparison between CNNs and Transformers.

Broader robustness, fairness, and deployment suitability have not been established.

Collaboration

Contributions are welcome, particularly:

• Large-batch ImageNet ViT-B/16 training using the original Derf recipe adapted to SALU + SWALU.
• Matched normalization and activation baselines.
• Additional independent ImageNet runs.
• Small-batch CIFAR repetitions, especially at batch sizes 4, 2 and 1.

Please report configurations, seeds, preprocessing, software versions, and individual results alongside any aggregates.

Author and acknowledgments

Mourad Zaied
RTIM Laboratory, University of Gabes, Tunisia

The author thanks Google Cloud for research credits provided through the Google Cloud Research Credits program, and the University of Gabes for its institutional support.

Reference

Mourad Zaied. Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers.

https://arxiv.org/abs/2606.02927