EfficientNet-B0 Sparse Stage Attention ($P_3/P_4/P_5$) & Alternating Weight Sharing ($[B_1, A, B, A]$)

Official model weights, reference architecture builders, and SHA-256 verification manifests for the paper:

Where Global Context and Weight Sharing Pay Off in Compact Vision Backbones
Michael Schroeder (2026)


Overview

This repository provides the trained checkpoints and model-construction modules for two complementary structural modifications to EfficientNet-B0 evaluated across the full 50,000-image ImageNet-1k validation split:

  1. Sparse Zero-Gated Stage Attention ($P_3/P_4/P_5$): Instead of wrapping the backbone in a heavy multi-scale feature pyramid (BiFPN) or replacing entire convolutional stages with Transformer blocks, we preserve the entire MBConv trunk and insert lightweight Pre-LN residual self-attention adapters with zero-initialized gates at three semantic scales:

    • $P_3$ ($28 \times 28$, $40\text{ ch}$): $7 \times 7$ non-overlapping windowed multi-head self-attention (MHSA)
    • $P_4$ ($14 \times 14$, $112\text{ ch}$): global MHSA
    • $P_5$ ($7 \times 7$, $320\text{ ch}$): global MHSA
      Under a strictly controlled 88-epoch schedule consuming bit-for-bit identical stateless image batches from the same initial backbone, Sparse Stage Attention outperforms Plain-B0 at every validation milestone, finishing $+0.954$ percentage points ahead ($73.878%$ vs. $72.924%$, and $74.020%$ in standalone training). Learned gate activity increases monotonically with semantic depth ($27.2% \rightarrow 45.4% \rightarrow 66.3%$ relative update-to-input RMS from $P_3 \rightarrow P_4 \rightarrow P_5$).
  2. Alternating Recurrent Weight Sharing ($[B_1, A, B, A]$): Tying repeated inverted-residual blocks within Stage 6 to an alternating two-template cycle ($[B_1, A, B, A]$, where Block 4 reuses Block 2's convolutional/SE weights $A$ while retaining pass-specific BatchNorm statistics that add only $+1,536$ parameters, $<0.03%$ of the model) reduces total model parameters by $10.9%$ ($582,384$ weights) and recovers $+10.82$ percentage points in a single continuation epoch ($5,004$ steps) to **$70.92%$**β€”outperforming single-template tying ($[B_1, A, A, A]$, $66.10%$) by $+4.82$ percentage points.


Included Checkpoints (weights/)

All checkpoints are evaluated on the complete 50,000-image ImageNet-1k validation set (224x224 resolution, standard 88-epoch ReLU6/SGD baseline protocol).

File Architecture / Arm Params Top-1 (%) Top-5 (%) Size SHA-256 Prefix
weights/sparse_attention_p3p4p5_standalone_e88.weights.h5 Sparse Stage Attention ($P_3/P_4/P_5$) β€” Standalone 88-ep recovery (e88) 6.27M 74.020% 91.696% 48.44 MB 60fb3ef2e705
weights/sparse_attention_p3p4p5_matched_e86.weights.h5 Sparse Stage Attention ($P_3/P_4/P_5$) β€” Bit-for-bit matched pair winner (e86) 6.27M 73.878% 91.616% 48.44 MB 830d6b48c11f
weights/plain_b0_matched_control_e87.weights.h5 Plain EfficientNet-B0 β€” Bit-for-bit matched control (e87) 5.33M 72.924% 91.030% 41.13 MB f8f0e061f947
weights/stage6_alternating_abab_1ep.weights.h5 Stage-6 Alternating Weight Sharing ($[B_1, A, B, A]$) β€” 1-ep continuation 4.75M ($-10.9%$) 70.916% 89.940% 36.67 MB df14469a83e9
weights/stages4to6_single_template_1ep.weights.h5 Stages 4–6 Single-Template Sharing ($[B_1, A, A, A]$) β€” 1-ep continuation 3.86M ($-27.6%$) 66.104% 86.992% 29.84 MB c3a2d4f5100b

Repository Structure

.
β”œβ”€β”€ README.md
β”œβ”€β”€ SHA256SUMS.txt                  # Exact SHA-256 checksums for all weight files
β”œβ”€β”€ artifacts_manifest.json         # Machine-readable metadata & checksum ledger
β”œβ”€β”€ code/
β”‚   β”œβ”€β”€ models.py                   # Plain EfficientNet-B0/B1/B2 ReLU6 model builder
β”‚   β”œβ”€β”€ attention_conv_model.py     # Sparse Stage Attention (P3/P4/P5) model builder
β”‚   └── shared_weight_efficientnet.py # Recurrent Weight Sharing ([B1,A,B,A] & [B1,A,A,A]) builder
└── weights/
    β”œβ”€β”€ sparse_attention_p3p4p5_standalone_e88.weights.h5
    β”œβ”€β”€ sparse_attention_p3p4p5_matched_e86.weights.h5
    β”œβ”€β”€ plain_b0_matched_control_e87.weights.h5
    β”œβ”€β”€ stage6_alternating_abab_1ep.weights.h5
    └── stages4to6_single_template_1ep.weights.h5

Quickstart: Loading the Models in TensorFlow / Keras

from code.attention_conv_model import build_attention_conv_efficientnet
from code.models import build_efficientnet
from code.shared_weight_efficientnet import build_shared_stage6_pair_efficientnet_b0

# 1. Load Sparse Stage Attention (P3/P4/P5) β€” 74.02% Top-1 (standalone) or 73.88% (matched)
attn_model = build_attention_conv_efficientnet(variant="b0", num_classes=1000)
attn_model.load_weights("weights/sparse_attention_p3p4p5_standalone_e88.weights.h5")

# 2. Load Bit-for-Bit Matched Plain-B0 Control β€” 72.92% Top-1
ctrl_model = build_efficientnet(variant="b0", num_classes=1000)
ctrl_model.load_weights("weights/plain_b0_matched_control_e87.weights.h5")

# 3. Load Stage-6 Alternating Weight Sharing ([B1, A, B, A], -10.9% params) β€” 70.92% Top-1
abab_model = build_shared_stage6_pair_efficientnet_b0(num_classes=1000, num_stage6_blocks=4)
abab_model.load_weights("weights/stage6_alternating_abab_1ep.weights.h5")

Verify file integrity at any time with:

sha256sum -c SHA256SUMS.txt

Citation

@article{schroeder2026sparse,
  title   = {Where Global Context and Weight Sharing Pay Off in Compact Vision Backbones},
  author  = {Schroeder, Michael},
  year    = {2026}
}
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train meatloafer/efficientnet-b0-stage-attention

Evaluation results

  • Top-1 Accuracy (Standalone Sparse Stage Attention P3/P4/P5, e88) on ImageNet-1k (50,000-image validation split)
    validation set self-reported
    74.020
  • Top-1 Accuracy (Bit-for-Bit Matched Sparse Stage Attention P3/P4/P5, e86) on ImageNet-1k (50,000-image validation split)
    validation set self-reported
    73.878
  • Top-1 Accuracy (Bit-for-Bit Matched Plain-B0 Control, e87) on ImageNet-1k (50,000-image validation split)
    validation set self-reported
    72.924
  • Top-1 Accuracy (Stage-6 Alternating Weight Sharing [B1, A, B, A], 1-Ep Continuation) on ImageNet-1k (50,000-image validation split)
    validation set self-reported
    70.916