File size: 4,932 Bytes
9a79f18 0356650 f28d51f 9a79f18 0356650 9a79f18 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 | ---
language: en
tags:
- attnres
- image-classification
- imagenet100
- resnet-50
datasets:
- imagenet-1m
metrics:
- accuracy
- macro-f1
- macro-auc
---
# AttnRes-50: 2D Attention Residuals for ResNet-50
## Model Description
Try to extends the original 1D AttnRes (https://arxiv.org/abs/2603.15031) to 2D spatial feature maps for computer vision.
**However it seems not work on Classification :(**
| Mode | Description |
|:---|:---|
| **none** | Standard ResNet-50 (baseline)
| **stage** | Intra-stage AttnRes — each block attends to prior block outputs within the same stage |
| **global** | Cross-stage AttnRes — each block attends to all prior block outputs across stages, with spatial/channel alignment via interpolation and 1×1 conv projections |
## Dataset: ImageNet-100
ImageNet-100 is a 100-class subset of ImageNet-1K (ILSVRC2012). This study uses the split defined by [Tian et al. (2020)](https://github.com/HobbitLong/CMC).
| Split | Samples |
|:---|---:|
| Total | 126,689 |
| Training (80%) | 101,351 |
| Validation (20%) | 25,338 |
The train/val split is deterministic (`torch.manual_seed(42)` with `randperm`).
## Training Details
### Hyperparameters
| Parameter | Value |
|:---|---|
| Optimizer | SGD with momentum (µ=0.9) |
| Learning rate | 0.1 (cosine annealing to 0 over 90 epochs) |
| Weight decay | 1×10⁻⁴ |
| Batch size | 32 |
| Epochs | 90 |
| Input size | 224×224 |
| GPU | NVIDIA A100 (80GB) — single GPU |
### Data Augmentation
- **Training**: RandomResizedCrop(224), RandomHorizontalFlip, ToTensor, Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225])
- **Validation**: Resize(256), CenterCrop(224), ToTensor, Normalize (same stats)
## Results
| Mode | Acc@1 | Acc@5 | Val Loss | Macro F1 | Macro AUC | Params |
|:---|---:|:---:|:---:|:---:|:---:|---:|
| **none** (Baseline) | **86.86%** | **96.83%** | **0.5034** | **86.65%** | **99.75%** | 23.77M |
| stage (Intra-Stage) | 85.80% | 96.62% | 0.5268 | 85.59% | 99.74% | 23.77M |
| global (Cross-Stage) | 85.45% | 96.57% | 0.5380 | 85.23% | 99.72% | 28.36M |
- Both attention variants **underperform the standard ResNet-50 baseline** on ImageNet-100 (~1–1.4% lower Acc@1).
## Usage
```python
import torch
from attnres_resnet_demo import AttnResNet50
# Load baseline (standard ResNet-50)
model = AttnResNet50(mode='none', num_classes=100)
ckpt = torch.load('result/best_none.pth', map_location='cpu')
model.load_state_dict(ckpt['model_state_dict'])
model.eval()
# Inference
from PIL import Image
import torchvision.transforms as T
transform = T.Compose([
T.Resize(256),
T.CenterCrop(224),
T.ToTensor(),
T.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]),
])
img = Image.open('example.jpg')
x = transform(img).unsqueeze(0)
with torch.no_grad():
logits = model(x)
pred = logits.argmax(dim=1)
```
## Checkpoints
| Mode | File | Size |
|:---|---:|---:|
| none | `best_none.pth` | 182 MB |
| stage | `best_stage.pth` | 182 MB |
| global | `best_global.pth` | 200 MB |
Each checkpoint contains:
```python
{
'epoch': int,
'mode': str, # 'none' | 'stage' | 'global'
'model_state_dict': OrderedDict,
'optimizer_state_dict': OrderedDict,
'best_acc1': float,
'num_classes': int, # 100
}
```
> **Note for global mode**: The model uses a lazy-initialized `ModuleDict` for cross-stage channel projections. When loading the checkpoint, run a dummy forward pass first to populate the projection layers before calling `load_state_dict`.
## Limitations
- **Small-scale study**: ImageNet-100 is only 100 classes; conclusions may not transfer to full ImageNet-1K.
- **No dual mode results**: The dual-path variant was not trained due to negative findings from stage and global.
- **Single training run per mode**: No repeated runs with different seeds to measure variance.
- **Standard training recipe**: Uses a basic SGD + cosine annealing schedule without modern tricks (warmup, label smoothing, mixup, EMA).
## Citation
If you use this code or findings, please cite the original AttnRes work:
```bibtex
@misc{chen2026attnres,
title = {Attention Residuals},
author = {Kimi Team and Chen, Guangyu and Zhang, Yu and Su, Jianlin and Xu, Weixin and Pan, Siyuan and Wang, Yaoyu and Wang, Yucheng and Chen, Guanduo and Yin, Bohong and Chen, Yutian and Yan, Junjie and Wei, Ming and Zhang, Y. and Meng, Fanqing and Hong, Chao and Xie, Xiaotong and Liu, Shaowei and Lu, Enzhe and Tai, Yunpeng and Chen, Yanru and Men, Xin and Guo, Haiqing and Charles, Y. and Lu, Haoyu and Sui, Lin and Zhu, Jinguo and Zhou, Zaida and He, Weiran and Huang, Weixiao and Xu, Xinran and Wang, Yuzhi and Lai, Guokun and Du, Yulun and Wu, Yuxin and Yang, Zhilin and Zhou, Xinyu},
year = {2026},
archiveprefix = {arXiv},
eprint = {2603.15031},
primaryclass = {cs.CL}
}
```
|