--- pipeline_tag: image-classification library_name: fgir-zoo tags: - fine-grained - ultra-fine-grained - image-classification - vision-transformer - parameter-efficient - resnet - adapter - petl --- # FGIR-ViT Vision-transformer checkpoints for **ultra-fine-grained image recognition (UFGIR)** — telling apart cultivars *within* a species (soybean, cotton and other leaf sets) where classes differ by tiny visual margins and samples per class are scarce. Almost every checkpoint is an ImageNet-21k **ViT-B/16 with the backbone frozen**; only a small set of adapter modules is trained, so a model is a ~340 MB file of which ~2 M parameters were actually learned. The `ila_saw_resnet_448` group applies SAW to a frozen timm ResNet-50 instead (~96 MB files). The repository gathers the checkpoints behind two lines of work from the ERIS Lab: - **ILA** — the down-sampling inter-layer adapter, and **SAW**, a semantic-adaptation warmup added on top of it (the `ila_*` groups). ILA was introduced in the ECCV 2024 EFM workshop paper below. - **AAA** — the attention-aggregation adapter, which replaces ILA's convolutional down-sampler with a bottleneck attention that fixes the adapter's misplaced attention (the `aaa_*` groups). The full fine-tuning FGIR benchmark (FGIRFT) is now a separate repository, [ERISLab/FGIRFT](https://huggingface.co/ERISLab/FGIRFT). 847 checkpoints, one seed per configuration, last epoch only. Everything is described in `manifest.csv` and loadable by name with the standalone `fgir_zoo` library (no training repo needed). Part of the [ERISLab FGIR-ViT collection](https://huggingface.co/collections/ERISLab/fgir-vit-ila-saw-and-aaa-6ab2dc3a730388cfe7a5f7b4). ## Training recipe Backbone `vit_b16` (ImageNet-21k), frozen. The trainable part is the adapter family named in the filename (a plain PETL adapter, ILA's down-sampling adapter, or the AAA attention-aggregation adapter); the SAW runs add a supervised-contrastive warmup stage before the classifier is trained. Images are ImageNet-normalized and evaluated at 224, 448 or 672 px. Each embedded `config` (an argparse namespace saved in the checkpoint) records the exact settings. ## What each folder holds Files are grouped by experiment line and named `___[_].pth`; the trailing number is the run's wandb serial, and the name ends in it because only the last epoch is kept (so "last" is implied). `` is a resolution tag (`_is224/448/672`) where one serial spans several resolutions. | Group | Line / paper | Backbone | What it is | |---|---|---|---| | `ila_main_224` / `ila_main_448` | ILA/SAW | vit_b16 | Main UFGIR comparison of ILA against PETL and FGIR baselines, at 224 / 448 | | `ila_ablations` | ILA/SAW | vit_b16 | ILA design ablations | | `ila_saw`, `ila_saw_deit`, `ila_saw_no_ila` | ILA/SAW | vit_b16 / pedeit | The SAW contrastive-warmup runs | | `ila_swin`, `ila_dinov2`, `ila_foundation` | ILA/SAW | peswin / pevit | ILA on other backbones and frozen foundation baselines | | `ila_equal_compute` | ILA/SAW | vit_b16 | Equal-compute control | | `deit_grid_448` | ILA/SAW | pedeit / pedeit3 | DeiT and DeiT III at 448: frozen, adapter and ILA++ arms on the ten UFGIR sets | | `dinov2_adaptformer_224` | ILA/SAW | pevit (DINOv2 ViT-B/14) | DINOv2 at 224: frozen head (3021), AdaptFormer (3002), AdaptFormer + ILA (3031) | | `saw_birds_448` | ILA/SAW | vit_b16 | NABirds adapter baseline at 448 (3112) | | `saw_resnet_warmup` | ILA/SAW | peresnet50 | The two ResNet-50 SAW warmups on the pooled leaves (291 adapter, 294 ILA) | | `ila_saw_resnet_448` | ILA/SAW | peresnet50 | SAW on a frozen ResNet-50 at 448: the SAW arms, their no-SAW and linear-probe baselines, and the two SAW warmups | | `aaa_main_224` / `aaa_main_448` / `aaa_main_672` | AAA | vit_b16 | The AAA finals (frozen / adapter / ILA++ / AAA arms), by resolution | | `aaa_ablations` | AAA | vit_b16 | AAA ablations on SoyGlobal and SoyLocal: insertion blocks (424/428/430, 224 px) and width (411/413/415, 448 px) | | `aaa_fungi` | AAA (breadth) | vit_b16 | FungiTastic breadth probe — kept for completeness, not a paper result | | `medical_idrid`, `medical_petl` | AAA (medical) | vit_b16 | Medical-imaging PETL and IDRiD resolution runs | The ILA/SAW paper evaluates the `ila_main`, `ila_saw` and `ila_saw_resnet_448` serials; the AAA paper evaluates the `aaa_main_*` serials. `manifest.csv` maps every file to its dataset, method, serial, resolution, `num_classes`, top-1 accuracy and sha256. ## SAW on ResNet-50 (`ila_saw_resnet_448`) The CNN counterpart of the SAW ladder, for the SAW paper in IEEE Transactions on AgriFood Electronics. The backbone is timm's `resnet50`, frozen, at 448 px (`peresnet50`); the adapter is a 1x1 bottleneck `ConvAdapter` on every ResNet block's output, and the ILA variant adds the down-sampling adapter before the last stage. The ten datasets are cotton, soyageing, soyageingr1/r3/r4/r5/r6, soygene, soyglobal and soylocal. | Serial | Role | `method` in `manifest.csv` | Files | |---|---|---|---| | 292 | Method: adapter + SAW | `cls_adapter_saw_fz` | 10 | | 295 | Method: adapter + ILA + SAW | `ila_dso_cls_adapter_saw_fz` | 10 | | 290 | Baseline: adapter | `cls_adapter_fz` | 10 | | 293 | Baseline: adapter + ILA | `ila_dso_cls_adapter_fz` | 10 | | 661 | Baseline: FZ, frozen linear probe | `fz` | 10 | | 291 | SAW stage-1 warmup for 292 | `cls_adapter_supcon_fz` | 1 | | 294 | SAW stage-1 warmup for 295 | `ila_dso_cls_adapter_supcon_fz` | 1 | - **292** trains the adapter and classifier on each dataset, starting from the 291 warmup. - **295** does the same with the ILA down-sampler, starting from the 294 warmup. - **290** trains the same adapter from a fresh initialisation, without SAW. - **293** trains adapter + ILA from a fresh initialisation, without SAW. - **661** (FZ) keeps the whole backbone frozen and trains only a linear head. All ten datasets are released; wandb `nycu_pcs/AAA` holds all 30 runs. - **291** and **294** are the supervised-contrastive warmups on the leaves set (100 epochs). Their head is a projection, so the forward returns a 2048-d embedding and the stored `accuracy` (0.03 and 0.00) is not a classification score. The filenames of SAW and no-SAW runs differ only in the serial (`_292` against `_290`), so the manifest's `method` marks SAW with `_saw_`, read from the run's warmup `ckpt_path`. Each file's `accuracy` is its own single seed (seed 3 for the fine-tunes, seed 2 for soygene 661, seed 1 for the warmups). The papers' three-seed means are in `ILA_SAW_Analysis/results_all/acc/resnet_ladder.csv`; over the ten datasets they average 56.05 (290), 64.64 (292), 54.30 (293) and 65.01 (295), and 30.73 for FZ. A real forward on one SoyLocal test image with the 295 checkpoint: ```python import torch, torchvision.transforms as T from PIL import Image import fgir_zoo model = fgir_zoo.load('fgirvit/ila_saw_resnet_448/soylocal_peresnet50_ila_dso_cls_adapter_fz_295') tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(), # config.test_resize_size = 600 T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]) x = tf(Image.open('soylocal/soybean200_square/2_6_test.png').convert('RGB')).unsqueeze(0) with torch.no_grad(): probs = model(x).softmax(-1) # -> (1, 200) pred = probs.argmax(-1).item() # pred = 1, confidence 0.905 — the true class is 1, so this sample is correct. ``` On the first 12 SoyLocal test images this checkpoint is right 3 times; its stored test accuracy on the full split is 49.8%. ## Load a checkpoint Standalone, no `fgir_vit` install — the model code is vendored inside `fgir_zoo`: ```bash pip install git+https://github.com/arkel23/fgir-zoo.git ``` ```python import fgir_zoo model = fgir_zoo.load('fgirvit/aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403') model.eval() # fgir_zoo.list_models(family='fgirvit') lists every name; model.config holds the run's settings. ``` Or fetch the file directly and load it yourself: ```python import torch, torchvision.transforms as T from PIL import Image from huggingface_hub import hf_hub_download path = hf_hub_download('ERISLab/FGIR-ViT', 'aaa_main_448/soylocal_vit_b16_aaa_dso_cls_adapter_fz_403.pth') ckpt = torch.load(path, map_location='cpu', weights_only=False) # dict: config, model, accuracy, epoch ``` A real forward on one SoyLocal test image, with the model loaded above (ImageNet-normalized, 448 px): ```python tf = T.Compose([T.Resize(600), T.CenterCrop(448), T.ToTensor(), # config.test_resize_size = 600 T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]) x = tf(Image.open('soylocal/soybean200_square/1_2_test.png').convert('RGB')).unsqueeze(0) logits = model(x) # -> (1, 200) pred = logits.softmax(-1).argmax(-1).item() # pred = 0, confidence 0.318 — the true class is 0, so this sample is correct. ``` On the first 12 SoyLocal test images this checkpoint is right 6 times, matching its 55.5% test accuracy on the full split. ## Results (each file's own score, not the paper's aggregate) SoyLocal (200 cultivars) top-1, the released single-seed checkpoints, by arm and resolution: | Arm | 224 | 448 | 672 | |---|---|---|---| | Frozen | 28.0 | 45.3 | 27.5 | | Adapter | 33.0 | 55.7 | 36.3 | | ILA++ | 54.7 | 55.2 | 54.0 | | AAA | 45.7 | 55.5 | 56.7 | These are individual checkpoints on one dataset and one seed; they are noisier than the papers' means over ten datasets and three seeds. The papers report the aggregate (ILA with SAW averages 73.5% at 448 across the ten UFGIR sets). Every per-file score is the `accuracy` column of `manifest.csv`. ## Superseded and non-paper runs Kept for completeness but not printed in any paper: `aaa_fungi` (on FungiTastic the plain adapter beats AAA, so it does not carry the method) and the medical breadth runs. Learning-rate searches, extra seeds, best-epoch copies and exploratory ablations were removed during consolidation and are not in this repository. ## Requirements - `torch>=2.5`, `timm==0.9.12` (the checkpoints were trained on it and the vendored model code forks that version — newer timm changes the ViT internals), `huggingface_hub`, `safetensors`, `einops`, `ml_collections`. - `peresnet50` needs `fgir_zoo` with the FGIR-ViT model code at commit `aabb221` or later. - The 15 `ila_saw` SAW contrastive-head checkpoints carry an older projection-head layout; their manifest `loads` is `False` and `fgir_zoo` refuses them rather than loading loosely. ## Citation The published ILA method: ```bibtex @inproceedings{rios2024ila, title = {Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient Ultra-Fine-Grained Image Recognition}, author = {Rios, Edwin Arkel and Oyerinde, Femiloye and Hu, Min-Chun and Lai, Bo-Cheng}, booktitle = {Efficient Deep Learning for Foundation Models (EFM) Workshop at the European Conference on Computer Vision (ECCV)}, year = {2024} } ``` The SAW extension and the AAA (attention-aggregation adapter) paper are under submission; this card will be updated with their citations when they are available.