Hugging Face's logo

UniversalAlgorithmic
/
SPG

TensorBoard
Safetensors
Model card Files
xet
Metrics Community
1
UniversalAlgorithmic commited on
Commit
83a2cac
·
verified ·
1 Parent(s): 97d8aaa

Upload 2 files

Browse files
Files changed (2) hide show
  1. README.md +54 -329
  2. demo_nas.ipynb +0 -0
README.md CHANGED
@@ -10,30 +10,36 @@
10
  | Model | SPG | # Params | Acc@1 (%) | Acc@5 (%) | Weights | Command to reproduce |
11
  |-------|------|----------|-----------|-----------|---------|----------------------|
12
  | MobileNet-V2 | ❌ | 3.5 M | 71.878 | 90.286 | <a href='https://download.pytorch.org/models/mobilenet_v2-b0353104.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#mobilenetv2'>Recipe</a> |
13
- | MobileNet-V2 | ✅ | 3.5 M | 72.104 | 90.316 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/image-classification/mobilenetv2/model_32.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | [examples/image-classification/run.sh](#retrain-model-on-imagenet-1k) |
 
14
  | ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 | <a href='https://download.pytorch.org/models/resnet50-0676ba61.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> |
15
- | ResNet-50 | ✅ | 25.6 M | 77.234 | 93.322 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/image-classification/resnet50/model_35.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | [examples/image-classification/run.sh](#retrain-model-on-imagenet-1k) |
 
16
  | EfficientNet-V2-M | ❌ | 54.1 M | 85.112 | 97.156 | <a href='https://download.pytorch.org/models/efficientnet_v2_m-dc08266a.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#efficientnet-v2'>Recipe</a> |
17
- | EfficientNet-V2-M | ✅ | 54.1 M | 85.218 | 97.208 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/image-classification/efficientnet_v2_m/model_7.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> | [examples/image-classification/run.sh](#retrain-model-on-imagenet-1k) |
 
18
  | ViT-B16 | ❌ | 86.6 M | 81.072 | 95.318 | <a href='https://download.pytorch.org/models/vit_b_16-c867db91.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#vit_b_16'>Recipe</a> |
19
- | ViT-B16 | ✅ | 86.6 M | 81.092 | 95.304 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/image-classification/vit_b_16/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> | [examples/image-classification/run.sh](#retrain-model-on-imagenet-1k) |
 
20
 
21
 
22
 
23
- `Table 2: Performance of pre-trained vs. SPG-retrained models. All models are evaluated a subset of COCO val2017, on the 21/20 categories that are present in the Pascal VOC dataset.`
24
-
25
- > ⚠️ `All model reported on TorchVision (with weight COCO_WITH_VOC_LABELS_V1) were benchmarked using only 20 categories. Researchers should first download the pre-trained model from TorchVision and conduct re-evaluation under the 21-categories (including "background") framework.`
26
 
27
  | Model | SPG | # Params | mIoU (%) | pixelwise Acc (%) | Weights | Command to reproduce |
28
  |---------------------|-----|----------|------------|---------------------|---------|----------------------|
29
- | FCN-ResNet50 | ❌ | 35.3 M | 58.9/60.5 | 90.9/91.4 | <a href='https://download.pytorch.org/models/fcn_resnet50_coco-1167a1af.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#fcn_resnet50'>Recipe</a> |
30
- | FCN-ResNet50 | ✅ | 35.3 M | 59.4/60.9 | 90.9/91.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/semantic-segmentation/fcn_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> | [examples/semantic-segmentation/run.sh](#retrain-model-on-ms-coco-2017) |
31
- | FCN-ResNet101 | ❌ | 54.3 M | 62.2/63.7 | 91.1/91.9 | <a href='https://download.pytorch.org/models/fcn_resnet101_coco-7ecb50ca.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
32
- | FCN-ResNet101 | ✅ | 54.3 M | 62.4/64.3 | 91.1/91.9 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/semantic-segmentation/fcn_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> | [examples/semantic-segmentation/run.sh](#retrain-model-on-ms-coco-2017) |
33
- | DeepLabV3-ResNet50 | ❌ | 42.0 M | 63.8/66.4 | 91.5/92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet50'>Recipe</a> |
34
- | DeepLabV3-ResNet50 | ✅ | 42.0 M | 64.2/66.6 | 91.6/92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/semantic-segmentation/deeplabv3_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> | [examples/semantic-segmentation/run.sh](#retrain-model-on-ms-coco-2017) |
35
- | DeepLabV3-ResNet101 | ❌ | 61.0 M | 65.3/67.4 | 91.7/92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet101_coco-586e9e4e.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
36
- | DeepLabV3-ResNet101 | ✅ | 61.0 M | 65.7/67.8 | 91.8/92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/semantic-segmentation/deeplabv3_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> | [examples/semantic-segmentation/run.sh](#retrain-model-on-ms-coco-2017) |
 
 
 
 
37
 
38
 
39
  `Table 3: Performance comparison of fine-tuned vs. SPG-retrained models across NLP and speech benchmarks.`
@@ -44,42 +50,43 @@
44
  | Task | SPG | Metric Type | Performance (%) | Weights | Command to reproduce |
45
  |-------|------|-------------------|-----------------|---------|----------------------|
46
  | CoLA | ❌ | Matthews coor | 56.53 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
47
- | CoLA | ✅ | Matthews coor | 62.13 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/text-classification/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> | [examples/text-classification/run.sh](#transfer-learning-on-glue) |
 
48
  | SST-2 | ❌ | Accuracy | 92.32 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
49
- | SST-2 | ✅ | Accuracy | 92.54 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/text-classification/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> | [examples/text-classification/run.sh](#transfer-learning-on-glue) |
 
50
  | MRPC | ❌ | F1/Accuracy | 88.85/84.09 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
51
- | MRPC | ✅ | F1/Accuracy | 91.10/87.25 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/text-classification/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> | [examples/text-classification/run.sh](#transfer-learning-on-glue) |
 
52
  | QQP | ❌ | F1/Accuracy | 87.49/90.71 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
53
- | QQP | ✅ | F1/Accuracy | 89.72/90.88 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/text-classification/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> | [examples/text-classification/run.sh](#transfer-learning-on-glue) |
 
54
  | QNLI | ❌ | Accuracy | 90.66 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
55
- | QNLI | ✅ | Accuracy | 91.10 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/text-classification/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> | [examples/text-classification/run.sh](#transfer-learning-on-glue) |
 
56
  | RTE | ❌ | Accuracy | 65.70 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
57
- | RTE | ✅ | Accuracy | 72.56 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/text-classification/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> | [examples/text-classification/run.sh](#transfer-learning-on-glue) |
 
58
  | Q/A* | ❌ | F1/Extra match | 88.52/81.22 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-question_answering-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering#fine-tuning-bert-on-squad10'>Recipe</a> |
59
- | Q/A* | ✅ | F1/Extra match | 88.67/81.51 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/question-answering/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> | [examples/question-answering/run.sh](#transfer-learning-on-squad) |
 
60
  | AC† | ❌ | Accuracy | 98.26 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-audio_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification#single-gpu'>Recipe</a> |
61
- | AC† | ✅ | Accuracy | 98.31 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> | [examples/audio-answering/run.sh](#transfer-learning-on-superb) |
62
-
63
 
64
- ## Model Zoo: Neural Architecture Search (NAS) via SPG Algorithm
65
 
66
- `Table 4: Performance of pre-trained vs. SPG-retrained models on ImageNet-1K`
67
- Depending on the base model, we explore the following architectures:
68
- - ResNet-18: ResNet-18, ResNet-27, ResNet-36, ResNet-45
69
- - ResNet-34: ResNet-34, ResNet-40, ResNet-46, ResNet-52
70
- - ResNet-50: ResNet-50, ResNet-53, ResNet-56, ResNet-59
71
 
72
- > ⚠️`Our SPG differs from most NAS algorithms, which typically use a gating network for architecture selection. In contrast, we neither employ a gating network nor a proxy network. Instead, after policy optimization, we keep only the base architecture (ResNet-18, ResNet-34, and ResNet-50) and remove all others (ResNet-27/36/45, ResNet-40/46/52, and ResNet-53/56/59).`
73
 
 
 
 
 
 
 
 
 
74
 
75
- | Model | SPG | # Params | Acc@1 (%) | Acc@5 (%) | Weights | Command to reproduce |
76
- |-------|------|----------|-----------|-----------|---------|----------------------|
77
- | ResNet-18 | ❌ | 11.7M | 69.758 | 89.078 | <a href='https://download.pytorch.org/models/resnet18-f37072fd.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> |
78
- | ResNet-18 | ✅ | 11.7M | 70.092 | 89.314 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/neural-archicture-search/resnet18/model_3.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet18-yellow'></a> | [examples/neural-architecture-search/run.sh](#neural-architecture-search-for-resnet-on-imagenet-1k) |
79
- | ResNet-34 | ❌ | 21.8M | 73.314 | 91.420 | <a href='https://download.pytorch.org/models/resnet34-b627a593.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> |
80
- | ResNet-34 | ✅ | 21.8M | 73.900 | 93.536 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/neural-archicture-search/resnet34/model_8.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet34-yellow'></a> | [examples/neural-architecture-search/run.sh](#neural-architecture-search-for-resnet-on-imagenet-1k) |
81
- | ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 | <a href='https://download.pytorch.org/models/resnet50-0676ba61.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> |
82
- | ResNet-50 | ✅ | 25.6 M | 77.234 | 93.322 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/neural-archicture-search/resnet50/model_9.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | [examples/neural-architecture-search/run.sh](#neural-architecture-search-for-resnet-on-imagenet-1k) |
83
 
84
 
85
  ## Requirements
@@ -104,7 +111,7 @@ Depending on the base model, we explore the following architectures:
104
  n01443537:
105
  ILSVRC2012_val_00000236.JPEG ...
106
  ```
107
- 4. Prepare the [MS-COCO 2017](https://cocodataset.org/#home) dataset manually and place it in `/path/to/coco`. For image classification examples, pass the argument `--data-path=/path/to/coco` to the training script. The extracted dataset directory should follow this structure:
108
  ```setup
109
  /path/to/coco/:
110
  annotations:
@@ -114,292 +121,10 @@ Depending on the base model, we explore the following architectures:
114
  val2017:
115
  000000000139.jpg ...
116
  ```
117
- 5. For [🗣️ Keyword Spotting subset](https://huggingface.co/datasets/s3prl/superb#ks), [Common Language](https://huggingface.co/datasets/speechbrain/common_language), [SQuAD](https://huggingface.co/datasets/rajpurkar/squad), [Common Voice](https://huggingface.co/datasets/legacy-datasets/common_voice), [GLUE](https://gluebenchmark.com/) and [WMT](https://huggingface.co/datasets/wmt/wmt17) datasets, manual downloading is not required — they will be automatically loaded via the Hugging Face Datasets library when running our `audio-classification`, `question-answering`, `speech-recognition`, `text-classification`, or `translation` examples.
118
-
119
- ## Training
120
-
121
- ### Retrain model on ImageNet-1K
122
- We use training recipes similar to those in [PyTorch Vision's classification reference](https://github.com/pytorch/vision/blob/main/references/classification/README.md) to retrain MobileNet-V2, ResNet, EfficientNet-V2, and ViT with our SPG on ImageNet-1K. The following command can be used:
123
-
124
- ```bash
125
- cd ./examples/image-classification
126
-
127
- # MobileNet-V2
128
- torchrun --nproc_per_node=4 train.py\
129
- --data-path /path/to/imagenet/\
130
- --model mobilenet_v2 --output-dir mobilenet_v2 --weights MobileNet_V2_Weights.IMAGENET1K_V1\
131
- --batch-size 192 --epochs 40 --lr 0.0004 --lr-step-size 10 --lr-gamma 0.5 --wd 0.00004 --apply-trp --trp-depths 1 --trp-p 0.15 --trp-lambdas 0.4 0.2 0.1
132
-
133
- # ResNet-50
134
- torchrun --nproc_per_node=4 train.py\
135
- --data-path /path/to/imagenet/\
136
- --model resnet50 --output-dir resnet50 --weights ResNet50_Weights.IMAGENET1K_V1\
137
- --batch-size 64 --epochs 40 --lr 0.0004 --lr-step-size 10 --lr-gamma 0.5 --print-freq 100\
138
- --apply-trp --trp-depths 1 --trp-p 0.2 --trp-lambdas 0.4 0.2 0.1
139
-
140
- # EfficientNet-V2 M
141
- torchrun --nproc_per_node=4 train.py \
142
- --data-path /path/to/imagenet/\
143
- --model efficientnet_v2_m --output-dir efficientnet_v2_m --weights EfficientNet_V2_M_Weights.IMAGENET1K_V1\
144
- --epochs 10 --batch-size 64 --lr 5e-9 --lr-scheduler cosineannealinglr --weight-decay 0.00002 \
145
- --lr-warmup-method constant --lr-warmup-epochs 8 --lr-warmup-decay 0. \
146
- --auto-augment ta_wide --random-erase 0.1 --label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --norm-weight-decay 0.0 \
147
- --train-crop-size 384 --val-crop-size 480 --val-resize-size 480 --ra-sampler --ra-reps 4 --print-freq 100\
148
- --apply-trp --trp-depths 1 --trp-p 0.2 --trp-lambdas 0.4 0.2 0.1
149
-
150
- # ViT-B-16
151
- torchrun --nproc_per_node=4 train.py\
152
- --data-path /path/to/imagenet/\
153
- --model vit_b_16 --output-dir vit_b_16 --weights ViT_B_16_Weights.IMAGENET1K_V1\
154
- --epochs 5 --batch-size 196 --opt adamw --lr 5e-9 --lr-scheduler cosineannealinglr --wd 0.3\
155
- --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \
156
- --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra --clip-grad-norm 1 --cutmix-alpha 1.0\
157
- --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 --print-freq 100
158
- ```
159
-
160
- ### Retrain model on MS-COCO 2017
161
- We use training recipes similar to those in [PyTorch Vision's segmentation reference](https://github.com/pytorch/vision/blob/main/references/segmentation/README.md) to retrain FCN and DeepLab-V3 with our SPG on COCO dataset. The following command can be used:
162
-
163
- ```bash
164
-
165
- cd ./examples/semantic-segmentation
166
-
167
- # FCN-ResNet50
168
- torchrun --nproc_per_node=4 train.py\
169
- --workers 4 --dataset coco --data-path /path/to/coco/\
170
- --model fcn_resnet50 --aux-loss --output-dir fcn_resnet50 --weights FCN_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\
171
- --epochs 5 --batch-size 16 --lr 0.0002 --aux-loss --print-freq 100\
172
- --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \
173
- --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
174
-
175
- # FCN-ResNet101
176
- torchrun --nproc_per_node=4 train.py\
177
- --workers 4 --dataset coco --data-path /path/to/coco/\
178
- --model fcn_resnet101 --aux-loss --output-dir fcn_resnet101 --weights FCN_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\
179
- --epochs 5 --batch-size 12 --lr 0.0002 --aux-loss --print-freq 100\
180
- --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \
181
- --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
182
-
183
- # DeepLabV3-ResNet50
184
- torchrun --nproc_per_node=4 train.py\
185
- --workers 4 --dataset coco --data-path /path/to/coco/\
186
- --model deeplabv3_resnet50 --aux-loss --output-dir deeplabv3_resnet50 --weights DeepLabV3_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\
187
- --epochs 5 --batch-size 16 --lr 0.0002 --aux-loss --print-freq 100\
188
- --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \
189
- --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
190
-
191
- # DeepLabV3-ResNet101
192
- torchrun --nproc_per_node=4 train.py\
193
- --workers 4 --dataset coco --data-path /path/to/coco/\
194
- --model deeplabv3_resnet101 --aux-loss --output-dir deeplabv3_resnet101 --weights DeepLabV3_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\
195
- --epochs 5 --batch-size 12 --lr 0.0002 --aux-loss --print-freq 100\
196
- --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \
197
- --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
198
- ```
199
- </details>
200
-
201
- ### Transfer learning on GLUE
202
- We use recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain BERT with our SPG on GLUE benchmark. The following command can be used:
203
-
204
- ```bash
205
- cd ./examples/text-classification && bash run.sh
206
- ```
207
-
208
- ### Transfer learning on SQuAD
209
- We use recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain Wav2Vec with our SPG on SQuAD dataset. The following command can be used:
210
-
211
- ```bash
212
- cd ./examples/audio-classification
213
- CUDA_VISIBLE_DEVICES=0 python run_audio_classification.py \
214
- --model_name_or_path facebook/wav2vec2-base \
215
- --dataset_name superb \
216
- --dataset_config_name ks \
217
- --trust_remote_code \
218
- --output_dir wav2vec2-base-ft-keyword-spotting \
219
- --overwrite_output_dir \
220
- --remove_unused_columns False \
221
- --do_train \
222
- --do_eval \
223
- --fp16 \
224
- --learning_rate 3e-5 \
225
- --max_length_seconds 1 \
226
- --attention_mask False \
227
- --warmup_ratio 0.1 \
228
- --num_train_epochs 8 \
229
- --per_device_train_batch_size 64 \
230
- --gradient_accumulation_steps 4 \
231
- --per_device_eval_batch_size 32 \
232
- --dataloader_num_workers 4 \
233
- --logging_strategy steps \
234
- --logging_steps 10 \
235
- --eval_strategy epoch \
236
- --save_strategy epoch \
237
- --load_best_model_at_end True \
238
- --metric_for_best_model accuracy \
239
- --save_total_limit 3 \
240
- --seed 0 \
241
- --push_to_hub \
242
- --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
243
- ```
244
-
245
-
246
- ### Transfer learning on SUPERB
247
- We use recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain BERT with our SPG on SUPERB benchmark. The following command can be used:
248
-
249
- ```bash
250
- cd ./examples/question-answering
251
- CUDA_VISIBLE_DEVICES=0 python run_qa.py \
252
- --model_name_or_path google-bert/bert-base-uncased \
253
- --dataset_name squad \
254
- --do_train \
255
- --do_eval \
256
- --per_device_train_batch_size 12 \
257
- --learning_rate 3e-5 \
258
- --num_train_epochs 2 \
259
- --max_seq_length 384 \
260
- --doc_stride 128 \
261
- --output_dir ./baseline \
262
- --overwrite_output_dir \
263
- --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1
264
- ```
265
-
266
- ### Neural Architecture Search for ResNet on ImageNet-1K
267
- We conduct Neural Architecture Search (NAS) for the ResNet architecture on the ImageNet dataset. The following command can be used:
268
-
269
- ```bash
270
- cd ./examples/neural-architecture-search
271
-
272
- # During Neural Architecture Search (NAS), we explore ResNet-18, ResNet-27, ResNet-36, and ResNet-45. After retraining with SPG algorithm, we retain only ResNet-18 and discard the others.
273
- torchrun --nproc_per_node=4 train.py\
274
- --data-path /home/cs/Documents/datasets/imagenet\
275
- --model resnet18 --output-dir resnet18 --weights ResNet18_Weights.IMAGENET1K_V1\
276
- --batch-size 128 --epochs 10 --lr 0.0004 --lr-step-size 2 --lr-gamma 0.5\
277
- --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\
278
- --apply-trp --trp-depths 3 3 3 --trp-planes 256 --trp-lambdas 0.4 0.2 0.1 --print-freq 100
279
-
280
- # During Neural Architecture Search (NAS), we explore ResNet-34, ResNet-40, ResNet-46, and ResNet-52. After retraining with SPG algorithm, we retain only ResNet-34 and discard the others.
281
- torchrun --nproc_per_node=4 train.py\
282
- --data-path /home/cs/Documents/datasets/imagenet\
283
- --model resnet34 --output-dir resnet34 --weights ResNet34_Weights.IMAGENET1K_V1\
284
- --batch-size 96 --epochs 10 --lr 0.0004 --lr-step-size 2 --lr-gamma 0.5\
285
- --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\
286
- --apply-trp --trp-depths 2 2 2 --trp-planes 256 --trp-lambdas 0.4 0.2 0.1 --print-freq 100
287
-
288
- # During Neural Architecture Search (NAS), we explore ResNet-50, ResNet-53, ResNet-56, and ResNet-59. After retraining with SPG algorithm, we retain only ResNet-50 and discard the others.
289
- torchrun --nproc_per_node=4 train.py\
290
- --data-path /home/cs/Documents/datasets/imagenet\
291
- --model resnet50 --output-dir resnet50 --weights ResNet50_Weights.IMAGENET1K_V1\
292
- --batch-size 64 --epochs 10 --lr 0.0004 --lr-step-size 2 --lr-gamma 0.5\
293
- --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\
294
- --apply-trp --trp-depths 1 1 1 --trp-planes 1024 --trp-lambdas 0.4 0.2 0.1 --print-freq 100
295
- ```
296
-
297
- ## Evaluation
298
-
299
- To evaluate our models on ImageNet, run:
300
-
301
- ```bash
302
-
303
- cd examples/image-classification
304
-
305
- # Required: Download our MobileNet-V2 weights to examples/image-classification/mobilenet_v2
306
- torchrun --nproc_per_node=4 train.py\
307
- --data-path /path/to/imagenet/\
308
- --model mobilenet_v2 --resume mobilenet_v2/model_32.pth --test-only
309
-
310
- # Required: Download our ResNet-50 weights to examples/image-classification/resnet50
311
- torchrun --nproc_per_node=4 train.py\
312
- --data-path /path/to/imagenet/\
313
- --model resnet50 --resume resnet50/model_35.pth --test-only
314
-
315
- # Required: Download our EfficientNet-V2 M weights to examples/image-classification/efficientnet_v2_m
316
- torchrun --nproc_per_node=4 train.py\
317
- --data-path /path/to/imagenet/\
318
- --model efficientnet_v2_m --resume efficientnet_v2_m/model_7.pth --test-only\
319
- --val-crop-size 480 --val-resize-size 480
320
-
321
- # Required: Download our ViT-B-16 weights to examples/image-classification/vit_b_16
322
- torchrun --nproc_per_node=4 train.py\
323
- --data-path /path/to/imagenet/\
324
- --model vit_b_16 --resume vit_b_16/model_4.pth --test-only
325
- ```
326
-
327
- To evaluate our models on COCO, run:
328
-
329
- ```bash
330
-
331
- cd examples/semantic-segmentation
332
-
333
- # eval baselines
334
- torchrun --nproc_per_node=1 train.py\
335
- --workers 4 --dataset coco --data-path /path/to/coco/\
336
- --model fcn_resnet50 --aux-loss --weights FCN_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\
337
- --test-only
338
- torchrun --nproc_per_node=1 train.py\
339
- --workers 4 --dataset coco --data-path /path/to/coco/\
340
- --model fcn_resnet101 --aux-loss --weights FCN_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\
341
- --test-only
342
- torchrun --nproc_per_node=1 train.py\
343
- --workers 4 --dataset coco --data-path /path/to/coco/\
344
- --model deeplabv3_resnet50 --aux-loss --weights DeepLabV3_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\
345
- --test-only
346
- torchrun --nproc_per_node=1 train.py\
347
- --workers 4 --dataset coco --data-path /path/to/coco/\
348
- --model deeplabv3_resnet101 --aux-loss --weights DeepLabV3_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\
349
- --test-only
350
-
351
-
352
- # eval our models
353
- # Required: Download our FCN-ResNet50 weights to examples/semantic-segmentation/fcn_resnet50
354
- torchrun --nproc_per_node=1 train.py\
355
- --workers 4 --dataset coco --data-path /path/to/coco/\
356
- --model fcn_resnet50 --aux-loss --resume fcn_resnet50/model_4.pth\
357
- --test-only
358
-
359
- # Required: Download our FCN-ResNet101 weights to examples/semantic-segmentation/fcn_resnet101
360
- torchrun --nproc_per_node=1 train.py\
361
- --workers 4 --dataset coco --data-path /path/to/coco/\
362
- --model fcn_resnet101 --aux-loss --resume fcn_resnet101/model_4.pth\
363
- --test-only
364
-
365
- # Required: Download our DeepLabV3-ResNet50 weights to examples/semantic-segmentation/deeplabv3_resnet50
366
- torchrun --nproc_per_node=1 train.py\
367
- --workers 4 --dataset coco --data-path /path/to/coco/\
368
- --model deeplabv3_resnet50 --aux-loss --resume deeplabv3_resnet50/model_4.pth\
369
- --test-only
370
-
371
- # Required: Download our DeepLabV3-ResNet101 weights to examples/semantic-segmentation/deeplabv3_resnet101
372
- torchrun --nproc_per_node=1 train.py\
373
- --workers 4 --dataset coco --data-path /path/to/coco/\
374
- --model deeplabv3_resnet101 --aux-loss --resume deeplabv3_resnet101/model_4.pth\
375
- --test-only
376
- ```
377
-
378
- To evaluate our models on GLUE, SquAD, and SUPERB, please re-run the `transfer learning` related commands we previously declared, as these commands are used not only for training but also for evaluation.
379
-
380
-
381
- For Network Architecture Search, please run the following command to evaluate our SPG-trained ResNet models:
382
- ```bash
383
-
384
- cd ./examples/neural-architecture-search
385
-
386
- # Required: Download our ResNet-18 weights to examples/neural-architecture-search/resnet18
387
- torchrun --nproc_per_node=4 train.py\
388
- --data-path /home/cs/Documents/datasets/imagenet\
389
- --model resnet18 --resume resnet18/model_3.pth --test-only
390
-
391
- # Required: Download our ResNet-34 weights to examples/neural-architecture-search/resnet34
392
- torchrun --nproc_per_node=4 train.py\
393
- --data-path /home/cs/Documents/datasets/imagenet\
394
- --model resnet34 --resume resnet34/model_8.pth --test-only
395
-
396
- # Required: Download our ResNet-50 weights to examples/neural-architecture-search/resnet50
397
- torchrun --nproc_per_node=4 train.py\
398
- --data-path /home/cs/Documents/datasets/imagenet\
399
- --model resnet50 --resume resnet50/model_9.pth --test-only
400
- ```
401
-
402
-
403
- ## License
404
- This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
405
-
 
10
  | Model | SPG | # Params | Acc@1 (%) | Acc@5 (%) | Weights | Command to reproduce |
11
  |-------|------|----------|-----------|-----------|---------|----------------------|
12
  | MobileNet-V2 | ❌ | 3.5 M | 71.878 | 90.286 | <a href='https://download.pytorch.org/models/mobilenet_v2-b0353104.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#mobilenetv2'>Recipe</a> |
13
+ | MobileNet-V2 | ✅HPO | 3.5 M | 72.104 | 90.316 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/mobilenetv2/model_32.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
14
+ | MobileNet-V2 | ✅NAS | 3.5 M | 72.208 | 90.822 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/mobilenetv2/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
15
  | ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 | <a href='https://download.pytorch.org/models/resnet50-0676ba61.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> |
16
+ | ResNet-50 | ✅HPO | 25.6 M | 77.234 | 93.322 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/resnet50/model_35.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
17
+ | ResNet-50 | ✅NAS | 25.6 M | 80.970 | 95.481 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> |
18
  | EfficientNet-V2-M | ❌ | 54.1 M | 85.112 | 97.156 | <a href='https://download.pytorch.org/models/efficientnet_v2_m-dc08266a.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#efficientnet-v2'>Recipe</a> |
19
+ | EfficientNet-V2-M | ✅HPO | 54.1 M | 85.218 | 97.208 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/efficientnet_v2_m/model_7.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
20
+ | EfficientNet-V2-M | ✅NAS | 54.1 M | 85.347 | 97.424 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/efficientnet_v2_m/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> |
21
  | ViT-B16 | ❌ | 86.6 M | 81.072 | 95.318 | <a href='https://download.pytorch.org/models/vit_b_16-c867db91.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#vit_b_16'>Recipe</a> |
22
+ | ViT-B16 | ✅HPO | 86.6 M | 81.092 | 95.304 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/vit_b_16/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
23
+ | ViT-B16 | ✅NAS | 86.6 M | 81.114 | 95.320 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/vit_b_16/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> |
24
 
25
 
26
 
27
+ `Table 2: Performance of pre-trained vs. SPG-retrained models. All models are evaluated a subset of COCO val2017, on the 21 categories that are present in the Pascal VOC dataset.`
 
 
28
 
29
  | Model | SPG | # Params | mIoU (%) | pixelwise Acc (%) | Weights | Command to reproduce |
30
  |---------------------|-----|----------|------------|---------------------|---------|----------------------|
31
+ | FCN-ResNet50 | ❌ | 35.3 M | 60.5 | 91.4 | <a href='https://download.pytorch.org/models/fcn_resnet50_coco-1167a1af.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#fcn_resnet50'>Recipe</a> |
32
+ | FCN-ResNet50 | ✅HPO | 35.3 M | 60.9 | 91.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/semantic-segmentation/fcn_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
33
+ | FCN-ResNet50 | ✅NAS | 35.3 M | 61.2 | 91.7 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/fcn_resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |
34
+ | FCN-ResNet101 | ❌ | 54.3 M | 63.7 | 91.9 | <a href='https://download.pytorch.org/models/fcn_resnet101_coco-7ecb50ca.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
35
+ | FCN-ResNet101 | ✅HPO | 54.3 M | 64.3 | 91.9 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/fcn_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
36
+ | FCN-ResNet101 | ✅NAS | 54.3 M | 64.6 | 92.0 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/fcn_resnet101/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |
37
+ | DeepLabV3-ResNet50 | ❌ | 42.0 M | 66.4 | 92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet50'>Recipe</a> |
38
+ | DeepLabV3-ResNet50 | ✅HPO | 42.0 M | 66.6 | 92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/deeplabv3_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
39
+ | DeepLabV3-ResNet50 | ✅NAS | 42.0 M | 66.8 | 92.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/deeplabv3_resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |
40
+ | DeepLabV3-ResNet101 | ❌ | 61.0 M | 67.4 | 92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet101_coco-586e9e4e.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
41
+ | DeepLabV3-ResNet101 | ✅HPO | 61.0 M | 67.8 | 92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/deeplabv3_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
42
+ | DeepLabV3-ResNet101 | ✅NAS | 61.0 M | 68.1 | 92.8 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/deeplabv3_resnet101/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |
43
 
44
 
45
  `Table 3: Performance comparison of fine-tuned vs. SPG-retrained models across NLP and speech benchmarks.`
 
50
  | Task | SPG | Metric Type | Performance (%) | Weights | Command to reproduce |
51
  |-------|------|-------------------|-----------------|---------|----------------------|
52
  | CoLA | ❌ | Matthews coor | 56.53 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
53
+ | CoLA | ✅HPO | Matthews coor | 62.13 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
54
+ | CoLA | ✅NAS | Matthews coor | 63.02 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
55
  | SST-2 | ❌ | Accuracy | 92.32 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
56
+ | SST-2 | ✅HPO | Accuracy | 92.54 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
57
+ | SST-2 | ✅NAS | Accuracy | 92.75 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
58
  | MRPC | ❌ | F1/Accuracy | 88.85/84.09 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
59
+ | MRPC | ✅HPO | F1/Accuracy | 91.10/87.25 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
60
+ | MRPC | ✅NAS | F1/Accuracy | 91.32/87.65 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
61
  | QQP | ❌ | F1/Accuracy | 87.49/90.71 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
62
+ | QQP | ✅HPO | F1/Accuracy | 89.72/90.88 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
63
+ | QQP | ✅NAS | F1/Accuracy | 89.88/91.03 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
64
  | QNLI | ❌ | Accuracy | 90.66 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
65
+ | QNLI | ✅HPO | Accuracy | 91.10 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
66
+ | QNLI | ✅NAS | Accuracy | 91.27 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
67
  | RTE | ❌ | Accuracy | 65.70 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
68
+ | RTE | ✅HPO | Accuracy | 72.56 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
69
+ | RTE | ✅NAS | Accuracy | 73.13 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
70
  | Q/A* | ❌ | F1/Extra match | 88.52/81.22 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-question_answering-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering#fine-tuning-bert-on-squad10'>Recipe</a> |
71
+ | Q/A* | ✅HPO | F1/Extra match | 88.67/81.51 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/question-answering/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/question-answering/run.sh'>run.sh</a> |
72
+ | Q/A* | ✅NAS | F1/Extra match | 88.79/81.68 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/question-answering/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/question-answering/run.sh'>run.sh</a> |
73
  | AC† | ❌ | Accuracy | 98.26 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-audio_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification#single-gpu'>Recipe</a> |
74
+ | AC† | ✅HPO | Accuracy | 98.31 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/audio-classification/run.sh'>run.sh</a> |
75
+ | AC† | ✅NAS | Accuracy | 98.37 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/audio-classification/run.sh'>run.sh</a> |
76
 
 
77
 
78
+ `Table 4: Performance of SFT vs. SPG-retrained models on GSM8K`
 
 
 
 
79
 
 
80
 
81
+ | Model | SPG | score | Weights | Command to reproduce |
82
+ |-------|------|-------|---------|----------------------|
83
+ | Gemma-2-2B-it | ❌ | 49.66 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Gemma2B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
84
+ | Gemma-2-2B-it | ✅ | 52.31 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Gemma2B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
85
+ | Qwen-2.5-0.5B-Instruct | ❌ | 39.12 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Qwen0.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
86
+ | Qwen-2.5-0.5B-Instruct | ✅ | 41.70 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Qwen0.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
87
+ | Qwen-2.5-1.5B-Instruct | ❌ | 58.68 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Qwen1.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
88
+ | Qwen-2.5-1.5B-Instruct | ✅ | 59.12 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Qwen1.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
89
 
 
 
 
 
 
 
 
 
90
 
91
 
92
  ## Requirements
 
111
  n01443537:
112
  ILSVRC2012_val_00000236.JPEG ...
113
  ```
114
+ 4. Prepare the [MS-COCO 2017](https://cocodataset.org/#home) dataset manually and place it in `/path/to/coco`. For semantic segmentation examples, pass the argument `--data-path=/path/to/coco` to the training script. The extracted dataset directory should follow this structure:
115
  ```setup
116
  /path/to/coco/:
117
  annotations:
 
121
  val2017:
122
  000000000139.jpg ...
123
  ```
124
+ 5. Prepare the [GSM8K](https://huggingface.co/datasets/openai/gsm8k) dataset manually and place it in `/path/to/gsm8k`. For language modeling examples, pass the argument `--data-path=/path/to/gsm8k` to the training script. The extracted dataset directory should follow this structure:
125
+ ```setup
126
+ /path/to/gsm8k/:
127
+ train.parquet
128
+ test.parquet
129
+ ```
130
+ 6. For [🗣️ Keyword Spotting subset](https://huggingface.co/datasets/s3prl/superb#ks), [Common Language](https://huggingface.co/datasets/speechbrain/common_language), [SQuAD](https://huggingface.co/datasets/rajpurkar/squad), [Common Voice](https://huggingface.co/datasets/legacy-datasets/common_voice), [GLUE](https://gluebenchmark.com/) and [WMT](https://huggingface.co/datasets/wmt/wmt17) datasets, manual downloading is not required — they will be automatically loaded via the Hugging Face Datasets library when running our `audio-classification`, `question-answering`, `speech-recognition`, `text-classification`, or `translation` examples.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
demo_nas.ipynb CHANGED
The diff for this file is too large to render. See raw diff