# SPG: Sequential Policy Gradient for Adaptive Hyperparameter Optimization ## Model zoo We provide baseline models and SPG-trained models, all available for download at the following links: `Table 1: Model comparison on the ImageNet-1K dataset.` | Model | SPG | # Params | Acc@1 (%) | Acc@5 (%) | Weights | |-------|------|----------|-----------|-----------|---------| | MobileNet-V2 | ❌ | 3.5 M | 71.878 | 90.286 |

| | MobileNet-V2 | ✅ | 3.5 M | 72.104 | 90.316 |

| | ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 |

| | ResNet-50 | ✅ | 25.6 M | 77.234 | 93.322 |

| | EfficientNet-V2-M | ❌ | 54.1 M | 85.112 | 97.156 |

| | EfficientNet-V2-M | ✅ | 54.1 M | 85.218 | 97.208 |

| | ViT-B16 | ❌ | 86.6 M | 81.072 | 95.318 |

| | ViT-B16 | ✅ | 86.6 M | 81.092 | 95.304 |

| `Table 2: All models are evaluated a subset of COCO val2017, on the 21 categories (including "background") that are present in the Pascal VOC dataset.` `All model reported on TorchVision (with weight COCO_WITH_VOC_LABELS_V1) were benchmarked using only 20 categories. Researchers should first download the pre-trained model from TorchVision and conduct re-evaluation under the 21-category framework.` | Model | SPG | # Params | mIoU (%) | pixelwise Acc (%) | Weights | |---------------------|-----|----------|------------|---------------------|---------| | FCN-ResNet50 | ❌ | 35.3 M | 58.9 | 90.9 |

| | FCN-ResNet50 | ✅ | 35.3 M | 59.4 | 90.9 |

| | FCN-ResNet101 | ❌ | 54.3 M | 62.2 | 91.1 |

| | FCN-ResNet101 | ✅ | 54.3 M | 62.4 | 91.1 |

| | DeepLabV3-ResNet50 | ❌ | 42.0 M | 63.8 | 91.5 |

| | DeepLabV3-ResNet50 | ✅ | 42.0 M | 64.2 | 91.6 |

| | DeepLabV3-ResNet101 | ❌ | 61.0 M | 65.3 | 91.7 |

| | DeepLabV3-ResNet101 | ✅ | 61.0 M | 65.7 | 91.8 |

| `Table 3: Performance of models for transfer learning trained with fine-tuning (FT) vs. SPG.` | Task | SPG | Metric Type | Performance (%) | Weights | |-------|------|------------------|-----------------|---------| | CoLA | ❌ | Matthews coor | 56.53 |

| | CoLA | ✅ | Matthews coor | 62.13 |

| | SST-2 | ❌ | Accuracy | 92.32 |

| | SST-2 | ✅ | Accuracy | 92.54 |

| | MRPC | ❌ | F1/Accuracy | 88.85/84.09 |

| | MRPC | ✅ | F1/Accuracy | 91.10/87.25 |

| | QQP | ❌ | F1/Accuracy | 87.49/90.71 |

| | QQP | ✅ | F1/Accuracy | 89.72/90.88 |

| | QNLI | ❌ | Accuracy | 90.66 |

| | QNLI | ✅ | Accuracy | 91.10 |

| | RTE | ❌ | Accuracy | 65.70 |

| | RTE | ✅ | Accuracy | 72.56 |

| | Q/A* | ❌ | F1/Extra match | 88.52/81.22 |

| | Q/A* | ✅ | F1/Extra match | 88.67/81.51 |

| | AC† | ❌ | Accuracy | 98.26 |

| | AC† | ✅ | Accuracy | 98.31 |

| ## Requirements 1. Install `torch>=2.0.0+cu118`. 2. To install other pip packages: ```setup pip install -r requirements.txt ``` 3. Prepare the [ImageNet](http://image-net.org/) dataset manually and place it in `/path/to/imagenet`. For image classification examples, pass the argument `--data-path=/path/to/imagenet` to the training script. The extracted dataset directory should follow this structure: ```setup /path/to/imagenet/: train/: n01440764: n01440764_18.JPEG ... n01443537: n01443537_2.JPEG ... val/: n01440764: ILSVRC2012_val_00000293.JPEG ... n01443537: ILSVRC2012_val_00000236.JPEG ... ``` 4. Prepare the [MS-COCO 2017](https://cocodataset.org/#home) dataset manually and place it in `/path/to/coco`. For image classification examples, pass the argument `--data-path=/path/to/coco` to the training script. The extracted dataset directory should follow this structure: ```setup /path/to/coco/: annotations: many_json_files.json ... train2017: 000000000009.jpg ... val2017: 000000000139.jpg ... ``` 5. For [🗣️ Keyword Spotting subset](https://huggingface.co/datasets/s3prl/superb#ks), [Common Language](https://huggingface.co/datasets/speechbrain/common_language), [SQuAD](https://huggingface.co/datasets/rajpurkar/squad), [Common Voice](https://huggingface.co/datasets/legacy-datasets/common_voice), [GLUE](https://gluebenchmark.com/) and [WMT](https://huggingface.co/datasets/wmt/wmt17) datasets, manual downloading is not required — they will be automatically loaded via the Hugging Face Datasets library when running our `audio-classification`, `question-answering`, `speech-recognition`, `text-classification`, or `translation` examples. ## Training ### Model retraining We utilize recipes similar to those in [PyTorch Vision's classification reference](https://github.com/pytorch/vision/blob/main/references/classification/README.md) to retrain MobileNet-V2, ResNet, EfficientNet-V2, and ViT using our SPG on ImageNet. You can run the following command: ```train cd image-classification # MobileNet-V2 torchrun --nproc_per_node=4 train.py\ --data-path /path/to/imagenet/\ --model mobilenet_v2 --output-dir mobilenet_v2 --weights MobileNet_V2_Weights.IMAGENET1K_V1\ --batch-size 192 --epochs 40 --lr 0.0004 --lr-step-size 10 --lr-gamma 0.5 --wd 0.00004 --apply-trp --trp-depths 1 --trp-p 0.15 --trp-lambdas 0.4 0.2 0.1 # ResNet-50 torchrun --nproc_per_node=4 train.py\ --data-path /path/to/imagenet/\ --model resnet50 --output-dir resnet50 --weights ResNet50_Weights.IMAGENET1K_V1\ --batch-size 64 --epochs 40 --lr 0.0004 --lr-step-size 10 --lr-gamma 0.5 --print-freq 100\ --apply-trp --trp-depths 1 --trp-p 0.2 --trp-lambdas 0.4 0.2 0.1 # EfficientNet-V2 M torchrun --nproc_per_node=4 train.py \ --data-path /path/to/imagenet/\ --model efficientnet_v2_m --output-dir efficientnet_v2_m --weights EfficientNet_V2_M_Weights.IMAGENET1K_V1\ --epochs 10 --batch-size 64 --lr 5e-9 --lr-scheduler cosineannealinglr --weight-decay 0.00002 \ --lr-warmup-method constant --lr-warmup-epochs 8 --lr-warmup-decay 0. \ --auto-augment ta_wide --random-erase 0.1 --label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --norm-weight-decay 0.0 \ --train-crop-size 384 --val-crop-size 480 --val-resize-size 480 --ra-sampler --ra-reps 4 --print-freq 100\ --apply-trp --trp-depths 1 --trp-p 0.2 --trp-lambdas 0.4 0.2 0.1 # ViT-B-16 torchrun --nproc_per_node=4 train.py\ --data-path /path/to/imagenet/\ --model vit_b_16 --output-dir vit_b_16 --weights ViT_B_16_Weights.IMAGENET1K_V1\ --epochs 5 --batch-size 196 --opt adamw --lr 5e-9 --lr-scheduler cosineannealinglr --wd 0.3\ --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \ --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra --clip-grad-norm 1 --cutmix-alpha 1.0\ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 --print-freq 100 ``` We utilize recipes similar to those in [PyTorch Vision's segmentation reference](https://github.com/pytorch/vision/blob/main/references/segmentation/README.md) to retrain FCN and DeepLab-V3 using our SPG on COCO dataset. You can run the following command: ```train cd semantic-segmentation # FCN-ResNet50 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model fcn_resnet50 --aux-loss --output-dir fcn_resnet50 --weights FCN_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\ --epochs 5 --batch-size 16 --lr 0.0002 --aux-loss --print-freq 100\ --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # FCN-ResNet101 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model fcn_resnet101 --aux-loss --output-dir fcn_resnet101 --weights FCN_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\ --epochs 5 --batch-size 12 --lr 0.0002 --aux-loss --print-freq 100\ --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # DeepLabV3-ResNet50 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model deeplabv3_resnet50 --aux-loss --output-dir deeplabv3_resnet50 --weights DeepLabV3_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\ --epochs 5 --batch-size 16 --lr 0.0002 --aux-loss --print-freq 100\ --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # DeepLabV3-ResNet101 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model deeplabv3_resnet101 --aux-loss --output-dir deeplabv3_resnet101 --weights DeepLabV3_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\ --epochs 5 --batch-size 12 --lr 0.0002 --aux-loss --print-freq 100\ --lr-warmup-method constant --lr-warmup-epochs 3 --lr-warmup-decay 0. \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 ``` ### Transfer learning We utilize recipes similar to those in [HuggingFace Transformers' Examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/README.md) to retrain BERT and Wav2Vec using our SPG on GLUE benchmark, SquAD dataset, and SUPERB benchmark. You can run the following command: ```train cd text-classification # Task: CoLA CUDA_VISIBLE_DEVICES=0 python run_glue.py \ --model_name_or_path google-bert/bert-base-cased \ --task_name "cola" \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 32 \ --learning_rate 2.5e-5 \ --num_train_epochs 6 \ --output_dir "cola" \ --overwrite_output_dir \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # Task: SST-2 CUDA_VISIBLE_DEVICES=0 python run_glue.py \ --model_name_or_path google-bert/bert-base-cased \ --task_name "sst2" \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 64 \ --learning_rate 3e-5 \ --num_train_epochs 5 \ --output_dir "sst2" \ --overwrite_output_dir \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # Task: MRPC CUDA_VISIBLE_DEVICES=0 python run_glue.py \ --model_name_or_path google-bert/bert-base-cased \ --task_name "mrpc" \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 16 \ --learning_rate 2e-5 \ --num_train_epochs 4 \ --output_dir "mrpc" \ --overwrite_output_dir \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # Task: QQP CUDA_VISIBLE_DEVICES=0 python run_glue.py \ --model_name_or_path google-bert/bert-base-cased \ --task_name "qqp" \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 32 \ --learning_rate 1e-5 \ --num_train_epochs 10 \ --output_dir "qqp" \ --overwrite_output_dir \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # Task: QNLI CUDA_VISIBLE_DEVICES=0 python run_glue.py \ --model_name_or_path google-bert/bert-base-cased \ --task_name "qnli" \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 32 \ --learning_rate 2e-5 \ --num_train_epochs 10 \ --output_dir "qnli" \ --overwrite_output_dir \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # Task: RTE CUDA_VISIBLE_DEVICES=0 python run_glue.py \ --model_name_or_path google-bert/bert-base-cased \ --task_name "rte" \ --do_train \ --do_eval \ --max_seq_length 128 \ --per_device_train_batch_size 32 \ --learning_rate 5e-5 \ --num_train_epochs 5 \ --output_dir "rte" \ --overwrite_output_dir \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # Task: audio classification cd ../audio-classification CUDA_VISIBLE_DEVICES=0 python run_audio_classification.py \ --model_name_or_path facebook/wav2vec2-base \ --dataset_name superb \ --dataset_config_name ks \ --trust_remote_code \ --output_dir wav2vec2-base-ft-keyword-spotting \ --overwrite_output_dir \ --remove_unused_columns False \ --do_train \ --do_eval \ --fp16 \ --learning_rate 3e-5 \ --max_length_seconds 1 \ --attention_mask False \ --warmup_ratio 0.1 \ --num_train_epochs 8 \ --per_device_train_batch_size 64 \ --gradient_accumulation_steps 4 \ --per_device_eval_batch_size 32 \ --dataloader_num_workers 4 \ --logging_strategy steps \ --logging_steps 10 \ --eval_strategy epoch \ --save_strategy epoch \ --load_best_model_at_end True \ --metric_for_best_model accuracy \ --save_total_limit 3 \ --seed 0 \ --push_to_hub \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 # Task: question answering cd ../question-answering CUDA_VISIBLE_DEVICES=0 python run_qa.py \ --model_name_or_path google-bert/bert-base-uncased \ --dataset_name squad \ --do_train \ --do_eval \ --per_device_train_batch_size 12 \ --learning_rate 3e-5 \ --num_train_epochs 2 \ --max_seq_length 384 \ --doc_stride 128 \ --output_dir ./baseline \ --overwrite_output_dir \ --apply-trp --trp-depths 1 --trp-p 0.1 --trp-lambdas 0.4 0.2 0.1 ``` ### Network Architecture Search We conduct Neural Architecture Search (NAS) on the ResNet architecture using the ImageNet dataset. You can run the following command: ```train cd ../neural-architecture-search torchrun --nproc_per_node=4 train.py\ --data-path /home/cs/Documents/datasets/imagenet\ --model resnet18 --output-dir resnet18 --weights ResNet18_Weights.IMAGENET1K_V1\ --batch-size 128 --epochs 10 --lr 0.0004 --lr-step-size 2 --lr-gamma 0.5\ --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\ --apply-trp --trp-depths 3 3 3 --trp-planes 256 --trp-lambdas 0.4 0.2 0.1 --print-freq 100 # ✅ Test: Acc@1 73.900 Acc@5 91.536 torchrun --nproc_per_node=4 train.py\ --data-path /home/cs/Documents/datasets/imagenet\ --model resnet34 --output-dir resnet34 --weights ResNet34_Weights.IMAGENET1K_V1\ --batch-size 96 --epochs 10 --lr 0.0004 --lr-step-size 2 --lr-gamma 0.5\ --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\ --apply-trp --trp-depths 2 2 2 --trp-planes 256 --trp-lambdas 0.4 0.2 0.1 --print-freq 100 # ✅ Test: Acc@1 76.896 Acc@5 93.136 torchrun --nproc_per_node=4 train.py\ --data-path /home/cs/Documents/datasets/imagenet\ --model resnet50 --output-dir resnet50 --weights ResNet50_Weights.IMAGENET1K_V1\ --batch-size 64 --epochs 10 --lr 0.0004 --lr-step-size 2 --lr-gamma 0.5\ --lr-warmup-method constant --lr-warmup-epochs 1 --lr-warmup-decay 0.\ --apply-trp --trp-depths 1 1 1 --trp-planes 1024 --trp-lambdas 0.4 0.2 0.1 --print-freq 100 ``` ## Evaluation To evaluate our models on ImageNet, run: ```eval cd image-classification # Required: Download our MobileNet-V2 weights to /path/to/image-classification/mobilenet_v2 torchrun --nproc_per_node=4 train.py\ --data-path /path/to/imagenet/\ --model mobilenet_v2 --resume mobilenet_v2/model_32.pth --test-only # Required: Download our ResNet-50 weights to /path/to/image-classification/resnet50 torchrun --nproc_per_node=4 train.py\ --data-path /path/to/imagenet/\ --model resnet50 --resume resnet50/model_35.pth --test-only # Required: Download our EfficientNet-V2 M weights to /path/to/image-classification/efficientnet_v2_m torchrun --nproc_per_node=4 train.py\ --data-path /path/to/imagenet/\ --model efficientnet_v2_m --resume efficientnet_v2_m/model_7.pth --test-only\ --val-crop-size 480 --val-resize-size 480 # Required: Download our ViT-B-16 weights to /path/to/image-classification/vit_b_16 torchrun --nproc_per_node=4 train.py\ --data-path /path/to/imagenet/\ --model vit_b_16 --resume vit_b_16/model_4.pth --test-only ``` To evaluate our models on COCO, run: ```eval cd semantic-segmentation # eval baselines torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model fcn_resnet50 --aux-loss --weights FCN_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\ --test-only torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model fcn_resnet101 --aux-loss --weights FCN_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\ --test-only torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model deeplabv3_resnet50 --aux-loss --weights DeepLabV3_ResNet50_Weights.COCO_WITH_VOC_LABELS_V1\ --test-only torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model deeplabv3_resnet101 --aux-loss --weights DeepLabV3_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\ --test-only # eval our models # Required: Download our FCN-ResNet50 weights to /path/to/semantic-segmentation/fcn_resnet50 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model fcn_resnet50 --aux-loss --resume fcn_resnet50/model_4.pth\ --test-only # Required: Download our FCN-ResNet101 weights to /path/to/semantic-segmentation/fcn_resnet101 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model fcn_resnet101 --aux-loss --resume fcn_resnet101/model_4.pth\ --test-only # Required: Download our DeepLabV3-ResNet50 weights to /path/to/semantic-segmentation/deeplabv3_resnet50 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model deeplabv3_resnet50 --aux-loss --resume deeplabv3_resnet50/model_4.pth\ --test-only # Required: Download our DeepLabV3-ResNet101 weights to /path/to/semantic-segmentation/deeplabv3_resnet101 torchrun --nproc_per_node=4 train.py\ --workers 4 --dataset coco --data-path /path/to/coco/\ --model deeplabv3_resnet101 --aux-loss --weights DeepLabV3_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1\ --test-only ``` To evaluate our models on GLUE, SquAD, and SUPERB, please re-run the `transfer learning` related commands we previously declared, as these commands are used not only for training but also for evaluation. For Network Architecture Search, please run the following command to evaluate our SPG-trained ResNet-18 model: ```eval cd neural-architecture-search # Required: Download our ResNet-34 weights to /path/to/neural-architecture-search/resnet34 torchrun --nproc_per_node=4 train.py\ --data-path /home/cs/Documents/datasets/imagenet\ --model resnet34 --resume resnet34/model_8.pth --test-only # Required: Download our ResNet-50 weights to /path/to/neural-architecture-search/resnet50 torchrun --nproc_per_node=4 train.py\ --data-path /home/cs/Documents/datasets/imagenet\ --model resnet50 --resume resnet50/model_9.pth --test-only ``` ## License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.