|
|
--- |
|
|
license: other |
|
|
tags: |
|
|
- vision |
|
|
- image-classification |
|
|
datasets: |
|
|
- imagenet-1k |
|
|
metrics: |
|
|
- accuracy |
|
|
|
|
|
--- |
|
|
|
|
|
# TFaugvit |
|
|
|
|
|
TFAugViT model is the tensorflow implementation of the |
|
|
[AugViT: Augmented Shortcuts for Vision Transformers](https://arxiv.org/pdf/2106.15941v1.pdf) by Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu and Yunhe Wang, |
|
|
and first released in [this repository](https://github.com/kingcong/augvit). |
|
|
|
|
|
|
|
|
## Model description |
|
|
|
|
|
Aug-ViT inserts additional paths with learnable parameters in parallel on the original shortcuts for alleviating the feature collapse. The block-circulant projection is used to implement augmented shortcut, which brings negligible increase of computational cost. |
|
|
|
|
|
## Intended uses & limitations |
|
|
|
|
|
This model can be used for image classification tasks and easily be fine-tuned to suite your purpose of use. |
|
|
|
|
|
### How to use |
|
|
|
|
|
Here is how to use this model to classify an image into one of the 1,000 ImageNet classes: |
|
|
|
|
|
```python |
|
|
from transformers import TFAutoModelForImageClassification |
|
|
from PIL import Image |
|
|
import requests |
|
|
|
|
|
url = "http://images.cocodataset.org/val2017/000000039769.jpg" |
|
|
image = Image.open(requests.get(url, stream=True).raw) |
|
|
|
|
|
model = TFAutoModelForImageClassification.from_pretrained("tensorgirl/TFaugvit",trust_remote_code=True) |
|
|
|
|
|
outputs = model({'pixel_values':image}) |
|
|
|
|
|
|
|
|
# model predicts one of the 1000 ImageNet classes |
|
|
predicted_class_idx = outputs.argmax(-1) |
|
|
``` |
|
|
|
|
|
## Training data |
|
|
|
|
|
The TFAugViT model is trained on [ImageNet-1k](https://huggingface.co/datasets/imagenet-1k), a dataset consisting of 1 million images and 1,000 classes. |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
Due to the use of einops library you cannot use the model,fit() directly on this model, you will have to either write a custom training loop by passing the inputs as shown above or you can wrap the model in a functional model of keras and specify the batch_size beforehand. |
|
|
If you want to train the model on some other data then either resize the images to 224x224 or change the model config image_size to suit your requirements. |
|
|
|
|
|
|
|
|
### Training hyperparameters |
|
|
|
|
|
The following hyperparameters were used during training: |
|
|
- optimizer: Adam |
|
|
- batch_size: 32 |
|
|
- training_precision: float32 |
|
|
- |
|
|
## Evaluation results |
|
|
|
|
|
| Model | ImageNet top-1 accuracy | # params | Resolution | |
|
|
|------------------|-------------------------|-----------|------------| |
|
|
| Aug-ViT-S | 81 | 22.2 M | 224x224 | |
|
|
| Aug-ViT-B | 82.4 | 86.5 M | 224x224| |
|
|
| Aug-ViT-B (Upsampled) | 84.2 | 86.5 M | 384x384| |
|
|
|
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- Transformers 4.33.2 |
|
|
- TensorFlow 2.13.0 |
|
|
- Tokenizers 0.13.3 |
|
|
|
|
|
### BibTeX entry and citation info |
|
|
|
|
|
```bibtex |
|
|
@inproceedings{aug-vit tf, |
|
|
title = {AugViT: Augmented Shortcuts for Vision Transformers}, |
|
|
author = {Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu and Yunhe Wang}, |
|
|
year = {2021}, |
|
|
URL = {https://arxiv.org/abs/2106.15941} |
|
|
} |
|
|
``` |
