| --- |
| license: apache-2.0 |
| tags: |
| - dino |
| - vision |
| datasets: |
| - imagenet-1k |
| --- |
| |
| # Vision Transformer (base-sized model, patch size 16) trained using DINO |
|
|
| Vision Transformer (ViT) model trained using the DINO method. It was introduced in the paper [Emerging Properties in Self-Supervised Vision Transformers](https://arxiv.org/abs/2104.14294) by Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin and first released in [this repository](https://github.com/facebookresearch/dino). |
|
|
| Disclaimer: The team releasing DINO did not write a model card for this model so this model card has been written by the Hugging Face team. |
|
|
| ## Model description |
|
|
| The Vision Transformer (ViT) is a transformer encoder model (BERT-like) pretrained on a large collection of images in a self-supervised fashion, namely ImageNet-1k, at a resolution of 224x224 pixels. |
|
|
| Images are presented to the model as a sequence of fixed-size patches (resolution 16x16), which are linearly embedded. One also adds a [CLS] token to the beginning of a sequence to use it for classification tasks. One also adds absolute position embeddings before feeding the sequence to the layers of the Transformer encoder. |
|
|
| Note that this model does not include any fine-tuned heads. |
|
|
| By pre-training the model, it learns an inner representation of images that can then be used to extract features useful for downstream tasks: if you have a dataset of labeled images for instance, you can train a standard classifier by placing a linear layer on top of the pre-trained encoder. One typically places a linear layer on top of the [CLS] token, as the last hidden state of this token can be seen as a representation of an entire image. |
|
|
| ## Intended uses & limitations |
|
|
| You can use the raw model for image classification. See the [model hub](https://huggingface.co/models?search=google/vit) to look for |
| fine-tuned versions on a task that interests you. |
|
|
| ### How to use |
|
|
| Here is how to use this model: |
|
|
| ```python |
| from transformers import ViTImageProcessor, ViTModel |
| from PIL import Image |
| import requests |
| |
| url = 'http://images.cocodataset.org/val2017/000000039769.jpg' |
| image = Image.open(requests.get(url, stream=True).raw) |
| |
| processor = ViTImageProcessor.from_pretrained('facebook/dino-vitb16') |
| model = ViTModel.from_pretrained('facebook/dino-vitb16') |
| |
| inputs = processor(images=image, return_tensors="pt") |
| outputs = model(**inputs) |
| last_hidden_states = outputs.last_hidden_state |
| ``` |
|
|
| ### BibTeX entry and citation info |
|
|
| ```bibtex |
| @article{DBLP:journals/corr/abs-2104-14294, |
| author = {Mathilde Caron and |
| Hugo Touvron and |
| Ishan Misra and |
| Herv{\'{e}} J{\'{e}}gou and |
| Julien Mairal and |
| Piotr Bojanowski and |
| Armand Joulin}, |
| title = {Emerging Properties in Self-Supervised Vision Transformers}, |
| journal = {CoRR}, |
| volume = {abs/2104.14294}, |
| year = {2021}, |
| url = {https://arxiv.org/abs/2104.14294}, |
| archivePrefix = {arXiv}, |
| eprint = {2104.14294}, |
| timestamp = {Tue, 04 May 2021 15:12:43 +0200}, |
| biburl = {https://dblp.org/rec/journals/corr/abs-2104-14294.bib}, |
| bibsource = {dblp computer science bibliography, https://dblp.org} |
| } |
| ``` |
