Add transformers-format weights (safetensors) + config

README.md

@@ -1,22 +1,18 @@
 ---
+library_name: transformers
+license: other
+license_name: fair-noncommercial-research-license
+pipeline_tag: image-feature-extraction
 tags:
 - eupe
-license: fair-noncommercial-research-license
-language:
-- en
+- dinov3
+- vision
+- image-feature-extraction
 ---
+
 # Model Card for EUPE
 
-Running AI models on smart edge devices can unlock various user experiences, but presents challenges
-due to limited compute and the need to handle multiple tasks simultaneously. This requires a vision
-encoder with small size but powerful and versatile representations. We present our method, Efficient
-Universal Perception Encoder (EUPE), which offers both inference efficiency and universally good
-representations for diverse downstream tasks. We achieve this by distilling from multiple domain-expert
-foundation vision encoders. Unlike previous agglomerative methods that directly scale down from
-multiple teachers to an efficient encoder, we demonstrate the importance of first scaling up to a large
-proxy teacher and then distilling from this single teacher. Experiments show that EUPE achieves
-on-par or better performance than individual domain experts of the same size on diverse task domains
-and also outperforms previous agglomerative encoders.
+Running AI models on smart edge devices can unlock various user experiences, but presents challenges due to limited compute and the need to handle multiple tasks simultaneously. This requires a vision encoder with small size but powerful and versatile representations. We present our method, Efficient Universal Perception Encoder (EUPE), which offers both inference efficiency and universally good representations for diverse downstream tasks. We achieve this by distilling from multiple domain-expert foundation vision encoders. Unlike previous agglomerative methods that directly scale down from multiple teachers to an efficient encoder, we demonstrate the importance of first scaling up to a large proxy teacher and then distilling from this single teacher. Experiments show that EUPE achieves on-par or better performance than individual domain experts of the same size on diverse task domains and also outperforms previous agglomerative encoders.
 
 ## Model Details
 
@@ -31,19 +27,18 @@ The models can accept larger images provided the image shapes are multiples of t
 
 ### Model Description
 
-- **Developed by:** Meta AI
-- **Model type:** Vision Transformer, ConvNeXt
-- **License:** [FAIR Research License](https://huggingface.co/facebook/fair-noncommercial-research-license/)
+- Developed by: Meta AI
+- Model type: Vision Transformer, ConvNeXt
+- License: FAIR Research License
 
 ### Model Sources
 
-- **Repository:** [https://github.com/facebookresearch/eupe](https://github.com/facebookresearch/eupe)
-- **Paper:** [https://arxiv.org/abs/2603.22387](https://arxiv.org/abs/2603.22387)
+- Repository: https://github.com/facebookresearch/eupe
+- Paper: https://arxiv.org/abs/2603.22387
 
 ## Uses
 
-The models are vision backbones providing multi-purpose features for downstream tasks, especially suitable for multi-task setting under limited compute budget. 
-The models can be used without fine-tuning, with downstream modules ranging from non-parametric operators, simple linear layers to heavier language decoders, to obtain competitive results:
+The models are vision backbones providing multi-purpose features for downstream tasks, especially suitable for multi-task setting under limited compute budget. The models can be used without fine-tuning, with downstream modules ranging from non-parametric operators, simple linear layers to heavier language decoders, to obtain competitive results:
 
 - on image classification, using k-NN classifiers on the class token
 - on semantic 3D keypoint correspondances
@@ -52,9 +47,32 @@ The models can be used without fine-tuning, with downstream modules ranging from
 
 ## Get Started
 
-Follow the [Installation](https://github.com/facebookresearch/EUPE/tree/main?tab=readme-ov-file#installation) to set up the environment. 
-Clone the [EUPE repo](https://github.com/facebookresearch/eupe) and download the PyTorch model checkpoints to local.
-The example below demonstrates how to obtain the class token and patch tokens given an input image.
+### With 🤗 Transformers
+
+This checkpoint is in the Transformers format: EUPE reuses the DINOv3 architecture, so it loads with `AutoModel`.
+
+```python
+import torch
+from transformers import AutoImageProcessor, AutoModel
+from transformers.image_utils import load_image
+
+image = load_image("http://images.cocodataset.org/val2017/000000039769.jpg")
+repo = "facebook/EUPE-ViT-B"
+
+processor = AutoImageProcessor.from_pretrained(repo)
+model = AutoModel.from_pretrained(repo)
+
+inputs = processor(images=image, return_tensors="pt")
+with torch.inference_mode():
+    outputs = model(**inputs)
+
+clstoken = outputs.pooler_output            # class token
+hidden_states = outputs.last_hidden_state   # class token, then (register and) patch tokens
+```
+
+### With the original code
+
+Follow the [Installation](https://github.com/facebookresearch/eupe) to set up the environment. Clone the [EUPE repo](https://github.com/facebookresearch/eupe) and download the PyTorch model checkpoints to local. The example below demonstrates how to obtain the class token and patch tokens given an input image.
 
 ```python
 import torch
@@ -89,157 +107,31 @@ with torch.inference_mode():
         batch_img = transform(img)[None]
         outputs = model.forward_features(batch_img)
 clstoken, patchtokens = outputs["x_norm_clstoken"], outputs["x_norm_patchtokens"]
-
 ```
 
 ## Results
 
 The reader is referred to the associated paper for details on the evaluation protocols.
 
-*Results for ViT backbones*
-
-<table>
-  <thead>
-    <tr>
-      <th rowspan="2">Model</th>
-      <th rowspan="2">#Params</th>
-      <th colspan="2">Image Understanding</th>
-      <th colspan="6">Vision Language Modeling</th>
-      <th colspan="3">Dense Prediction</th>
-    </tr>
-    <tr>
-      <th>IN1k-ZS</th>
-      <th>IN1k-KNN</th>
-      <th>TextVQA</th>
-      <th>SQA</th>
-      <th>Realworld</th>
-      <th>POPE</th>
-      <th>GQA</th>
-      <th>MMEp</th>
-      <th>SPair</th>
-      <th>NYUv2↓</th>
-      <th>ADE20k</th>
-    </tr>
-  </thead>
-  <tbody>
-    <tr>
-      <td>EUPE-ViT-T</td>
-      <td>6M</td>
-      <td>50.5</td>
-      <td>66.3</td>
-      <td>42.0</td>
-      <td>69.5</td>
-      <td>50.0</td>
-      <td>82.4</td>
-      <td>61.4</td>
-      <td>1258.0</td>
-      <td>37.2</td>
-      <td>0.571</td>
-      <td>36.7</td>
-    </tr>
-    <tr>
-      <td>EUPE-ViT-S</td>
-      <td>20M</td>
-      <td>69.8</td>
-      <td>78.2</td>
-      <td>44.1</td>
-      <td>69.3</td>
-      <td>51.7</td>
-      <td>84.5</td>
-      <td>65.0</td>
-      <td>1304.9</td>
-      <td>46.5</td>
-      <td>0.455</td>
-      <td>46.6</td>
-    </tr>
-    <tr>
-      <td>EUPE-ViT-B</td>
-      <td>86M</td>
-      <td>79.7</td>
-      <td>84.1</td>
-      <td>50.4</td>
-      <td>69.7</td>
-      <td>55.5</td>
-      <td>85.9</td>
-      <td>67.3</td>
-      <td>1374.5</td>
-      <td>51.3</td>
-      <td>0.391</td>
-      <td>52.4</td>
-    </tr>
-  </tbody>
-</table>
-
-*Results for ConvNeXt backbones
-
-<table>
-  <thead>
-    <tr>
-      <th rowspan="2">Model</th>
-      <th rowspan="2">#Params</th>
-      <th colspan="6">Vision Language Modeling</th>
-      <th colspan="3">Dense Prediction</th>
-    </tr>
-    <tr>
-      <th>TextVQA</th>
-      <th>SQA</th>
-      <th>Realworld</th>
-      <th>POPE</th>
-      <th>GQA</th>
-      <th>MMEp</th>
-      <th>SPair</th>
-      <th>NYUv2↓</th>
-      <th>ADE20k</th>
-    </tr>
-  </thead>
-  <tbody>
-    <tr>
-      <td>EUPE-ConvNeXt-T</td>
-      <td>29M</td>
-      <td>43.7</td>
-      <td>68.8</td>
-      <td>47.9</td>
-      <td>83.4</td>
-      <td>63.0</td>
-      <td>1278.1</td>
-      <td>41.3</td>
-      <td>0.430</td>
-      <td>43.5</td>
-    </tr>
-    <tr>
-      <td>EUPE-ConvNeXt-S</td>
-      <td>50M</td>
-      <td>45.0</td>
-      <td>68.9</td>
-      <td>50.5</td>
-      <td>84.0</td>
-      <td>64.7</td>
-      <td>1284.2</td>
-      <td>40.1</td>
-      <td>0.388</td>
-      <td>46.8</td>
-    </tr>
-    <tr>
-      <td>EUPE-ConvNeXt-B</td>
-      <td>89M</td>
-      <td>46.4</td>
-      <td>70.1</td>
-      <td>53.3</td>
-      <td>84.7</td>
-      <td>65.8</td>
-      <td>1348.9</td>
-      <td>37.7</td>
-      <td>0.365</td>
-      <td>48.9</td>
-    </tr>
-  </tbody>
-</table>
+Results for ViT backbones
 
-## Citation
+| Model | #Params | IN1k-ZS | IN1k-KNN | TextVQA | SQA | Realworld | POPE | GQA | MMEp | SPair | NYUv2↓ | ADE20k |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| EUPE-ViT-T | 6M | 50.5 | 66.3 | 42.0 | 69.5 | 50.0 | 82.4 | 61.4 | 1258.0 | 37.2 | 0.571 | 36.7 |
+| EUPE-ViT-S | 20M | 69.8 | 78.2 | 44.1 | 69.3 | 51.7 | 84.5 | 65.0 | 1304.9 | 46.5 | 0.455 | 46.6 |
+| EUPE-ViT-B | 86M | 79.7 | 84.1 | 50.4 | 69.7 | 55.5 | 85.9 | 67.3 | 1374.5 | 51.3 | 0.391 | 52.4 |
 
-**BibTeX**
+Results for ConvNeXt backbones
 
-```
+| Model | #Params | TextVQA | SQA | Realworld | POPE | GQA | MMEp | SPair | NYUv2↓ | ADE20k |
+|---|---|---|---|---|---|---|---|---|---|---|
+| EUPE-ConvNeXt-T | 29M | 43.7 | 68.8 | 47.9 | 83.4 | 63.0 | 1278.1 | 41.3 | 0.430 | 43.5 |
+| EUPE-ConvNeXt-S | 50M | 45.0 | 68.9 | 50.5 | 84.0 | 64.7 | 1284.2 | 40.1 | 0.388 | 46.8 |
+| EUPE-ConvNeXt-B | 89M | 46.4 | 70.1 | 53.3 | 84.7 | 65.8 | 1348.9 | 37.7 | 0.365 | 48.9 |
+
+## Citation
+
+```bibtex
 @misc{zhu2026eupe,
   title={Efficient Universal Perception Encoder},
   author={Zhu, Chenchen and Suri, Saksham and Jose, Cijo and Oquab, Maxime and Szafraniec, Marc and Wen, Wei and Xiong, Yunyang and Labatut, Patrick and Bojanowski, Piotr and Krishnamoorthi, Raghuraman and Chandra, Vikas},
@@ -249,4 +141,4 @@ The reader is referred to the associated paper for details on the evaluation pro
   primaryClass={cs.CV},
   url={https://arxiv.org/abs/2603.22387},
 }
-```
+```

config.json

@@ -0,0 +1,55 @@
+{
+  "apply_layernorm": true,
+  "architectures": [
+    "DINOv3ViTModel"
+  ],
+  "attention_dropout": 0.0,
+  "drop_path_rate": 0.0,
+  "dtype": "float32",
+  "hidden_act": "gelu",
+  "hidden_size": 768,
+  "image_size": 224,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "key_bias": false,
+  "layer_norm_eps": 1e-05,
+  "layerscale_value": 1e-05,
+  "mlp_bias": true,
+  "model_type": "dinov3_vit",
+  "num_attention_heads": 12,
+  "num_channels": 3,
+  "num_hidden_layers": 12,
+  "num_register_tokens": 4,
+  "out_features": [
+    "stage12"
+  ],
+  "out_indices": [
+    12
+  ],
+  "patch_size": 16,
+  "pos_embed_jitter": null,
+  "pos_embed_rescale": 2.0,
+  "pos_embed_shift": null,
+  "proj_bias": true,
+  "query_bias": true,
+  "reshape_hidden_states": true,
+  "rope_theta": 100.0,
+  "stage_names": [
+    "stem",
+    "stage1",
+    "stage2",
+    "stage3",
+    "stage4",
+    "stage5",
+    "stage6",
+    "stage7",
+    "stage8",
+    "stage9",
+    "stage10",
+    "stage11",
+    "stage12"
+  ],
+  "transformers_version": "5.10.0.dev0",
+  "use_gated_mlp": false,
+  "value_bias": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd2bf1ea838c5bb2d5acb2c87b168403acc5e5ebc214de65531edf048175b18b
+size 342662192

preprocessor_config.json

@@ -0,0 +1,22 @@
+{
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.485,
+    0.456,
+    0.406
+  ],
+  "image_processor_type": "DINOv3ViTImageProcessor",
+  "image_std": [
+    0.229,
+    0.224,
+    0.225
+  ],
+  "resample": 2,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "height": 256,
+    "width": 256
+  }
+}