Image Feature Extraction
Transformers
Safetensors
motif_vision
feature-extraction
motif
vision-transformer
self-supervised
video
custom_code
Instructions to use Motif-Technologies/Motif-Vision-Encoder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Motif-Technologies/Motif-Vision-Encoder with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-feature-extraction", model="Motif-Technologies/Motif-Vision-Encoder", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Motif-Technologies/Motif-Vision-Encoder", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -13,10 +13,10 @@ tags:
|
|
| 13 |
|
| 14 |
# Motif Vision Encoder
|
| 15 |
|
| 16 |
-
Motif Vision Encoder is a
|
| 17 |
-
backbone. A single
|
| 18 |
-
1-frame clip (`T=1`), a video is `T>1` — so the same weights produce
|
| 19 |
-
features
|
| 20 |
|
| 21 |
Trained on **~1/3 the data of DINOv3** (0.5B vs 1.7B samples), it still reaches competitive
|
| 22 |
performance across image and video benchmarks — and leads on DAVIS video tracking.
|
|
@@ -94,8 +94,8 @@ Outputs (`BaseModelOutputWithPooling`): `last_hidden_state` `(B, 1+4+N, 4096)`,
|
|
| 94 |
## Evaluation
|
| 95 |
|
| 96 |
Compared against the strongest publicly reported self-supervised / vision backbones. Higher is
|
| 97 |
-
better for every column
|
| 98 |
-
column in
|
| 99 |
different protocol and is excluded from the ranking.
|
| 100 |
|
| 101 |
DAVIS S/M/L follow the DINOv3 protocol (J&F-mean at video short side 420/480, 840/960,
|
|
@@ -114,17 +114,18 @@ single-resolution (S) figure is available.
|
|
| 114 |
|
| 115 |
<sub>‡ VideoMAEv2 ViT-g: K400 top-1 after **full fine-tuning**, not the frozen attentive-probe protocol used for the other rows.</sub>
|
| 116 |
|
| 117 |
-
Motif sets a new **state of the art on DAVIS video segmentation**, leading at every resolution
|
| 118 |
-
(74.0 / 80.5 / 83.5 J&F), and stays **competitive across the remaining image and video
|
| 119 |
-
benchmarks** — despite training on only **~0.5B samples**, roughly **1/3 of DINOv3's data**.
|
| 120 |
-
KITTI depth is still being evaluated and will be added as the run completes.
|
| 121 |
-
|
| 122 |
Protocol: DINOv3-style linear/attentive probes for image tasks; V-JEPA 2-style protocol for
|
| 123 |
video. Baseline DAVIS / ADE20K / K400 / KITTI figures are taken from the DINOv3 technical
|
| 124 |
report's unified evaluation (Tab. 3, 5, 6) and ImageNet from Tab. 7; OpenCLIP and VideoMAEv2
|
| 125 |
are not in that report; VideoMAEv2's K400 figure is cited from its own paper (see ‡), and
|
| 126 |
OpenCLIP's remaining benchmarks are not reported under a comparable protocol.
|
| 127 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
## License
|
| 129 |
|
| 130 |
Released under the **MIT License** (see `LICENSE`). The model was trained on data governed by the
|
|
|
|
| 13 |
|
| 14 |
# Motif Vision Encoder
|
| 15 |
|
| 16 |
+
Motif Vision Encoder is a unified image + video self-supervised vision encoder on a ViT
|
| 17 |
+
backbone. A single 3D-convolutional tokenizer ingests both modalities — an image is a
|
| 18 |
+
1-frame clip (`T=1`), a video is `T>1` — so the same weights produce dense patch-level
|
| 19 |
+
features and a language-aligned global (CLS) representation.
|
| 20 |
|
| 21 |
Trained on **~1/3 the data of DINOv3** (0.5B vs 1.7B samples), it still reaches competitive
|
| 22 |
performance across image and video benchmarks — and leads on DAVIS video tracking.
|
|
|
|
| 94 |
## Evaluation
|
| 95 |
|
| 96 |
Compared against the strongest publicly reported self-supervised / vision backbones. Higher is
|
| 97 |
+
better for every column except KITTI depth (lower is better). Best comparable value per
|
| 98 |
+
column in bold, second best <u>underlined</u>; the footnoted figure (‡) is measured under a
|
| 99 |
different protocol and is excluded from the ranking.
|
| 100 |
|
| 101 |
DAVIS S/M/L follow the DINOv3 protocol (J&F-mean at video short side 420/480, 840/960,
|
|
|
|
| 114 |
|
| 115 |
<sub>‡ VideoMAEv2 ViT-g: K400 top-1 after **full fine-tuning**, not the frozen attentive-probe protocol used for the other rows.</sub>
|
| 116 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
Protocol: DINOv3-style linear/attentive probes for image tasks; V-JEPA 2-style protocol for
|
| 118 |
video. Baseline DAVIS / ADE20K / K400 / KITTI figures are taken from the DINOv3 technical
|
| 119 |
report's unified evaluation (Tab. 3, 5, 6) and ImageNet from Tab. 7; OpenCLIP and VideoMAEv2
|
| 120 |
are not in that report; VideoMAEv2's K400 figure is cited from its own paper (see ‡), and
|
| 121 |
OpenCLIP's remaining benchmarks are not reported under a comparable protocol.
|
| 122 |
|
| 123 |
+
Motif sets a new state of the art on DAVIS video segmentation, leading at every resolution
|
| 124 |
+
(74.0 / 80.5 / 83.5 J&F), and stays competitive across the remaining image and video
|
| 125 |
+
benchmarks — despite training on only ~0.5B sample*, roughly 1/3 of DINOv3's data.
|
| 126 |
+
KITTI depth is still being evaluated and will be added as the run completes.
|
| 127 |
+
|
| 128 |
+
|
| 129 |
## License
|
| 130 |
|
| 131 |
Released under the **MIT License** (see `LICENSE`). The model was trained on data governed by the
|