README.md

metadata

license: mit
library_name: transformers
pipeline_tag: image-feature-extraction
tags:
  - motif
  - vision-transformer
  - self-supervised
  - image-feature-extraction
  - video
  - custom_code

Motif Vision Encoder

Motif Vision Encoder is a unified image + video self-supervised vision encoder on a ViT backbone. A single 3D-convolutional tokenizer ingests both modalities — an image is a 1-frame clip (T=1), a video is T>1 — so the same weights produce dense patch-level features and a language-aligned global (CLS) representation.

Trained on ~1/3 the data of DINOv3 (0.5B vs 1.7B samples), it still reaches competitive performance across image and video benchmarks — and leads on DAVIS video tracking.

Point tracking on a video clip (top: Motif, bottom: V-JEPA 2.1) — a query point propagated across frames by patch-feature cosine similarity. Motif stays locked on the subject noticeably better than V-JEPA 2.1.

• Architecture: ViT-7B (embed 4096 / depth 40 / heads 32), patch 16, 3D axial RoPE (base=100), SwiGLU FFN, LayerScale, per-head QK-norm, gated attention, 4 register tokens.
• Tokenizer: Conv3d(kernel=stride=(tubelet, patch, patch)) — image (B,3,H,W) → T=1, video (B,T,3,H,W). Token layout [CLS] + [register × 4] + [patch × N].
• This repo: inference-only. Weights + self-contained modeling_motif_vision_encoder.py (loaded via trust_remote_code). Training/eval code lives in the GitHub repo.

Usage

The model ships a self-contained modeling_motif_vision_encoder.py, so it loads with trust_remote_code=True.

Image

import torch
from transformers import AutoImageProcessor, AutoModel
from transformers.image_utils import load_image

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = load_image(url)

repo = "Motif-Technologies/Motif-Vision-Encoder-7B"
processor = AutoImageProcessor.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()

inputs = processor(images=image, return_tensors="pt").to(model.device, torch.bfloat16)
with torch.inference_mode():
    outputs = model(**inputs)

outputs.last_hidden_state   # (1, 1 + 4 + N, 4096)  CLS + registers + patch tokens
outputs.pooler_output       # (1, 4096)             global (CLS) representation

patch_tokens = outputs.last_hidden_state[:, 5:, :]   # (1, N, 4096), N = (H/16)*(W/16)

The processor resizes the shorter side to 512, center-crops to 512×512, and normalizes with ImageNet mean/std (BICUBIC). H/W must be multiples of 16.

Video

An image is a 1-frame clip; a video is the same call with a (B, T, 3, H, W) tensor. Apply the same per-frame transform (resize → center-crop → ImageNet norm) and stack over time:

import torch

video = torch.randn(1, 8, 3, 256, 256, device="cuda", dtype=torch.bfloat16)  # (B, T, 3, H, W)
with torch.inference_mode():
    outputs = model(pixel_values=video)

Model details

Backbone	ViT-7B, patch 16, embed 4096, depth 40, heads 32, SwiGLU
Register tokens	4
Position encoding	3D axial RoPE (T,H,W), base=100.0
Video tokenizer	3D Conv, tubelet size 2
Precision	bf16 weights
Training	DINO + iBOT + KoLeo self-distillation, Gram anchoring, contrastive caption alignment
Training data	~0.47B samples — 448.6M images (96%) + 18.5M video clips (4%)

Outputs (BaseModelOutputWithPooling): last_hidden_state (B, 1+4+N, 4096), pooler_output (B, 4096).

Evaluation

Compared against the strongest publicly reported self-supervised / vision backbones. Higher is better for every column except KITTI depth (lower is better). Best comparable value per column in bold, second best underlined; the footnoted figure (‡) is measured under a different protocol and is excluded from the ranking.

DAVIS S/M/L follow the DINOv3 protocol (J&F-mean at video short side 420/480, 840/960, 1260/1440 px). V-JEPA 2.1 is not part of the DINOv3 Table 5 tracking benchmark, so only its single-resolution (S) figure is available.

Model	Training data	DAVIS S J&F ↑	DAVIS M J&F ↑	DAVIS L J&F ↑	ImageNet-1K lin. probe ↑	ADE20K mIoU ↑	K400 ↑	KITTI depth MSE ↓
Motif Vision Encoder	0.5B	74.0	80.5	83.5	87.2	52.0	87.4	in progress
DINOv3	1.7B	71.1	79.7	83.3	88.4	55.9	87.8	2.3
PEcore	5.4B	48.2	53.1	49.8	89.3	38.9	87.9	4.1
SigLIP2	10B	56.1	62.3	62.9	89.1	45.4	86.9	–
OpenCLIP	2B	–	–	–	–	–	–	–
V-JEPA 2.1	0.022B	69.0	–	–	85.5	47.9	87.7	2.5
VideoMAEv2	1.35M	–	–	–	–	–	90.0‡	–

_{‡ VideoMAEv2 ViT-g: K400 top-1 after full fine-tuning, not the frozen attentive-probe protocol used for the other rows.}

Motif sets a new state of the art on DAVIS video segmentation, leading at every resolution (74.0 / 80.5 / 83.5 J&F), and stays competitive across the remaining image and video benchmarks — despite training on only ~0.5B samples, roughly 1/3 of DINOv3's data. KITTI depth is still being evaluated and will be added as the run completes.

Protocol: DINOv3-style linear/attentive probes for image tasks; V-JEPA 2-style protocol for video. Baseline DAVIS / ADE20K / K400 / KITTI figures are taken from the DINOv3 technical report's unified evaluation (Tab. 3, 5, 6) and ImageNet from Tab. 7; OpenCLIP and VideoMAEv2 are not in that report; VideoMAEv2's K400 figure is cited from its own paper (see ‡), and OpenCLIP's remaining benchmarks are not reported under a comparable protocol.

License

Released under the MIT License (see LICENSE). The model was trained on data governed by the respective dataset licenses; downstream users are responsible for compliance with those terms.

Citation

@misc{motif_vision_encoder,
  title  = {Motif Vision Encoder: a unified image/video self-supervised vision encoder},
  author = {Motif Technologies},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Motif-Technologies/Motif-Vision-Encoder-7B}}
}