Point tracking on a video clip (top: Motif, bottom: V-JEPA 2.1) — a query point propagated across frames by patch-feature cosine similarity. Motif stays locked on the subject noticeably better than V-JEPA 2.1.
- **Architecture**: ViT-7B (embed 4096 / depth 40 / heads 32), patch 16, 3D axial RoPE (`base=100`), SwiGLU FFN, LayerScale, per-head QK-norm, gated attention, 4 register tokens. - **Tokenizer**: `Conv3d(kernel=stride=(tubelet, patch, patch))` — image `(B,3,H,W)` → `T=1`, video `(B,T,3,H,W)`. Token layout `[CLS] + [register × 4] + [patch × N]`. - **This repo**: inference-only. Weights + self-contained `modeling_motif_vision_encoder.py` (loaded via `trust_remote_code`). Training/eval code lives in the GitHub repo. ## Usage The model ships a self-contained `modeling_motif_vision_encoder.py`, so it loads with `trust_remote_code=True`. ### Image ```python import torch from transformers import AutoImageProcessor, AutoModel from transformers.image_utils import load_image url = "http://images.cocodataset.org/val2017/000000039769.jpg" image = load_image(url) repo = "Motif-Technologies/Motif-Vision-Encoder-7B" processor = AutoImageProcessor.from_pretrained(repo) model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval() inputs = processor(images=image, return_tensors="pt").to(model.device, torch.bfloat16) with torch.inference_mode(): outputs = model(**inputs) outputs.last_hidden_state # (1, 1 + 4 + N, 4096) CLS + registers + patch tokens outputs.pooler_output # (1, 4096) global (CLS) representation patch_tokens = outputs.last_hidden_state[:, 5:, :] # (1, N, 4096), N = (H/16)*(W/16) ``` The processor resizes the shorter side to 512, center-crops to 512×512, and normalizes with ImageNet mean/std (BICUBIC). `H`/`W` must be multiples of 16. ### Video An image is a 1-frame clip; a video is the same call with a `(B, T, 3, H, W)` tensor. Apply the same per-frame transform (resize → center-crop → ImageNet norm) and stack over time: ```python import torch video = torch.randn(1, 8, 3, 256, 256, device="cuda", dtype=torch.bfloat16) # (B, T, 3, H, W) with torch.inference_mode(): outputs = model(pixel_values=video) ``` ## Model details | | | |---|---| | Backbone | ViT-7B, patch 16, embed 4096, depth 40, heads 32, SwiGLU | | Register tokens | 4 | | Position encoding | 3D axial RoPE (T,H,W), `base=100.0` | | Video tokenizer | 3D Conv, tubelet size 2 | | Precision | bf16 weights | | Training | DINO + iBOT + KoLeo self-distillation, Gram anchoring, contrastive caption alignment | | Training data | ~0.47B samples — 448.6M images (96%) + 18.5M video clips (4%) | Outputs (`BaseModelOutputWithPooling`): `last_hidden_state` `(B, 1+4+N, 4096)`, `pooler_output` `(B, 4096)`. ## Evaluation Compared against the strongest publicly reported self-supervised / vision backbones. Higher is better for every column **except KITTI depth** (lower is better). Best **comparable** value per column in **bold**, second best underlined; the footnoted figure (‡) is measured under a different protocol and is excluded from the ranking. DAVIS S/M/L follow the DINOv3 protocol (J&F-mean at video short side 420/480, 840/960, 1260/1440 px). V-JEPA 2.1 is not part of the DINOv3 Table 5 tracking benchmark, so only its single-resolution (S) figure is available. | Model | Training