dotimothy/sima-dinov2

DINOv2 for zero-shot change detection, compiled for the SiMa.ai MLSoC Modalix

DINOv2 (Meta) is a general image feature extractor. This is its ViT-S/14 with 4 registers (dinov2_vits14_reg4_pretrain), compiled for the Modalix MLA with SiMa.ai's Model Compiler. The whole network is one MLA stage, with nothing on the CPU (4 EV74 plugins for the data path), and one 448x448 image takes 24.5 ms.

Neat Change Studio uses it zero-shot to find what changed between a before and an after photo: nothing was trained for change detection.

file	precision	input	MLA latency per image	size
dinov2-448/dinov2_vits14_reg_448_bf16_mpk.tar.gz	BF16	448 x 448	24.5 ms	77 MB

• Latency: Model.benchmark on a Modalix DevKit (NEAT 0.5.0), plugins included: 50 images/s at 14.0 W. A before/after pair is two images.
• Features vs PyTorch FP32 (12 real photos): patch cosine 0.994, centred patch cosine 0.997.

How it finds changes

Both photos go through DINOv2, and so do overlapping tiles of them, which show small objects larger. A patch of the after photo with no similar patch nearby in the before photo is new; a patch of the before photo with no similar patch nearby in the after photo is gone. The score map is thresholded into a mask and boxes, each labelled appeared, disappeared or changed.

On testCYWS (28 before/after pairs of phone photos, 47 annotated changes, 15 pairs where nothing changed), scored with Change Studio's own metrics:

setting	changes found	pixel F1	false-alarm pairs
whole photo only (board bf16)	27 / 47	0.566	1 / 15
whole photo + 0.6 tiles (the app's balanced preset)	28 / 47	0.614	2 / 15
+ tiles, recall-leaning (threshold 0.35)	30 / 47	0.603	3 / 15
CYWS bf16, trained for change detection (sima-cyws)	33 / 47	0.587	2 / 15

• Caveat: for both models the settings were tuned on these same pairs, so treat the numbers as optimistic for new scenes.
• The tile rows were scored with FP32 features. On the whole photo, the board's bf16 features score 0.566 against 0.569 for FP32, so the compiled model loses next to nothing here.
• The tile settings: tiles count only where the whole-photo map is at least 0.2, threshold 0.40, neighbourhood radius 0, border 1, smallest box 0.2% of the photo, blur 1.

Input and output

• Input: one float32 448x448x3 tensor, HWC, RGB, pixel / 255. The ImageNet mean/std is inside the model.
• Output: float32 (1, 384, 1, 1029) after the final LayerNorm: 1029 = 1 class token + 4 registers + 32 x 32 patch tokens, row by row. The runtime may hand it back channels last, as (1029, 384).

Using it

On a Modalix board, use Neat Change Studio. Its web demo lists the models in this repository (models.json) and downloads one onto the board from the Models window (Add Model). From a terminal:

webapp/run.sh --list-hub          # what is here and what is on the board
webapp/run.sh --fetch dinov2-448  # downloaded into models/ and checked

Without the app:

hf download dotimothy/sima-dinov2 dinov2-448/dinov2_vits14_reg_448_bf16_mpk.tar.gz --local-dir .

import numpy as np, pyneat

opt = pyneat.ModelOptions()
opt.preprocess.kind = pyneat.InputKind.Tensor               # raw tensors in, no image preprocessing
model = pyneat.Model("dinov2-448/dinov2_vits14_reg_448_bf16_mpk.tar.gz", opt)

def tensor(hwc):
    t = pyneat.Tensor.from_numpy(np.ascontiguousarray(hwc, np.float32), copy=True)
    t.layout = pyneat.TensorLayout.HWC
    return t

image = np.zeros((448, 448, 3), np.float32)                 # your photo: RGB / 255, HWC
run = model.build([tensor(image)])
run.push_tensors([tensor(image)])
sample = run.pull(10000)
t = sample.tensors[0] if sample.tensors else sample.tensor
y = np.array(t.to_numpy(), np.float32, copy=True)
feat = y.reshape(384, -1).T if y.shape[-1] != 384 else y.reshape(-1, 384)   # (1029, 384)
cls, patches = feat[0], feat[5:].reshape(32, 32, 384)
run.close()

These files only run on the MLA. For PyTorch, use the original weights.

License and attribution

Apache-2.0. DINOv2, its code and pretrained weights, is by Meta Platforms, Inc. (https://github.com/facebookresearch/dinov2) and released under the Apache License 2.0.

These files are derived from those weights and are Apache-2.0 too:

• the network is rewritten to compile for the MLA, with the same function (within 5e-5);
• the ImageNet normalisation and the position embedding for 448x448 are folded in;
• it is compiled in bf16.

Nothing was retrained. The code that does this, and the change detection on top, is at dotimothy/neat-change-studio.

@article{oquab2023dinov2,
  title   = {DINOv2: Learning Robust Visual Features without Supervision},
  author  = {Oquab, Maxime and Darcet, Timoth{\'e}e and Moutakanni, Th{\'e}o and others},
  journal = {Transactions on Machine Learning Research},
  year    = {2024}
}
@inproceedings{darcet2024registers,
  title     = {Vision Transformers Need Registers},
  author    = {Darcet, Timoth{\'e}e and Oquab, Maxime and Mairal, Julien and Bojanowski, Piotr},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2024}
}