Argus

Copied from https://huggingface.co/phanerozoic/argus and exported to ONNX.

Argus is a multi-task vision model: task heads on a frozen EUPE-ViT-B backbone (Zhu et al., Meta FAIR, arXiv:2603.22387), 103M parameters in total, of which 17M are the heads.

Task	Head	Output	Trained on	Result
Classification	Linear softmax on the L2-normalized CLS token	1,000 ImageNet classes	ImageNet-1k	85.53% top-1, 97.69% top-5 (val)
Segmentation	BatchNorm and 1×1 conv	150 ADE20K classes	ADE20K	52.72 mIoU (val)
Depth	DPT decoder on blocks 2, 5, 8 and 11, 256 bins over 0.001–10 m	Metric depth	NYU Depth V2	RMSE 0.452, abs rel 0.117, δ<1.25 0.852 (654-image Eigen test split, Eigen crop, evaluate.py)
Detection	Cofiber pyramid with a CLIP-text-aligned cosine classifier, FCOS decoding	80 COCO classes	COCO 2017	42.64 mAP (val2017)
Correspondence	None: cosine matching of patch features	Patch matches	—	54.35 PCK@0.1 (SPair-71k test)

The detection head is documented in phanerozoic/cofiber-detection. Its class-agnostic AR@100 across 20 RF100-VL domains is 0.289, with the per-domain results in rf100vl_zero_shot_cross_domain_eval.json. Smaller variants are argus-lite (EUPE-ViT-S) and argus-edge (EUPE-ViT-T).

Usage

from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)
image = Image.open("image.jpg").convert("RGB")

model.classify(image, top_k=5)          # [{"class_id", "class_name", "score"}, ...]
model.segment(image)                    # [H, W] ADE20K class indices
model.depth(image)                      # [H, W] depth in meters
model.detect(image, score_thresh=0.3)   # [{"box", "score", "label", "class_name"}, ...]
model.embed(image)                      # [768] L2-normalized CLS embedding
model.perceive(image)                   # classification, segmentation and depth
model.correspond(image, other_image)    # {"matches", "scores", "grid"}

Each method also takes a list of images and returns a list of results. Images are resized to the resolution of the head: 224 for classification and embedding, 512 for segmentation, 416 for depth and 768 for detection, which letterboxes the image. Segmentation and depth maps come back at each image's own height and width, and boxes in its pixel coordinates. segment() and depth() return per-pixel confidence and depth standard deviation with return_confidence=True, and model.seg_class_names names the segmentation indices. The depth head is trained on indoor scenes and predicts at most 10 m.

variant="bf16_backbone" loads the 241 MB model.bf16_backbone.safetensors in place of the 412 MB model.safetensors, into the same FP32 model. model.compile() compiles the backbone blocks and the heads with torch.compile, model.quantize_int8() applies INT8 weight-only quantization through torchao, and model.export_onnx(out_dir, include_nms=True, verify=True) exports one shared dynamic-spatial backbone plus task-head graphs to ONNX and checks them against PyTorch with ONNX Runtime.

The default ONNX export resolutions are 224 for classification and embedding, 512 for segmentation, 416 for depth, and 768 for detection. The exporter writes a versioned argus_manifest.json with tensor names, shapes, preprocessing, labels, and NMS settings. The shared backbone accepts dynamic image height and width; each head is exported at its documented task resolution. Batch can also be dynamic.

For a single fixed-size export, pass backbone_resolution=768. This fixes the shared backbone and all task heads at 768 pixels while retaining the depth intermediate outputs and writing the same manifest contract.

Browser preview

browser.html is a single-file ONNX Runtime Web preview for the exported models. Before opening it, export with model.export_onnx("./onnx", include_nms=True), which writes the onnx/argus_manifest.json contract, one shared backbone graph and the task heads.

Serve the repository root over HTTP and open /browser.html; browsers block model and metadata fetches from file://. The page loads ONNX Runtime Web from jsDelivr, reads graph names, shapes, preprocessing, labels and NMS settings from the manifest, then runs classification, segmentation, depth, detection, CLS embedding and optional patch correspondence. It loads the shared backbone once and reuses it at each task's documented resolution.

evaluate.py scores the classification, segmentation, depth and detection heads on ImageNet-1k, ADE20K, NYU Depth V2 and COCO:

python evaluate.py imagenet --root imagenet/val
python evaluate.py ade20k --root ADEChallengeData2016
python evaluate.py nyu --root nyu_test
python evaluate.py coco --images val2017 --annotations instances_val2017.json

License

The EUPE-ViT-B weights are released under the FAIR Noncommercial Research License. The checkpoint contains them and is released under the same license; see LICENSE.

Citation

@misc{zhu2026eupe,
  title={Efficient Universal Perception Encoder},
  author={Zhu, Chenchen and Suri, Saksham and Jose, Cijo and Oquab, Maxime and Szafraniec, Marc and Wen, Wei and Xiong, Yunyang and Labatut, Patrick and Bojanowski, Piotr and Krishnamoorthi, Raghuraman and Chandra, Vikas},
  year={2026},
  eprint={2603.22387},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}