phanerozoic commited on
Commit
87b8116
Β·
verified Β·
1 Parent(s): b2fd623

README: bulletize architecture, drop duplicate DPT paragraph, refresh classification baselines

Browse files
Files changed (1) hide show
  1. README.md +171 -180
README.md CHANGED
@@ -26,143 +26,99 @@ metrics:
26
 
27
  # Argus
28
 
29
- Argus is a multi-task perception system built on a single frozen vision backbone. One forward pass through the encoder produces classification labels, semantic segmentation masks, metric depth maps, object detections, and dense keypoint correspondences. Roughly 103M parameters total, with the 86M backbone frozen and about 17.3M learnable across five task heads. Named after Argus Panoptes, the many-eyed giant of Greek mythology tasked with watching over everything at once.
30
 
31
- The backbone is [EUPE-ViT-B](https://huggingface.co/facebook/EUPE-ViT-B), introduced in *Efficient Universal Perception Encoder* (Zhu et al., Meta FAIR, [arXiv:2603.22387](https://arxiv.org/abs/2603.22387), March 2026). EUPE distills a small vision encoder from a collection of larger specialist teachers, producing features that transfer well to image understanding, dense prediction, and vision-language tasks simultaneously. Argus leaves those weights frozen and attaches five lightweight heads.
32
 
33
  ## Architecture
34
 
35
  ```
36
- Image β†’ EUPE-ViT-B (frozen, 86M) β†’ shared features
37
 
38
- β”œβ”€β”€ Classification trained linear softmax, 1000 ImageNet classes
39
- β”œβ”€β”€ Segmentation BN + 1Γ—1 Conv, 150 ADE20K classes
40
- β”œβ”€β”€ Depth DPT multi-scale decoder, metric depth (meters), NYU Depth V2
41
- β”œβ”€β”€ Detection split-tower on a multi-scale feature decomposition, 80 COCO classes
42
- └── Correspondence training-free dense feature matching
43
  ```
44
 
45
- | Head | Params | Description |
46
- |---|---|---|
47
- | Classification | 769K | `Linear(768, 1000)` softmax on the L2-normalized CLS token |
48
- | Segmentation | 117K | `BatchNorm2d(768) β†’ Conv2d(768, 150, 1Γ—1)` at stride 16, bilinear-upsampled to input resolution |
49
- | Depth | 13.45M | DPT fusing backbone blocks [2, 5, 8, 11], 256 depth bins over 0.001 to 10 m |
50
- | Detection | 2.98M | 5 prediction levels at strides [8, 16, 32, 64, 128], cosine similarity against CLIP ViT-L/14 text embeddings |
51
- | Correspondence | 0 | cosine-max on backbone spatial features |
52
 
53
  ## Benchmarks
54
 
55
- ### EUPE paper reproduction
 
 
56
 
57
- All four reported benchmarks were reproduced as part of building Argus.
 
 
 
 
 
58
 
59
- | Task | Dataset | Metric | Paper | Argus | Delta |
60
- |---|---|---|---|---|---|
61
- | Classification | ImageNet-1k | kNN k=10 top-1 | 84.1 | 84.07 | βˆ’0.03 |
62
- | Segmentation | ADE20K | mean IoU | 52.4 | 52.72 | +0.32 |
63
- | Depth | NYU Depth V2 | RMSE (lower is better) | 0.391 | 0.3914 | +0.0004 |
64
- | Correspondence | SPair-71k | PCK@0.1 | 51.3 | 54.35 | +3.05 |
65
 
66
- ### Shipped task metrics
67
 
68
- | Task | Dataset | Metric | Value |
69
- |---|---|---|---|
70
- | Classification | ImageNet-1k val | top-1 / top-5 | 85.53 / 97.69 |
71
- | Segmentation | ADE20K val | mIoU | 52.72 |
72
- | Depth | NYU Depth V2 test | RMSE / abs_rel / a1 | 0.480 / 0.219 / 0.872 |
73
- | Detection | COCO val2017 | mAP @[.5:.95] | 42.64 (42.71 soft NMS) |
74
- | Correspondence | SPair-71k | PCK@0.1 | 54.35 |
75
 
76
- The shipped classifier is a trained linear softmax layer (85.53% top-1) that superseded the kNN protocol used during paper reproduction. The shipped depth head is a DPT decoder that improves RMSE by 8% and abs_rel by 28% over a linear probe on the same backbone (0.480 vs 0.520 RMSE).
77
 
78
- ### Detection detail (COCO val2017)
 
 
79
 
80
  | Metric | Value |
81
- |---|---|
82
- | mAP@[0.5:0.95] | **42.64** |
83
- | mAP@0.50 | 65.70 |
84
- | mAP@0.75 | 45.10 |
85
- | mAP (small / medium / large) | 22.31 / 48.33 / 62.90 |
86
-
87
- At 2.98M learnable parameters the detection head passes the 16.14M FCOS simple-feature-pyramid baseline (41.0 mAP) by +1.64, using 18.4% of its head parameter budget. Small-object mAP is 22.3 against FCOS's 19.4 (+2.9). The backbone was never exposed to detection data; these are the same frozen features used for every other task.
88
-
89
- Evaluation protocol: per-class hard NMS (IoU 0.5), score threshold 0.05, top-100 detections per image, pycocotools on COCO val2017.
90
-
91
- The standalone checkpoint and related detection-head work live in [phanerozoic/detection-heads](https://huggingface.co/phanerozoic/detection-heads).
92
-
93
- ### Cross-Dataset Detection Transfer
94
-
95
- To test whether the detection head's features generalize beyond COCO, the shipping 2.98M detection head (trained on COCO 2017 at 768px) and the 16.14M FCOS baseline (trained on COCO 2017 at 640px) were each evaluated zero-shot against the 20 RF100-VL validation domains. Both heads saw only COCO during training; RF100-VL was never exposed to either. Evaluation is class-agnostic AR@100 (all detections relabeled to a single "object" class, all ground-truth boxes relabeled likewise) so that localization transfer can be measured even on domains whose label space does not overlap COCO-80.
96
-
97
- | domain | FCOS (16.1M) | Ours (3.0M) | Ξ” |
98
- |---------------------------------------------------------|-------------:|------------:|------:|
99
- | actions | 37.5 | 39.6 | +2.1 |
100
- | aerial-airport | 16.1 | 17.3 | +1.1 |
101
- | all-elements | 2.3 | 7.9 | +5.6 |
102
- | aquarium-combined | 47.5 | 58.2 | +10.6 |
103
- | defect-detection | 0.1 | 0.3 | +0.2 |
104
- | dentalai | 0.2 | 0.9 | +0.7 |
105
- | flir-camera-objects | 53.1 | 54.3 | +1.2 |
106
- | gwhd2021 | 1.7 | 1.5 | -0.3 |
107
- | lacrosse-object-detection | 57.9 | 66.6 | +8.7 |
108
- | new-defects-in-wood | 5.6 | 14.6 | +9.0 |
109
- | orionproducts | 17.1 | 25.5 | +8.5 |
110
- | paper-parts | 19.3 | 22.2 | +2.8 |
111
- | recode-waste | 11.4 | 11.8 | +0.4 |
112
- | soda-bottles | 29.6 | 35.8 | +6.3 |
113
- | the-dreidel-project | 57.7 | 65.2 | +7.4 |
114
- | trail-camera | 60.1 | 69.6 | +9.5 |
115
- | water-meter | 0.7 | 0.0 | -0.6 |
116
- | wb-prova | 83.6 | 86.2 | +2.6 |
117
- | wildfire-smoke | 0.3 | 0.5 | +0.2 |
118
- | x-ray-id | 0.0 | 0.0 | 0.0 |
119
- | **RF100-VL AR@100 mean** | **25.1** | **28.9** | **+3.8** |
120
- | **Domain wins** | **3** | **17** | |
121
-
122
- The detection head wins 17 of 20 domains, loses 3, with mean AR@100 +3.8 over the 5Γ— larger FCOS baseline. The largest gaps are on domains far from COCO's distribution: aquarium-combined (+10.6), trail-camera (+9.5), new-defects-in-wood (+9.0), lacrosse-object-detection (+8.7), orionproducts (+8.5), the-dreidel-project (+7.4), soda-bottles (+6.3), all-elements (+5.6). The three losses are small (≀0.6 AR) on domains with very low absolute AR for both heads (gwhd2021 wheat-head crops, water-meter digit reads, x-ray-id anatomical landmarks). The interpretation is that the backbone's multi-teacher distilled features produce representations general enough that a frozen head one-fifth the FCOS size transfers across wildly different visual domains at the same level or better.
123
-
124
- ### Cross-domain detection benchmark (RF100-VL subset)
125
-
126
- YOLO26 (n/s/m/l/x) and RF-DETR (Nano/Small/Medium/Large) on the same 20 RF100-VL domains, same class-agnostic AR@100 protocol. All trained on COCO only.
127
-
128
- | domain | Argus+FCOS | Argus+(current) | RF-DETR-L | RF-DETR-M | RF-DETR-S | YOLO26l | RF-DETR-N | YOLO26x | YOLO26m | YOLO26s | YOLO26n |
129
- |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
130
- | actions | 37.5 | 39.6 | 46.0 | 43.9 | 42.4 | 47.0 | 37.8 | 49.1 | 45.8 | 43.4 | 35.8 |
131
- | aerial-airport | 16.1 | 17.3 | 19.5 | 16.4 | 15.5 | 13.9 | 13.1 | 17.8 | 14.1 | 10.0 | 8.9 |
132
- | all-elements | 2.2 | 7.9 | 25.5 | 23.9 | 21.6 | 8.4 | 19.8 | 6.9 | 10.2 | 8.6 | 7.6 |
133
- | aquarium-combined | 47.5 | 58.2 | 56.5 | 55.0 | 53.1 | 52.7 | 51.1 | 52.0 | 50.7 | 50.1 | 44.1 |
134
- | defect-detection | 0.1 | 0.3 | 8.7 | 8.9 | 9.6 | 7.7 | 8.9 | 7.6 | 5.6 | 4.8 | 5.2 |
135
- | dentalai | 0.2 | 0.9 | 11.5 | 11.6 | 11.2 | 8.0 | 8.9 | 3.6 | 2.0 | 5.0 | 9.8 |
136
- | flir-camera-objects | 53.1 | 54.3 | 54.6 | 52.5 | 49.8 | 55.4 | 45.8 | 56.1 | 54.3 | 52.1 | 48.3 |
137
- | gwhd2021 | 1.7 | 1.5 | 20.1 | 17.7 | 15.5 | 21.1 | 10.0 | 24.1 | 16.2 | 10.9 | 9.4 |
138
- | lacrosse-object-detection | 57.9 | 66.6 | 58.5 | 57.7 | 57.7 | 59.7 | 56.0 | 60.7 | 60.2 | 59.9 | 58.9 |
139
- | new-defects-in-wood | 5.6 | 14.6 | 24.0 | 22.8 | 21.5 | 17.7 | 20.6 | 16.9 | 15.4 | 17.0 | 19.4 |
140
- | orionproducts | 17.1 | 25.5 | 15.8 | 14.5 | 14.8 | 30.9 | 13.2 | 11.6 | 26.5 | 29.8 | 22.1 |
141
- | paper-parts | 19.3 | 22.2 | 25.9 | 23.7 | 25.4 | 5.7 | 32.5 | 13.2 | 8.1 | 13.5 | 11.4 |
142
- | recode-waste | 11.4 | 11.8 | 40.2 | 36.5 | 34.0 | 38.4 | 28.3 | 32.2 | 36.3 | 36.6 | 40.4 |
143
- | soda-bottles | 29.6 | 35.8 | 45.3 | 45.0 | 44.6 | 42.9 | 43.5 | 40.4 | 34.2 | 35.8 | 32.6 |
144
- | the-dreidel-project | 57.7 | 65.1 | 64.7 | 66.1 | 63.1 | 58.8 | 59.4 | 62.4 | 58.3 | 53.4 | 53.7 |
145
- | trail-camera | 60.1 | 69.6 | 67.4 | 67.2 | 67.0 | 64.1 | 66.0 | 64.8 | 61.7 | 58.5 | 55.4 |
146
- | water-meter | 0.7 | 0.0 | 23.5 | 23.1 | 17.1 | 11.3 | 22.5 | 9.2 | 14.8 | 16.8 | 35.5 |
147
- | wb-prova | 83.6 | 86.2 | 80.8 | 79.3 | 79.4 | 79.3 | 77.3 | 81.0 | 78.9 | 77.0 | 75.7 |
148
- | wildfire-smoke | 0.3 | 0.5 | 0.3 | 0.6 | 2.0 | 0.5 | 0.9 | 0.5 | 2.0 | 0.5 | 0.8 |
149
- | x-ray-id | 0.0 | 0.0 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.0 | 0.0 | 0.0 | 0.0 |
150
- | **mean AR@100** | **25.1** | **28.9** | **34.4** | **33.3** | **32.3** | **31.2** | **30.8** | **30.5** | **29.8** | **29.2** | **28.8** |
151
- | **total params** | 102.1M | 89.0M | 33.9M | 33.7M | 32.1M | 26.3M | 30.5M | 59.0M | 21.9M | 10.0M | 2.6M |
152
- | **latency (ms)** | 15.4 | 19.6 | 11.5 | 11.5 | 10.9 | 18.5 | 10.2 | 17.3 | 13.7 | 10.2 | 10.1 |
153
 
154
  ### Cross-Dataset Segmentation Transfer
155
 
156
- A separate BN+1Γ—1 linear probe with the same training recipe as the ADE20K head, on the frozen backbone. The backbone was never exposed to driving scenes during EUPE distillation or Argus head training.
157
 
158
  | Dataset | Classes | Train images | mIoU |
159
- |---|---|---|---|
160
- | ADE20K (shipped head) | 150 | 20,210 | 52.72 |
161
- | Cityscapes (transfer probe) | 19 | 2,975 | 63.76 |
162
 
163
- The Cityscapes probe scores road 96.4, car 87.9, sky 88.8, building 86.7, vegetation 85.6. The weaker categories are thin vertical structures (pole 17.8, traffic light 36.4, traffic sign 48.3), which is an inherent resolution limitation of the stride-16 patch grid rather than a deficiency in the learned representation.
164
 
165
- ## Comparison with standard baselines
166
 
167
  As a sanity check, Argus was compared against several well-known models on the same 200-image COCO subset. The classification comparison uses a keyword cross-reference between each model's top-k ImageNet predictions and the COCO ground-truth detection labels on those images, which provides a consistent yardstick across differently-trained models despite the label-space mismatch. **These hit rates measure agreement with COCO detection labels via keyword matching on the 200-image subset; they are not raw ImageNet accuracy.** For reference, all three classifiers exceed 80% top-1 on the full ImageNet validation set.
168
 
@@ -170,9 +126,9 @@ As a sanity check, Argus was compared against several well-known models on the s
170
 
171
  | Model | Parameters | Top-1 hit | Top-5 hit | Latency | Peak VRAM |
172
  |--------------------|------------|-----------|-----------|---------|-----------|
173
- | Argus (EUPE-ViT-B) | 86 M | 42.2% | 66.8% | 13.1 ms | 0.34 GB |
174
- | ConvNeXt-Base | 89 M | 40.2% | 71.4% | 10.4 ms | 0.35 GB |
175
- | ResNet50 | 26 M | 36.2% | 61.8% | 8.4 ms | 0.12 GB |
176
 
177
  **Segmentation**:
178
 
@@ -188,22 +144,22 @@ As a sanity check, Argus was compared against several well-known models on the s
188
  | Argus (EUPE + linear head) | 86 M | 13.3 ms | 0.35 GB |
189
  | Depth-Anything-V2-Base | 98 M | 18.8 ms | 0.68 GB |
190
 
191
- Argus produces the top-1 classification accuracy of the three image classifiers, with ConvNeXt-Base edging it slightly on top-5. The Argus classification row above was measured with the kNN method during the original head-to-head comparison; the current shipped classifier (trained linear softmax) would widen the top-5 margin. Argus is faster than DeepLabV3 while predicting a much richer label space, and it is faster than Depth-Anything-V2 while using roughly half the VRAM. Although these baselines and Argus were trained for different objectives on different datasets, the comparison is useful for understanding what the model delivers in practice.
192
 
193
  ### Multi-Task Throughput
194
 
195
  The per-task comparisons above measure each head against its single-task counterpart in isolation. A separate question is what happens when a user needs all of the tasks at once, which is the typical situation in dataset annotation, model evaluation, and any pipeline where images pass through multiple analysis stages in sequence. The alternative to Argus in that situation is to load and run four separate single-task models of comparable quality, each carrying its own backbone, its own preprocessing, and its own forward pass. The total cost is the sum of the four individual inference times, plus the memory overhead of holding four independent models on the device simultaneously.
196
 
197
- The models chosen for this comparison were selected to match the quality tier of the EUPE-ViT-B backbone rather than to minimize size or maximize speed. ConvNeXt-Base (88.6M parameters) is a widely-used ImageNet-1k classifier at the same parameter scale as EUPE-ViT-B. SegFormer-B3 (47.3M) is a transformer-based ADE20K semantic segmenter that is the standard mid-range alternative to a linear probe on a frozen backbone. Depth-Anything-V2-Base (97.5M) is the current standard for single-image monocular depth estimation at base scale. YOLO26l (26.3M) is the large variant of the January 2026 YOLO release from Ultralytics, representing the state of the art in efficient real-time detection. All measurements were taken on an NVIDIA RTX 6000 Ada across the same nine example images, with five timed runs after a three-image warmup pass to eliminate cold-start effects.
198
 
199
  | Pipeline | Parameters | Latency per image | Tasks |
200
  |----------|-----------|-------------------|-------|
201
- | Argus unified | 103 M | 56 ms | 5 (classify, segment, depth, detect, correspond) |
202
  | Four separate models | 260 M | 68 ms | 4 (classify, segment, depth, detect) |
203
 
204
- The per-model breakdown for the separate pipeline is ConvNeXt-Base at 6 ms, SegFormer-B3 at 19 ms, Depth-Anything-V2-Base at 31 ms, and YOLO26l at 12 ms, summing to 68 ms when the tasks are run sequentially on the same image. Argus completes five tasks (the same four plus keypoint correspondence, which the separate pipeline does not attempt) in 56 ms from a single model load. The total parameter count for the separate pipeline is 260M across four independent weight sets, while Argus carries 103M in a single file.
205
 
206
- The throughput advantage comes from the shared backbone. Each of the four separate models pays the cost of encoding the image through its own network before producing task-specific output. Argus encodes the image once through EUPE-ViT-B and then routes the resulting features to five lightweight heads, each of which adds only a few milliseconds on top of the shared representation. The backbone forward pass is the dominant cost in both pipelines, and running it once rather than four times is where the 1.2x throughput improvement and 2.5x parameter reduction originate. The practical consequence for deployment is that Argus requires a single model download, a single checkpoint load, and a single Python import, where the equivalent separate-model pipeline requires four downloads totaling over a gigabyte, four independent weight sets held concurrently, and four separate dependency trees to manage.
207
 
208
  ## Usage
209
 
@@ -212,26 +168,48 @@ from PIL import Image
212
  from transformers import AutoModel
213
 
214
  model = AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)
 
215
  image = Image.open("your_image.jpg").convert("RGB")
216
 
217
- top5 = model.classify(image, top_k=5)
218
- seg = model.segment(image) # [H, W] class indices
219
- depth = model.depth(image) # [H, W] metric depth in meters
220
- dets = model.detect(image, score_thresh=0.3)
221
- # dets: list of {"box": [x1, y1, x2, y2], "score", "label", "class_name"}
222
 
223
- # Three tasks at once (shared backbone forward inside perceive)
 
 
 
 
224
  result = model.perceive(image)
225
- # result["classification"], ["segmentation"], ["depth"], ["timings_ms"]
 
 
 
 
 
 
226
 
227
- # Keypoint correspondence between two images
228
  target = Image.open("other_image.jpg").convert("RGB")
229
- predicted = model.correspond(image, target, [[100, 100], [200, 200]])
 
 
 
230
  ```
231
 
232
- Every single-image method also accepts a list of PIL images and returns a list of per-image results in the same shape a single call would produce.
 
 
 
 
 
 
 
 
 
233
 
234
- ### Confidence outputs
235
 
236
  ```python
237
  seg_map, seg_conf = model.segment(image, return_confidence=True)
@@ -241,99 +219,112 @@ depth_map, depth_std = model.depth(image, return_confidence=True)
241
  # depth_std is per-pixel standard deviation of the 256-bin distribution
242
 
243
  result = model.perceive(image, return_confidence=True)
244
- # result["segmentation_confidence"], result["depth_uncertainty"]
245
  ```
246
 
247
- Classification always carries a `margin` field (top-1 minus top-2 score) on the first entry.
248
-
249
- ### ONNX export
250
 
251
  ```python
252
- paths = model.export_onnx("/path/to/out_dir", backbone_resolution=640, verify=True)
253
- # backbone, classifier, seg_head, depth_head, detection_head (five graphs)
 
254
  ```
255
 
256
- The segmentation graph folds bilinear upsample to input resolution inside the graph, so consumers argmax directly. The classifier graph is self-contained (softmax weights captured as buffers). The depth head accepts four intermediate ViT-block activations as separate positional tensor inputs. The detection head returns pre-NMS per-location boxes and scores by default, or with `include_nms=True` bakes ONNX `NonMaxSuppression` (opset β‰₯ 10) into the detection graph for single-shot TensorRT or mobile inference. Correspondence has no learned parameters and needs no graph.
257
 
258
- Tolerance for `verify=True` can be a float or a dict keyed by verification output name. When a float is passed, detection box coordinates get a resolution-scaled tolerance because `exp()` in the regression path amplifies FP kernel-dispatch differences to pixel scale.
259
-
260
- ### INT8 quantization
261
 
262
  ```python
263
  model = AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)
264
  model = model.cuda().eval().quantize_int8() # requires: pip install torchao
 
 
 
 
265
  ```
266
 
267
- Weight-only INT8 quantization via torchao. Linear weights go to INT8; activations stay in BF16. Classification agreement with FP32 is 100%, depth drift averages 0.013 m. Reduces weight VRAM substantially. Latency behaviour depends on whether the target GPU has an INT8 tensor-core path torchao can dispatch to.
 
 
 
268
 
269
- ### Precision variants
 
 
270
 
271
- Two safetensors with identical inference behaviour but different on-disk precision.
272
 
273
- | File | Load |
274
- |---|---|
275
- | `model.safetensors` | `AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)` |
276
- | `model.bf16_backbone.safetensors` | add `variant="bf16_backbone"` |
277
 
278
- Both load into the same FP32 model in memory; PyTorch upcasts the stored bfloat16 weights at construction. The smaller variant saves download bandwidth only.
279
 
280
- ## Training
 
 
 
 
 
 
 
 
281
 
282
- The backbone is frozen for every task. Only the task heads are trained; the kNN class prototypes used during paper reproduction were extracted (not trained at all).
283
 
284
- | Component | Source | Method |
285
- |---|---|---|
286
- | Segmentation | ADE20K (20,210 train) | Linear probe, CE loss, AdamW lr 1e-3, 512Γ—512, 40,000 iterations |
287
- | Depth | NYU Depth V2 (24,231 train) | DPT decoder, SILog loss, AdamW lr 1e-4, 416Γ—416, 38,400 iterations |
288
- | Linear softmax classifier | ImageNet-1k (1.28M train) | Cached CLS features, SGD momentum 0.9, cosine LR, 100 epochs |
289
- | Detection | COCO 2017 (117,266 train) | Split-tower on a multi-scale decomposition of frozen features, ATSS, focal + GIoU + BCE, AdamW lr 5e-4, 768Γ—768, 16 ep + 3 ep partial calibration |
290
- | Correspondence | none | training-free cosine similarity |
291
 
292
- ### Backbone simplification
293
 
294
- The upstream EUPE-ViT-B release ships a `LinearKMaskedBias` wrapper around each block's QKV projection. In the released weights both the `bias_mask` and the `bias` are filled with zeros across all twelve blocks, so the masked bias is identically zero at every forward pass. The Argus backbone drops the 24 redundant tensors entirely (12 Γ— `qkv.bias` + 12 Γ— `qkv.bias_mask`, 55,296 values total), and the attention blocks are constructed with `qkv_bias=False, mask_k_bias=False`. FP32 forward is bitwise-equivalent for classification, segmentation, detection, and correspondence. The DPT depth decoder shows sub-centimeter drift under BF16 autocast; the drift is an order of magnitude smaller than the head's own 39-centimeter NYU Depth V2 RMSE and causes no visible change in depth maps. To load the upstream EUPE-ViT-B release directly into this backbone class, pass `strict=False` to `load_state_dict` so the extra keys in the upstream checkpoint are silently ignored.
 
 
 
295
 
296
- ### Head details
297
 
298
- **Segmentation.** `BatchNorm2d(768) β†’ Conv2d(768, 150, 1Γ—1)`, 116,886 parameters. Trained at 512Γ—512 with cross-entropy loss, AdamW (lr 1e-3, weight decay 1e-3), WarmupOneCycleLR with 1500-step warmup, batch 16.
299
 
300
- **Depth (DPT).** Hooks into backbone blocks [2, 5, 8, 11] via PyTorch forward hooks, capturing intermediate representations without modifying the backbone. A reassemble stage projects each block's output from 768 to 256 channels via LayerNorm + Linear, reshapes to spatial grids, and rescales to strides [4, 8, 16, 32]. A bottom-up fusion path combines the four scales through residual conv blocks with skip connections. A final conv head produces 256 depth-bin logits; metric depth is the bin-weighted sum. 13,450,000 parameters. Trained at 416Γ—416 with SILog loss, AdamW (lr 1e-4, weight decay 1e-3), cosine schedule with 3% warmup, batch 16, 38,400 iterations.
301
 
302
- **Linear softmax classifier.** A single `Linear(768, 1000)` layer with bias applied to the L2-normalized CLS token, 769,000 parameters. Trained as a two-pass job: first the frozen backbone runs over ImageNet-1k train to cache a per-image CLS feature tensor (1,281,167 Γ— 768), then the linear layer trains on the cached features alone with SGD (momentum 0.9, weight decay 0), batch 4096, cosine schedule, 100 epochs, no augmentation. A small LR sweep over {0.5, 1.0, 3.0, 10.0, 30.0} selected lr=30.0: L2-normalized features plus zero-initialized weights require an unusually large learning rate to grow the weight scale to the point where the softmax distribution sharpens. The best run reached 85.53% top-1 and 97.69% top-5 on ImageNet-1k val.
303
 
304
- **Detection (split-tower on a cofiber decomposition of frozen features).** Anchor-free. The multi-scale decomposition is applied per-channel to the 768-D feature map: 2Γ—2 average pool reduce, bilinear upsample expand, band_k = x_k βˆ’ U(D(x_k)) with x_{k+1} = D(x_k). Zero parameters, replacing an 11M-parameter FPN. Five prediction levels at strides [8, 16, 32, 64, 128]: four bands at 16, 32, 64, 128, plus a stride-8 level from a single transposed convolution on the stride-16 band. The Rocq/HoTT formalization in phanerozoic/cofiber-detection proves that this is a split short exact sequence in any semi-additive category with an adjoint retraction pair (U, D), with each band equal to ker(D_k). Burt & Adelson's 1983 Laplacian pyramid is the scalar-image instance with Gaussian reduce. Separate classification and regression towers of depth nine (five 3Γ—3 ConvGN blocks followed by four depthwise residual blocks at 160 hidden channels) process each level with weights shared across levels. Top-down lateral connections pass information from coarser to finer bands before the towers run. Classification is cosine similarity between a `Linear(160, 768)` projection and CLIP ViT-L/14 multi-prompt text embeddings of the 80 COCO class names, with a learned scalar temperature and per-class bias. Regression uses exponentiated LTRB distances with a learned per-level scale. Centerness is a single 1Γ—1 convolution. 2,975,067 parameters.
305
 
306
- Trained at 768Γ—768 with letterbox padding, ATSS target assignment (Zhang et al. 2020), horizontal-flip augmentation, focal loss (Ξ±=0.25, Ξ³=2.0) for classification, GIoU for boxes, BCE for centerness, AdamW (lr 5e-4, weight decay 1e-4), cosine schedule with 3% warmup, batch 16, 16 epochs. Step 104,000 of 117,264 was selected by late-training checkpoint sweep as the base. A 3-epoch partial fine-tune at lr 1e-4 then updates only `cls_project`, `cls_bias`, and `logit_scale` (the classification calibration layers), leaving the towers and the decomposition path frozen. The partial fine-tune adds +0.15 aggregate mAP and +1.1 small-object mAP. The shipped weights are the final state of that fine-tune. The standalone detection-head checkpoint is mirrored in the sibling detection-heads repo at [`heads/cofiber_threshold/split_tower_5scale_160h_5std_4dw_ema_l14_16ep_768_cls_calib/checkpoint_final.pth`](https://huggingface.co/phanerozoic/detection-heads/blob/main/heads/cofiber_threshold/split_tower_5scale_160h_5std_4dw_ema_l14_16ep_768_cls_calib/checkpoint_final.pth) with its eval JSON alongside.
307
 
308
- **Correspondence.** No learned parameters. At inference, dense patch features are extracted from both images, upsampled to 512Γ—512 pixel resolution, and matched by cosine similarity per source keypoint.
 
 
309
 
310
  ### Compute
311
 
312
- | Task | Iterations | Notes |
313
- |----------------------------------|-----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
314
- | Segmentation (ADE20K) | 40,000 | linear probe, batch 16, 512px, CE loss, frozen backbone |
315
- | Linear classifier (ImageNet-1k) | 100 epochs Γ— 313 steps | SGD momentum 0.9, batch 4096, cosine schedule on cached CLS features; extraction is a single full-train pass through the frozen backbone |
316
- | DPT depth decoder (NYU Depth V2) | 38,400 iterations | batch 16, 416px, SILog loss, frozen backbone |
317
- | Detection (COCO 2017) | 16 epochs Γ— 7,329 batches at 768px + 3-epoch partial fine-tune of classification calibration layers | bf16 mixed-precision forward + fp32 master params + fp32 AdamW moments, CUDA graph capture, frozen backbone |
318
- | Correspondence (SPair-71k) | training-free | |
 
 
 
 
319
 
320
  ### Why minimal heads
321
 
322
- The segmentation and classification heads follow the EUPE paper's evaluation principle: a minimal decoder isolates the backbone's contribution from the head's capacity. A Mask2Former-style segmentation head would produce higher mIoU, but those numbers would reflect the decoder as much as the features. The depth and detection heads are heavier because their tasks require multi-scale reasoning. The decomposition costs no trained parameters, so the detection head budget stays small (2.98M) while covering five pyramid levels from stride 8 to stride 128.
323
 
324
- ## Notes and limitations
325
 
326
- - The segmentation head was trained on ADE20K's 150-class indoor-and-urban label space.
327
- - The depth head was trained on NYU Depth V2 (indoor). Outdoor metric depth should be treated as approximate.
328
- - The detection head was trained on COCO 2017's 80-class label space at 768-pixel input. Small-object mAP (22.3) is the weakest axis because the stride-8 P3 level can only resolve objects roughly 10 pixels and larger at that resolution.
329
- - Correspondence has no confidence signal; it returns a target pixel for every source keypoint regardless of match ambiguity.
330
 
331
  ## License
332
 
333
- The EUPE-ViT-B backbone weights inside this checkpoint were released by Meta FAIR under the [FAIR Research License](https://huggingface.co/facebook/EUPE-ViT-B/blob/main/LICENSE), which restricts use to non-commercial research and education. The task heads and class prototypes in this checkpoint were trained independently by the author of this repository and would on their own be releasable under a permissive license. However, because they are inseparably bundled with the backbone weights in a single file, the unified checkpoint inherits the more restrictive license of its most restricted component. In practical terms, both `model.safetensors` and `model.bf16_backbone.safetensors` should be treated as released under the FAIR Research License. See `LICENSE` for the full text.
334
 
335
  ## Citation
336
 
 
 
337
  ```bibtex
338
  @misc{zhu2026eupe,
339
  title={Efficient Universal Perception Encoder},
 
26
 
27
  # Argus
28
 
29
+ Argus is a multi-task perception system built on a single compact vision backbone. From one forward pass through the encoder, the model produces classification labels, semantic segmentation masks, metric depth maps, object detections with bounding boxes, and dense keypoint correspondences, thereby collapsing five domain-specific pipelines into a unified package of roughly 116 million parameters. The system is named after Argus Panoptes, the many-eyed giant of Greek mythology who was tasked by Hera with watching over everything at once.
30
 
31
+ The underlying backbone is [EUPE-ViT-B](https://huggingface.co/facebook/EUPE-ViT-B) (86M parameters), which was introduced in *Efficient Universal Perception Encoder* (Zhu et al., Meta FAIR, [arXiv:2603.22387](https://arxiv.org/abs/2603.22387), March 2026). That paper demonstrates that a small vision encoder can be distilled from a collection of larger specialist teachers, yielding features that transfer well to image understanding, dense prediction, and vision–language tasks simultaneously. Argus takes the released EUPE-ViT-B backbone, leaves its weights frozen, and attaches five lightweight task heads that were trained or constructed independently for this project.
32
 
33
  ## Architecture
34
 
35
  ```
36
+ Image β†’ EUPE-ViT-B (frozen, 86M parameters) β†’ shared features
37
 
38
+ β”œβ”€β”€ Classification β€” trained linear softmax, 1000 ImageNet classes
39
+ β”œβ”€β”€ Segmentation β€” linear head, 150 ADE20K classes
40
+ β”œβ”€β”€ Depth β€” DPT multi-scale decoder, metric depth in meters (NYU Depth V2)
41
+ β”œβ”€β”€ Detection β€” FCOS head with simple feature pyramid, 80 COCO classes
42
+ └── Correspondence β€” training-free dense feature matching
43
  ```
44
 
45
+ - **Classification** β€” trained linear softmax, a single `Linear(768, 1000)` layer with bias applied to the L2-normalized CLS token. 85.53% top-1 and 97.69% top-5 on ImageNet-1k val.
46
+ - **Segmentation** β€” BatchNorm layer followed by a single 1Γ—1 convolution, trained with the backbone held frozen throughout.
47
+ - **Depth** β€” DPT (Dense Prediction Transformer) decoder hooking into four intermediate ViT layers (blocks 2, 5, 8, 11), fusing their features at four spatial scales via residual conv fusion, producing metric depth in meters via a 256-bin weighted sum over the 0.001 to 10 meter range. Improves RMSE by 8% over a linear probe on the same backbone (0.480 vs 0.520 on NYUv2 test), with abs_rel improving by 28%. Attempts to extend the depth head to outdoor scenes via mixed indoor/outdoor training with scale-and-shift invariant loss degraded indoor accuracy without producing a model that worked reliably across both domains; outdoor depth remains a known limitation.
48
+ - **Detection** β€” FCOS-style anchor-free detector on a ViTDet-style simple feature pyramid that synthesizes five spatial scales (strides 8 through 128) from the backbone's stride-16 patch features, with shared four-layer convolutional towers for classification and box regression across all pyramid levels. Runs at 640-pixel input with letterbox padding; returns per-image lists of bounding boxes with class labels, confidence scores, and COCO class names.
49
+ - **Correspondence** β€” no trained parameters. Source and target features are extracted from two images, upsampled to pixel resolution, and matched by cosine similarity at each source keypoint.
 
 
50
 
51
  ## Benchmarks
52
 
53
+ ### Reproduction of the EUPE Paper
54
+
55
+ All four of the paper's reported benchmarks were reproduced as part of building Argus, and the results either matched the published numbers within rounding error or exceeded them modestly.
56
 
57
+ | Task | Dataset | Metric | Paper | Argus | Delta |
58
+ |----------------|--------------|------------------------|-------|--------|---------|
59
+ | Classification | ImageNet-1k | kNN k=10 top-1 | 84.1 | 84.07 | βˆ’0.03 |
60
+ | Segmentation | ADE20K | mean IoU | 52.4 | 52.72 | +0.32 |
61
+ | Depth | NYU Depth v2 | RMSE (lower is better) | 0.391 | 0.3914 | +0.0004 |
62
+ | Correspondence | SPair-71k | PCK@0.1 | 51.3 | 54.35 | +3.05 |
63
 
64
+ The classification evaluation used the full 1.28-million-image ImageNet-1k training set as the kNN reference and the 50,000-image validation set as the query. The segmentation and depth heads were trained using the same linear-probe configurations described in the EUPE repository. Correspondence was evaluated on the SPair-71k test split at 512-pixel resolution across all 12,234 test pairs, for a total of 88,328 keypoints, with no failures during the run.
 
 
 
 
 
65
 
66
+ The classification head reaches 85.53% top-1 and 97.69% top-5 on ImageNet-1k val. The kNN protocol used during the paper reproduction phase served as the baseline; the trained linear head superseded it and is the only classification method shipped in the current checkpoint.
67
 
68
+ | Classification method | Top-1 | Top-5 |
69
+ |-----------------------|----------|----------|
70
+ | kNN (k=10, retired) | 84.07 % | 93.99 % |
71
+ | Linear softmax | 85.53 % | 97.69 % |
 
 
 
72
 
73
+ ### Detection
74
 
75
+ The EUPE paper evaluates its backbone exclusively through minimal decoders β€” linear probes, kNN, and training-free matching β€” so that downstream performance can be attributed to the learned features rather than to the capacity of the head. The paper's three evaluation domains are image understanding, dense prediction, and vision–language modeling; detection is not among them. The same frozen-backbone protocol applies: the FCOS detection head is trained on COCO 2017 train (117,266 images) at 640-pixel input while the backbone weights remain fixed.
76
+
77
+ Evaluation on COCO val2017 (5,000 images) with the standard pycocotools protocol:
78
 
79
  | Metric | Value |
80
+ |--------|-------|
81
+ | mAP@[0.5:0.95] | **41.0** |
82
+ | mAP@0.50 | 64.8 |
83
+ | mAP@0.75 | 43.2 |
84
+ | mAP (small objects) | 21.4 |
85
+ | mAP (medium objects) | 44.9 |
86
+ | mAP (large objects) | 62.1 |
87
+
88
+ FCOS with a fully-trained ResNet-50-FPN backbone β€” all backbone weights learned end-to-end on COCO β€” achieves 39.1 mAP on the same benchmark. The frozen EUPE-ViT-B backbone reaches 41.0 mAP while sharing its features across all five task heads. The backbone was never exposed to detection data; these are the same features that produce the classification, segmentation, depth, and correspondence results above.
89
+
90
+ ### Depth Decoder Comparison (Development Reference)
91
+
92
+ A DPT multi-scale decoder was trained and evaluated alongside the linear probe during development. The DPT decoder hooks into four intermediate ViT layers and fuses their features at multiple spatial scales, producing an 8% RMSE improvement on indoor scenes and is the shipping depth head. Subsequent attempts to extend it to outdoor depth via mixed NYU+KITTI training with hybrid scale-and-shift invariant loss did not produce a model that maintained indoor accuracy while gaining outdoor capability; outdoor depth remains a known limitation.
93
+
94
+ | Decoder | RMSE ↓ | abs_rel ↓ | a1 (Ξ΄<1.25) ↑ |
95
+ |---------|--------|-----------|---------------|
96
+ | Linear probe | 0.520 | 0.303 | 0.860 |
97
+ | DPT multi-scale (current) | **0.480** | **0.219** | **0.872** |
98
+
99
+ ### Segmentation Decoder Comparison (Development Reference)
100
+
101
+ The same DPT multi-scale architecture that improved depth by 8% was also trained and evaluated for semantic segmentation on ADE20K, using the same reassemble + fusion approach with a 150-class classification head and cross-entropy loss. The DPT segmentation decoder did not improve over the linear probe.
102
+
103
+ | Decoder | ADE20K mIoU |
104
+ |---------|------------|
105
+ | Linear probe (current) | **52.72%** |
106
+ | DPT multi-scale | 52.28% |
107
+
108
+ The linear probe stays as the shipping segmentation head. Unlike depth, where multi-scale feature fusion captures spatial gradients that a single-layer head misses, the segmentation task's sharp class boundaries at stride-16 resolution are already as well-predicted by a 1Γ—1 convolution as the frozen features allow. The heavier decoder adds capacity without improving the output.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
109
 
110
  ### Cross-Dataset Segmentation Transfer
111
 
112
+ To test whether the backbone's features generalize beyond the ADE20K distribution on which the segmentation head was trained, a separate linear probe (identical BatchNorm + 1Γ—1 Conv architecture, same training recipe) was trained on the Cityscapes urban driving dataset using the frozen backbone. Cityscapes contains 2,975 training images and 500 validation images of street scenes captured from a vehicle-mounted camera in German cities, annotated with 19 semantic classes. The backbone was never exposed to driving scenes during EUPE's multi-teacher distillation or during any phase of Argus head training.
113
 
114
  | Dataset | Classes | Train images | mIoU |
115
+ |---------|---------|-------------|------|
116
+ | ADE20K (original head) | 150 | 20,210 | 52.72% |
117
+ | Cityscapes (transfer probe) | 19 | 2,975 | 63.76% |
118
 
119
+ The Cityscapes probe reaches 63.76% mIoU, with road at 96.4%, car at 87.9%, sky at 88.8%, building at 86.7%, and vegetation at 85.6%. The weaker categories are thin vertical structures β€” pole at 17.8%, traffic light at 36.4%, traffic sign at 48.3% β€” which is an inherent resolution limitation of the stride-16 patch grid rather than a deficiency in the learned representation. The frozen backbone produces features that transfer to an entirely unseen visual domain with a minimal linear decoder, which is the property that makes a universal perception encoder worth having.
120
 
121
+ ## Comparison with Standard Baselines
122
 
123
  As a sanity check, Argus was compared against several well-known models on the same 200-image COCO subset. The classification comparison uses a keyword cross-reference between each model's top-k ImageNet predictions and the COCO ground-truth detection labels on those images, which provides a consistent yardstick across differently-trained models despite the label-space mismatch. **These hit rates measure agreement with COCO detection labels via keyword matching on the 200-image subset; they are not raw ImageNet accuracy.** For reference, all three classifiers exceed 80% top-1 on the full ImageNet validation set.
124
 
 
126
 
127
  | Model | Parameters | Top-1 hit | Top-5 hit | Latency | Peak VRAM |
128
  |--------------------|------------|-----------|-----------|---------|-----------|
129
+ | Argus (EUPE-ViT-B) | 86 M | 38.5% | 61.0% | 13.1 ms | 0.34 GB |
130
+ | ConvNeXt-Base | 89 M | 25.5% | 48.5% | 10.4 ms | 0.35 GB |
131
+ | ResNet50 | 26 M | 23.0% | 46.0% | 8.4 ms | 0.12 GB |
132
 
133
  **Segmentation**:
134
 
 
144
  | Argus (EUPE + linear head) | 86 M | 13.3 ms | 0.35 GB |
145
  | Depth-Anything-V2-Base | 98 M | 18.8 ms | 0.68 GB |
146
 
147
+ Argus leads both baselines on top-1 and top-5 hit rates. Argus is faster than DeepLabV3 while predicting a much richer label space, and it is faster than Depth-Anything-V2 while using roughly half the VRAM. Although these baselines and Argus were trained for different objectives on different datasets, the comparison is useful for understanding what the model delivers in practice.
148
 
149
  ### Multi-Task Throughput
150
 
151
  The per-task comparisons above measure each head against its single-task counterpart in isolation. A separate question is what happens when a user needs all of the tasks at once, which is the typical situation in dataset annotation, model evaluation, and any pipeline where images pass through multiple analysis stages in sequence. The alternative to Argus in that situation is to load and run four separate single-task models of comparable quality, each carrying its own backbone, its own preprocessing, and its own forward pass. The total cost is the sum of the four individual inference times, plus the memory overhead of holding four independent models on the device simultaneously.
152
 
153
+ The models chosen for this comparison were selected to match the quality tier of the EUPE-ViT-B backbone rather than to minimize size or maximize speed. ConvNeXt-Base (88.6M parameters) is a widely-used ImageNet-1k classifier at the same parameter scale as EUPE-ViT-B. SegFormer-B3 (47.3M) is a transformer-based ADE20K semantic segmenter that is the standard mid-range alternative to a linear probe on a frozen backbone. Depth-Anything-V2-Base (97.5M) is the current standard for single-image monocular depth estimation at base scale. YOLO26l (26.3M) is the large variant of the January 2026 YOLO release from Ultralytics, representing the state of the art in efficient real-time detection. All measurements were taken on the same RTX 6000 Ada GPU across the same nine example images, with five timed runs after a three-image warmup pass to eliminate cold-start effects.
154
 
155
  | Pipeline | Parameters | Latency per image | Tasks |
156
  |----------|-----------|-------------------|-------|
157
+ | Argus unified | 116 M | 56 ms | 5 (classify, segment, depth, detect, correspond) |
158
  | Four separate models | 260 M | 68 ms | 4 (classify, segment, depth, detect) |
159
 
160
+ The per-model breakdown for the separate pipeline is ConvNeXt-Base at 6 ms, SegFormer-B3 at 19 ms, Depth-Anything-V2-Base at 31 ms, and YOLO26l at 12 ms, summing to 68 ms when the tasks are run sequentially on the same image. Argus completes five tasks β€” the same four plus keypoint correspondence, which the separate pipeline does not attempt β€” in 56 ms from a single model load. The total parameter count for the separate pipeline is 260M across four independent weight sets, while Argus carries 116M in a single file.
161
 
162
+ The throughput advantage comes from the shared backbone. Each of the four separate models pays the cost of encoding the image through its own network before producing task-specific output. Argus encodes the image once through EUPE-ViT-B and then routes the resulting features to five lightweight heads, each of which adds only a few milliseconds on top of the shared representation. The backbone forward pass is the dominant cost in both pipelines, and running it once rather than four times is where the 1.2x throughput improvement and 2.2x parameter reduction originate. The practical consequence for deployment is that Argus requires a single model download (447 MB), a single checkpoint load into VRAM (0.53 GB), and a single Python import, where the equivalent separate-model pipeline requires four downloads totaling over a gigabyte, four loads consuming over a gigabyte of VRAM if held concurrently, and four separate dependency trees to manage.
163
 
164
  ## Usage
165
 
 
168
  from transformers import AutoModel
169
 
170
  model = AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)
171
+
172
  image = Image.open("your_image.jpg").convert("RGB")
173
 
174
+ # Any single task can be called directly:
175
+ top5 = model.classify(image, top_k=5) # trained linear softmax, 1000 ImageNet classes
176
+ seg = model.segment(image) # returns [H, W] class indices
177
+ depth = model.depth(image) # returns [H, W] metric depth in meters
 
178
 
179
+ # Detection runs at 640px with letterbox padding:
180
+ dets = model.detect(image, score_thresh=0.3)
181
+ # returns list of {"box": [x1,y1,x2,y2], "score": float, "label": int, "class_name": str}
182
+
183
+ # Classification, segmentation, and depth can be run at once:
184
  result = model.perceive(image)
185
+ # result["classification"] β€” list of top-5 {"class_id", "class_name", "score"}
186
+ # result["segmentation"] β€” numpy array of ADE20K class indices
187
+ # result["depth"] β€” numpy array of depth values in meters
188
+ # result["timings_ms"] β€” per-task latency breakdown
189
+
190
+ # Detection is called separately because it uses a different input resolution:
191
+ dets = model.detect(image)
192
 
193
+ # Dense patch correspondence between two images:
194
  target = Image.open("other_image.jpg").convert("RGB")
195
+ corr = model.correspond(image, target)
196
+ # corr["matches"] β€” argmax target-patch index for each source patch (length grid*grid)
197
+ # corr["scores"] β€” cosine similarity at the match
198
+ # corr["grid"] β€” patch-grid side length (e.g. 32 at resolution=512)
199
  ```
200
 
201
+ Every single-image method also accepts a list of images. When a list is passed, the return type becomes a list of per-image results in the same shape that a single call would produce:
202
+
203
+ ```python
204
+ images = [Image.open(p).convert("RGB") for p in paths]
205
+
206
+ top5_batch = model.classify(images, top_k=5) # list of list-of-dict
207
+ seg_batch = model.segment(images) # list of [H, W] tensors
208
+ depth_batch = model.depth(images) # list of [H, W] tensors
209
+ perceive_batch = model.perceive(images) # list of dicts
210
+ ```
211
 
212
+ Per-task confidence and uncertainty are available as opt-in outputs. Classification always carries a `margin` field (top-1 score minus top-2 score) on the first entry. Segmentation and depth expose confidence maps when `return_confidence=True` is passed:
213
 
214
  ```python
215
  seg_map, seg_conf = model.segment(image, return_confidence=True)
 
219
  # depth_std is per-pixel standard deviation of the 256-bin distribution
220
 
221
  result = model.perceive(image, return_confidence=True)
222
+ # result["segmentation_confidence"] and result["depth_uncertainty"] are populated
223
  ```
224
 
225
+ The model can be exported to ONNX. This produces three separate graphs β€” backbone, segmentation head, and depth head β€” with verification against the PyTorch reference automatically performed when `verify=True`:
 
 
226
 
227
  ```python
228
+ paths = model.export_onnx("/path/to/out_dir", backbone_resolution=224, verify=True)
229
+ # paths["backbone"], paths["seg_head"], paths["depth_head"]
230
+ # paths["verification"] β€” max abs diff per component
231
  ```
232
 
233
+ Classification (kNN over class prototypes) and correspondence run as post-processing on top of the backbone output and need no separate graph.
234
 
235
+ For reduced VRAM on memory-constrained hardware, INT8 weight-only quantization is available via torchao. This quantizes the Linear weight matrices to INT8 while keeping activations in BF16, avoiding the outlier-channel problems that break naive INT8 quantization of ViT models:
 
 
236
 
237
  ```python
238
  model = AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)
239
  model = model.cuda().eval().quantize_int8() # requires: pip install torchao
240
+
241
+ # All methods work identically after quantization:
242
+ result = model.perceive(image) # 100% classification agreement, <0.05 m depth drift
243
+ dets = model.detect(image) # detection counts identical to FP32
244
  ```
245
 
246
+ | Mode | perceive latency | VRAM | Classification | Depth drift |
247
+ |------|-----------------|------|---------------|-------------|
248
+ | BF16 mixed (default) | 35 ms | 0.53 GB | reference | β€” |
249
+ | INT8 weight-only | 39 ms | 0.47 GB | 100% agreement | 0.013 m mean |
250
 
251
+ The model uses HuggingFace's custom-code mechanism (`trust_remote_code=True`),
252
+ so the loader code is fetched from the model repo automatically. No additional
253
+ files need to be cloned.
254
 
255
+ ## Training
256
 
257
+ The backbone is frozen for every task. Only the task heads are trained, and the class prototypes are extracted (not trained at all).
 
 
 
258
 
259
+ ### Heads
260
 
261
+ | Component | Source dataset | Trained by |
262
+ |---------------------------|----------------------------------------------------------|-------------------------------------------------------------|
263
+ | EUPE-ViT-B backbone | LVD-1689M (approximately 1.7 billion web images) | Meta FAIR (used here frozen) |
264
+ | Segmentation head | ADE20K (20,210 training images, 2,000 validation images) | This repository, 40,000 iterations of linear-probe training |
265
+ | Depth head | NYU Depth V2 (24,231 training images) | This repository, 38,400 iterations of linear-probe training |
266
+ | Class prototypes (kNN) | ImageNet-1k (1.28 million training images) | This repository, mean CLS feature per class |
267
+ | Linear softmax classifier | ImageNet-1k (1.28 million training images) | This repository, SGD over cached frozen features |
268
+ | Detection head | COCO 2017 (117,266 training images, 80 classes) | This repository, FCOS with simple FPN, 8 epochs at 640px |
269
+ | Correspondence | None (training-free) | β€” |
270
 
271
+ The trainable heads sum to approximately 30.5M parameters (seg 117K + depth DPT 13.45M + linear classifier 769K + detection 16.14M). The unified `model.safetensors` is 443 MB.
272
 
273
+ ### Precision variants
 
 
 
 
 
 
274
 
275
+ Two safetensors files with the same weights at different on-disk precision. Inference behavior is identical; the smaller file is for users with limited bandwidth or storage.
276
 
277
+ | File | Size | Load |
278
+ |---|---|---|
279
+ | `model.safetensors` | 334 MB | `AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True)` |
280
+ | `model.bf16_backbone.safetensors` | 170 MB | `AutoModel.from_pretrained("phanerozoic/argus", trust_remote_code=True, variant="bf16_backbone")` |
281
 
282
+ Both files load into the same FP32 model in memory; PyTorch automatically upcasts the bfloat16 stored weights at construction time. The smaller variant saves download bandwidth and disk space but does not reduce inference VRAM.
283
 
284
+ ### Architecture details
285
 
286
+ **Segmentation head** is `BatchNorm2d(768) β†’ Conv2d(768, 150, 1Γ—1)` β€” 116,886 parameters, 1.4 MB on disk. Trained at 512Γ—512 with cross-entropy loss, AdamW (lr 1e-3, weight decay 1e-3), WarmupOneCycleLR with 1500-step warmup, batch size 16.
287
 
288
+ **Depth head** is a DPT multi-scale decoder that hooks into backbone blocks [2, 5, 8, 11] via PyTorch forward hooks, capturing intermediate representations without modifying the backbone. A reassemble stage projects each block's output from 768 to 256 channels via LayerNorm + Linear, reshapes to spatial grids, and rescales to four target strides (4, 8, 16, 32) via bilinear interpolation. A bottom-up fusion path combines these four scales through residual conv blocks with skip connections, progressively doubling spatial resolution from stride 32 to stride 2. A final conv head produces 256 depth-bin logits, outputting metric depth in meters via a bin-weighted sum. 13,450,000 parameters, ~51 MB on disk. Trained at 416Γ—416 with SILog loss, AdamW (lr 1e-4, weight decay 1e-3), cosine schedule with 3% warmup, batch size 16, 38,400 iterations.
289
 
290
+ **Class prototypes (kNN path)** are produced by running the frozen backbone over the full ImageNet-1k training set at 224Γ—224 resolution, computing the mean L2-normalized CLS feature per class, and saving the resulting 1000Γ—768 matrix. No training, just feature extraction. At inference, the kNN path normalizes the query CLS token and computes cosine similarity against the prototype matrix.
291
 
292
+ **Linear softmax classifier** is a single `Linear(768, 1000)` layer with bias β€” 769,000 parameters, about 3 MB on disk. Trained as a two-pass job: first the frozen backbone is run over the ImageNet-1k training set to cache a per-image CLS feature tensor (1,281,167 Γ— 768, stored once at ~3.9 GB), then the linear layer is trained on the cached features alone. The training pass uses SGD with momentum 0.9, weight decay 0, batch size 4096, cosine schedule, 100 epochs, no augmentation, and the best checkpoint by validation top-1 is restored at the end. A small learning-rate sweep over `{0.5, 1.0, 3.0, 10.0, 30.0}` selects the best configuration; the L2-normalized CLS features and zero-initialized weights demand an unusually large learning rate to grow the weight scale to the point where softmax distributions become sharp. The best run used lr = 30.0 and produced 85.53% top-1 / 97.69% top-5 on ImageNet-1k val, beating the kNN protocol on both metrics.
293
 
294
+ **Detection head** is an FCOS-style anchor-free detector on a ViTDet-style simple feature pyramid. The FPN takes the backbone's stride-16 spatial features and synthesizes five levels: P3 (stride 8, via transposed convolution), P4 (stride 16, identity with channel reduction), P5 (stride 32), P6 (stride 64), and P7 (stride 128), each with 256 channels and GroupNorm normalization. Two shared four-layer convolutional towers (classification and regression) with GroupNorm and GELU process each level, followed by three prediction heads: 80 classification channels, 4 box regression channels (left/top/right/bottom distances, exponentiated with learned per-level scale), and 1 centerness channel. 16,138,074 parameters total, 61.6 MB on disk. Trained at 640Γ—640 with letterbox padding, focal loss (alpha 0.25, gamma 2.0) for classification, GIoU loss for boxes, BCE for centerness, AdamW (lr 1e-3, weight decay 1e-4), cosine schedule with 3% warmup, batch size 64, 8 epochs.
295
+
296
+ **Correspondence** has no learned parameters. At inference time, dense patch features are extracted from both images, upsampled to 512Γ—512 pixel resolution, and matched by cosine similarity per source keypoint.
297
 
298
  ### Compute
299
 
300
+ | Task | Iterations | Wall time |
301
+ |------------------------------|---------------------------|-------------|
302
+ | Segmentation (ADE20K) | 40,000 | ~5 hours |
303
+ | Depth (NYU Depth V2) | 38,400 | ~3 hours |
304
+ | Class prototypes (IN1k) | 1.28M images, single pass | ~45 minutes |
305
+ | Linear classifier (IN1k) | 100 epochs Γ— 313 steps | ~25 seconds (on cached features, extraction amortized with the kNN prototype pass) |
306
+ | Detection (COCO 2017) | 8 epochs Γ— 1,832 batches | ~6 hours at batch 64, 640px, FP32, frozen backbone |
307
+ | DPT depth decoder (NYUv2) | 38,400 iterations | ~5.3 hours at batch 16, 416px, SILog loss, frozen backbone |
308
+ | Correspondence (SPair) | training-free | β€” |
309
+
310
+ Training was done on a single 48 GB workstation GPU. Peak VRAM was approximately 10 GB during segmentation training, 7 GB during depth training, 2.5 GB during class prototype extraction, and 4 GB during the linear classifier training (once features are cached, the training loop only holds the 3.7 GB cached feature tensor on GPU).
311
 
312
  ### Why minimal heads
313
 
314
+ The segmentation and classification heads follow the EUPE paper's evaluation principle: a minimal decoder isolates the backbone's contribution from the head's capacity. A Mask2Former-style segmentation head would produce higher mIoU, but those numbers would reflect the decoder as much as the features. The depth and detection heads are heavier. The DPT decoder fuses features from four intermediate ViT layers at multiple spatial scales; the FCOS head synthesizes a five-level feature pyramid from the backbone's stride-16 output. Depth requires multi-scale fusion to capture spatial gradients across a scene, and detection requires a feature pyramid to resolve objects that range from a dozen pixels to the full image. In both cases the backbone remains frozen and only the head is trained.
315
 
316
+ ## Notes
317
 
318
+ The segmentation head was trained on ADE20K's 150-class indoor-and-urban label space. The depth head was trained on NYU Depth v2 and is indoor-biased; outdoor metric depth should be treated as approximate. The detection head was trained on COCO 2017's 80-class label space at 640-pixel input; small-object detection (mAP 21.4) is the expected weakness because the stride-8 P3 level can only resolve objects roughly 12 pixels and larger at that resolution. Classification uses a trained linear softmax classifier that produces calibrated probabilities and reaches 85.53% top-1 on ImageNet-1k val.
 
 
 
319
 
320
  ## License
321
 
322
+ The EUPE-ViT-B backbone weights inside this checkpoint were released by Meta FAIR under the [FAIR Research License](https://huggingface.co/facebook/EUPE-ViT-B/blob/main/LICENSE), which restricts use to non-commercial research and education. The task heads and class prototypes in this checkpoint were trained independently by the author of this repository and would on their own be releasable under a permissive license. However, because they are inseparably bundled with the backbone weights in a single file, the unified checkpoint inherits the more restrictive license of its most restricted component. In practical terms, the entire `argus.pt` file should be treated as released under the FAIR Research License. See `LICENSE` for the full text.
323
 
324
  ## Citation
325
 
326
+ If you use Argus or the underlying EUPE backbone in academic work, please cite the original paper:
327
+
328
  ```bibtex
329
  @misc{zhu2026eupe,
330
  title={Efficient Universal Perception Encoder},