README.md

---
license: cc-by-nc-4.0
language:
  - en
pipeline_tag: image-feature-extraction
tags:
  - image-retrieval
  - landmark-retrieval
  - place-recognition
  - visual-localization
  - dinov2
  - frozen-backbone
  - robotics
base_model: facebook/dinov2-large
---

# fusion-perception-1 (v0.1-preview)

<p align="center">
<img src="assets/fusion-perception-banner.png" alt="Fusion Perception — place recognition, Eximius Labs" width="100%">
</p>

A landmark and place retrieval descriptor: a frozen **DINOv2 ViT-L/14** backbone (Apache-2.0)
with a small trained projection head (3.1M parameters, the only weights in this repository).
One image in, one 512-d unit vector out. Rank a gallery by cosine similarity.

![Fusion Perception architecture](assets/fusion_perception_overview.png)

Five queries against the full evaluation gallery, single-stage cosine search with no reranking.
The first four are the shipped head working; the last is a query it gets wrong.

![Retrieval against 1,007,323 photographs](assets/fusion_perception_retrieval_demo.gif)

The intended use is re-localization and place matching at scale, the case a semantic search
embedding handles poorly: same building, different viewpoint, different season, different time
of day, among a million unrelated photographs.

**Headline.** With AMES reranking of a top-1600 shortlist, this system sets the best
**RParis+1M** numbers we are aware of, on Medium and on Hard, under both of its training
protocols. On **ROxford+1M** it is close to the frontier but not at it. Both statements are
laid out below, and so is the reason to discount part of the result.

## Why not a semantic search embedding

A multimodal search embedding already produces one vector per image, so the first question is
whether a separate descriptor is needed at all. We measured it. Our own text, image, video and
audio model, [fusion-embedding-2](https://huggingface.co/EximiusLabs/fusion-embedding-2-2b-preview)
(revision `1720d8b1`), was scored on both benchmarks with the same evaluator, the same official
ground-truth files, the same bounding-box-cropped queries and the same galleries. Its image path
is the frozen Qwen3-VL-Embedding-2B vision-language model, called through that model's released
`embed_image` API. Global descriptors only, no reranking, no distractors.

| System | Descriptor | ROxf M | ROxf H | RPar M | RPar H |
| --- | --- | --- | --- | --- | --- |
| fusion-embedding-2 image tower | 1024-d | 47.52 | 18.69 | 78.42 | 57.06 |
| Frozen DINOv2 ViT-L/14 CLS, three scales, no training | 1024-d | 72.94 | 53.75 | 92.58 | 84.02 |
| Fusion Perception v0.1, standard head | 512-d | 76.51 | 58.09 | 92.85 | 84.88 |

The search embedding is behind on all four cells: 28.99 mAP-M and 39.40 mAP-H on ROxford, 14.43
and 27.82 on RParis, against the head in this repository. It is also behind the untrained DINOv2
readout by 25.42 / 35.06 and 14.16 / 26.96, so the difference is not something the head training
created. The gap is widest on Hard, the setting whose positives are the difficult views of a
landmark, taken from another viewpoint or under occlusion or different conditions, which is the
case re-localization actually runs into.

This is a statement about one task. The search embedding is built for cross-modal retrieval and
is not trained for instance-level matching, and nothing here measures the tasks it is built for.

Read-out note: twelve documented configurations of the search embedding were measured, crossing
the document instruction on both sides against the query instruction on the query side, the
1024-d default against the full 2048-d rung, and gallery mean-centering on against off. The row
above is its strongest configuration with protocol-cropped queries. Its weakest cropped
configuration scores 38.20 M on ROxford and 70.18 M on RParis. The only cell where any
configuration beats the row above is ROxford Hard under an uncropped-query control, which is
outside the protocol and reaches 22.71, still 35.38 below the head. All twelve are recorded in
`results.json`.

## Results with +1M distractors

mAP in percent under the revisited Oxford / Paris protocol, with the standard 1,001,001-image
`revisitop1m` distractor set. Two-stage rows all rerank a **top-1600** shortlist; shortlist size
is the single largest confound in this table, so rows reranking 400 or 800 candidates are left
out rather than compared against.

| System | Stage | ROxf+1M M | ROxf+1M H | RPar+1M M | RPar+1M H |
| --- | --- | --- | --- | --- | --- |
| RN101-SuperGlobal [1, Tab. 1] | global only | 78.8 | 61.9 | 83.9 | 69.1 |
| Fusion Perception v0.1, standard | global only | 67.09 | 45.38 | 86.65 | 73.88 |
| Fusion Perception v0.1, decon | global only | 65.62 | 41.50 | 87.02 | 74.23 |
| RN101-SuperGlobal, rerank top-1600 [1, Supp. Tab. 6] | two stage | 85.9 | 74.3 | 87.7 | 77.0 |
| AMES, CVNet locals, after SG rerank [2, Supp. Tab. 9] | two stage | 86.6 | 74.1 | 88.8 | 79.9 |
| AMES, DINOv2 locals, no SG rerank [2, Supp. Tab. 9] | two stage | 87.1 | 76.1 | 89.8 | 81.0 |
| AMES, DINOv2 locals, after SG rerank [2, Supp. Tab. 9] | two stage | 88.2 | 77.7 | 90.1 | 82.0 |
| L2G CANN-FIRE + MDS rerank [3, Tab. 1] | two stage | 90.5 | 79.8 | 92.1 | 83.4 |
| **Fusion Perception v0.1, standard, + AMES** | two stage | 87.21 | 75.14 | **92.61** | **86.03** |
| **Fusion Perception v0.1, decon, + AMES** | two stage | 87.11 | 73.73 | **92.92** | **86.35** |

Our rows are one training run each. The seed spread measured for this recipe is given under
[Evaluation integrity](#1-training-set-overlap-with-the-benchmarks); on RParis the shipped
standard head is the weakest of four runs, so these rows are not a favourable draw.

**RParis+1M.** Both heads pass the best published numbers on both difficulty settings, by
+0.51 M and +2.63 H for the standard head and +0.82 M and +2.95 H for the decontaminated head,
measured against L2G. The two heads are not separable from each other at this sample size, so
the result to take from the pair is that it holds whether or not the overlapping training
classes are removed.

**ROxford+1M.** Medium is level with AMES on DINOv2 locals without SuperGlobal reranking
(87.21 and 87.11 against 87.1) and 3.3 to 3.4 points below L2G. Hard is 1.0 to 2.4 points below
that same AMES row and 4.7 to 6.1 points below L2G. ROxford is not a result we lead on.

**What the reranker contributes.** The global-only rows are in the table so the split is
visible. On ROxford+1M, reranking a top-1600 shortlist adds 20.1 mAP-M over our own first-stage
ranking; on RParis+1M it adds 6.0. The asymmetry has a first-stage cause: our global descriptor
scores 67.09 M on ROxford+1M against RN101-SuperGlobal's 78.8, while on RParis+1M it scores
86.65 against their 83.9. The reranker starts from a much worse shortlist on ROxford and from a
better one on RParis.

Sources: [1] SuperGlobal, ICCV 2023, [arXiv 2308.06954](https://arxiv.org/abs/2308.06954),
Table 1 and supplementary Table 6. [2] AMES, ECCV 2024,
[arXiv 2408.03282](https://arxiv.org/abs/2408.03282), supplementary Table 9. [3] L2G, ICCV 2025
Workshop (MRR), [arXiv 2509.04351](https://arxiv.org/abs/2509.04351), Table 1.

One citation subtlety worth stating: the AMES paper's main Table 1 reports only the collapsed
ROP+1M average for these configurations. The four per-dataset +1M cells quoted above are in
**supplementary Table 9**, not the main table. L2G's Table 1 reproduces the AMES best row and
agrees with it.

## Evaluation integrity

Two contamination channels affect these numbers. Both are stated here rather than in an
appendix, because a reader deciding whether to trust the table needs both.

### 1. Training-set overlap with the benchmarks

We matched GLDv2-clean category names against the ROxford and RParis query landmarks and
excluded **87 classes containing 2,529 images** from the decontaminated run. We then checked
that match set at image level rather than asserting it, and it does not hold up as a
contamination measurement.

Embedding every flagged image and all 140 bounding-box-cropped benchmark queries with the
frozen DINOv2-L descriptor, and calibrating the decision threshold on the benchmarks' own
ground truth (1% false-positive rate against database images of other landmarks, 69%
true-positive rate against known same-landmark positives), **43% of the flagged images
actually depict a landmark the benchmarks query**. A visual audit of 170 randomly sampled
flagged images, judged against the query crops, gives the same answer: 43% plus or minus 8%,
agreeing with the automatic verdict at Cohen's kappa 0.84. At class level, 13 of the 87
classes are a queried landmark, 23 if structures inside the same complex are counted.

Precision varies sharply by rule. The `unique:<landmark>` rules reach 58% at image level. The
`blanket:oxford` rule, which covers half the set, reaches 18%: it excludes Oxford Street and
Oxford Circus in London, the Selfridges Oxford Street store, an Oxford College in Taiwan and
villages across Oxfordshire. The `ctx:paris+notre-dame` rules reach 8%: thirteen of their
fifteen classes are Paris parish churches named Notre-Dame that the benchmarks never query.

The set is also not a superset of the published cleanup. RGLDv2-clean (Song et al., CVPR 2024)
removes 1,565 images from 18 landmark ids; our 87 classes contain **10 of those 18** and 1,216
of those 1,565 images. The eight we miss include Basilique du Sacre-Coeur de Montmartre, whose
Commons category spells Coeur with the oe ligature and so slipped past a rule that flagged five
other Sacre-Coeur basilicas. Running the same calibrated detector over all of GLDv2-clean finds
995 images at a queried landmark; the 87 classes hold 757 of them, a **recall of 76%**.

The decon head is therefore a conservative ablation, not a cleaner benchmark protocol: it
removes the overlapping classes it does catch plus roughly 1,400 images unrelated to the
benchmarks, so the decon-versus-standard gap bounds the cost of removing overlapping training
data from above. All 87 classes and their matching rules remain listed in
[`leak_class_ids.json`](leak_class_ids.json). The detectors behind every number above, their
calibrations and the per-image scores are in [`audit/`](audit): `rgld_detector.json` for the
DINOv2 detector, `clip_detector.json` for the CLIP replication, `leak_raw_scores.json` for the
per-image similarities against all 140 query crops, and `rgld_removal_set_diff.json` for the
RGLDv2-clean removal set reconstructed from the authors' released CSV.

We therefore ship two heads:

- **standard** trains on all of GLDv2-clean (1,570,970 images, 76,563 classes after the
  minimum-three-images-per-class filter). This is the same training data every published system
  in the table above uses, so it is the row to compare against them.
- **decon** removes those 87 classes first (1,568,445 images, 76,478 classes).

The measured effect is smaller than the run-to-run variation of the training itself, so we do
not claim a direction for it. Retraining both heads with three seeds each, varying nothing but
the seed, gives these no-distractor global-only means with sample standard deviations over the
three runs:

| cell | standard | decon | delta |
| --- | --- | --- | --- |
| ROxford M | 76.07 ± 0.90 | 76.41 ± 0.20 | -0.34 |
| ROxford H | 56.68 ± 1.16 | 56.08 ± 0.73 | +0.59 |
| RParis M | 93.20 ± 0.25 | 93.28 ± 0.03 | -0.08 |
| RParis H | 85.71 ± 0.51 | 86.12 ± 0.18 | -0.42 |

Three of the four differences are smaller than the seed spread within a condition, and the
fourth is 1.1 pooled standard deviations, which an exact permutation test at three runs per
condition cannot separate from zero. Two of them reverse sign relative to the single-run
numbers the shipped heads produced. Across the six runs ROxford Medium and RParis Medium are
negatively correlated (r = -0.74), which is expected because the epoch is selected on ROxford,
so the apparent trade between the two datasets in a single-run comparison is largely that
selection effect.

What this supports is a bound rather than a null result: removing the 87 overlapping classes
does not change these cells by more than roughly 1 mAP in either direction. It does not explain
the RParis result, which sits 16 points above the ROxford figure.

The shipped heads are one run each and were trained without a fixed seed, so the spread above
is the error bar to read every single-run number in this card with. On RParis the shipped
standard head is the weakest of the four runs measured (92.85 M against 93.47, 93.15 and 92.97;
84.88 H against 86.27, 85.56 and 85.29), so the reranked RParis rows this release leads with
come from an unfavourable draw rather than a fortunate one. On ROxford Hard the shipped head is
the strongest of the four (58.09 against 55.47, 56.78 and 57.78) and should be read as such.
Per-seed results are in [`audit/seed_analysis_n6.json`](audit/seed_analysis_n6.json), which carries both the three-seed statistics quoted here and a six-seed extension of them.

### 2. The backbone saw the benchmarks during pretraining

DINOv2's pretraining corpus LVD-142M contains GLDv2-clean as is (1,580,470 images) plus
6,321,880 images sampled from it, and it contains 1,000,000 images retrieved using the
**Revisiting Oxford** base set as seeds and another 1,000,000 retrieved using the
**Revisiting Paris** base set (DINOv2, arXiv 2304.07193, Table 15). The backbone is frozen here,
so this is not something our training introduced, but it is exposure our descriptor benefits
from and it cannot be removed without changing the backbone.

The AMES authors declined DINOv2 as their default global descriptor for exactly this reason:

> We do not consider DINOv2 as the default option since ROxford and RParis are listed among
> the datasets used for its self-supervised training.
>
> — AMES, arXiv 2408.03282, section 4

**This creates an asymmetry in the table above, and it runs against us.** Our system uses
DINOv2 for the global descriptor and DINOv2 local descriptors for reranking, so the caveat
applies to it in full. It also applies to the two AMES rows built on DINOv2 locals, including
their strongest +1M row. It does **not** apply to L2G, which uses FIRE local features and no
DINOv2, nor to SuperGlobal, nor to the AMES row built on CVNet locals. The strongest published
row we exceed on RParis+1M is therefore a row that carries less pretraining exposure than ours
does. A reader who discounts DINOv2-based results should discount ours first.

## Results without distractors

Same protocol, database only, for readers who want the comparison at the original benchmark
scale.

| System | Stage | ROxf M | ROxf H | RPar M | RPar H |
| --- | --- | --- | --- | --- | --- |
| Fusion Perception v0.1, standard | global only | 76.51 | 58.09 | 92.85 | 84.88 |
| Fusion Perception v0.1, decon | global only | 76.42 | 56.18 | 93.22 | 85.37 |
| RN101-SuperGlobal, rerank top-1600 [1, Supp. Tab. 6] | two stage | 91.2 | 80.6 | 94.2 | 88.4 |
| AMES, DINOv2 locals, after SG rerank [2, Supp. Tab. 9] | two stage | 93.6 | 84.8 | 95.3 | 90.7 |
| L2G CANN-FIRE + MDS rerank [3, Tab. 1] | two stage | 92.9 | 83.0 | 97.1 | 91.7 |
| **Fusion Perception v0.1, standard, + AMES** | two stage | 91.00 | 80.49 | 95.53 | 91.43 |
| **Fusion Perception v0.1, decon, + AMES** | two stage | 90.89 | 79.55 | 95.53 | 91.35 |

The pattern is the same and less pronounced: RParis competitive with the best published rows,
ROxford several points behind them. The margin on RParis widens once the 1M distractors are
added, which is the useful direction for a system meant to run against a large index.

## Architecture

```
  image
    ├─ scale 1.000  short side 224 px ─┐
    ├─ scale 1.414  short side 322 px ─┤  frozen DINOv2 ViT-L/14 (Apache-2.0)
    └─ scale 2.000  short side 448 px ─┘  CLS token, L2-normalized per scale
                                       │
                          mean of 3, re-normalized -> 1024-d
                                       │
       Linear(1024, 2048) -> GELU -> Linear(2048, 512) -> BatchNorm -> L2
                                       │
                              512-d retrieval descriptor
```

Aspect ratio is preserved; each side is snapped to a multiple of 14 and the long side is capped
at 1022 px. The head is trained with ArcFace (margin 0.3, scale 32), label smoothing 0.1,
class-balanced square-root sampling, AdamW with a cosine schedule, batch 4096, 40 epochs with
early stopping. Model selection is on ROxford only; RParis was held out until the final table.
The backbone is never fine-tuned.

Reranking is optional. The two-stage rows use AMES (Suma et al., ECCV 2024) with the authors'
public checkpoint and their published DINOv2-B local descriptors, unchanged, over a top-1600
shortlist, combined as `0.55 * global_cosine + 0.45 * sigmoid(0.3 * ames_logit)`. That is the
AMES paper default. Our contribution to those rows is the first-stage ranking.

`rerank.py` in this repository runs that second stage, so the two-stage cells above are
reproducible from what ships here. The AMES code is Apache-2.0 and is fetched with `torch.hub`;
their checkpoint and their local descriptors download from the authors' host on first use. None
of it is redistributed here, and the local descriptors are DINOv2-B features, so the pretraining
caveat above applies to the two-stage rows as well.

## Usage

<details>
<summary>Requirements</summary>

`torch`, `transformers>=4.46`, `safetensors`, `pillow`, `numpy`, and `huggingface_hub` if loading by repo id.
The DINOv2 backbone downloads from Meta's repository under Apache-2.0. Runs on CPU; float16 on
CUDA is the precision every reported number was produced at.
</details>

```python
from PIL import Image
from inference import FusionPerceptionRetrieval

fp = FusionPerceptionRetrieval.from_pretrained("EximiusLabs/fusion-perception-1-preview")

gallery = fp.embed([Image.open(p) for p in paths])   # [N, 512], L2-normalized
query   = fp.embed(Image.open("query.jpg"))          # [512]

scores, idx = fp.search(query, gallery, topk=10)
```

Select the decontaminated head with `protocol="decon"`. Pass `bbox=[(x1, y1, x2, y2), ...]` to
crop before embedding, which is what the revisitop query protocol does.

Which head to use: **standard** if you are comparing against published numbers, **decon** if
your application is anywhere near Oxford or Paris landmarks and you want the training set to
have no knowledge of them. For everything else the two are close enough that the choice does not
matter much.

### Two stage

```bash
# once per dataset: pull the AMES authors' local descriptors, 5.5 GB here, 7 GB for RParis
python rerank.py --dataset roxford5k --ames-dir ./ames_assets --fetch-only

# rerank the top-1600 shortlist and score Medium and Hard
python rerank.py --dataset roxford5k --ames-dir ./ames_assets \
    --images-root /data/roxford5k/jpg --head standard --out roxford5k_standard.json
```

Defaults are the reported cell: `--topk 1600 --lambdas 0.55 --temps 0.3`. Pass `--descriptors`
instead of `--images-root` to reuse descriptors you already have, and `--distractor-locals` with
`--distractor-desc` for the +1M setting. The script needs `h5py` and a GPU in addition to the
requirements above.

## Reproduction

`REPRODUCE.md` in this repository lists the scripts, the evaluation harness, the exact
two-stage commands, where the distractor features come from, and the compute and storage the
full run required (roughly $100 of GPU time and 1.08 TB of local-descriptor storage). The
no-distractor two-stage rows in the table above were reproduced with `rerank.py` and match to
the last reported digit; the figures and the one deviation worth knowing about are in
`REPRODUCE.md`.

## Scope and limitations

- This release ships the **retrieval descriptor only**. Dense segmentation heads and a projector
  into the Fusion Embedding shared space exist in the same line of work and are not included
  here.
- Research preview. Evaluated on ROxford and RParis, which are landmark benchmarks: performance
  on indoor scenes, repetitive industrial environments, or low-texture surfaces is not
  characterized by these numbers.
- The strong result is specific to RParis+1M. This is not a general state-of-the-art claim.
- Reranking is what closes most of the gap to published two-stage systems, and it costs memory
  for stored local descriptors. The global-only rows are the operating point with no extra
  storage cost, and they are several points weaker.
- The open backbone is DINOv2 (Apache-2.0). A DINOv3-based variant is stronger on some tasks but
  is licensed in a way that does not permit an open-weights release.

## License

Released for research under
[CC-BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/). The DINOv2 backbone is
Apache-2.0 and is downloaded from Meta rather than re-hosted here. The AMES code
([pavelsuma/ames](https://github.com/pavelsuma/ames)) is Apache-2.0 and is fetched from its
repository; the AMES checkpoint and local descriptors used for the two-stage rows belong to
their authors, are downloaded from the authors' host, and are not redistributed. `rerank.py`
calls their code rather than reimplementing it. Cite AMES for any use of the two-stage numbers.

## Citation

```bibtex
@misc{fusionperception2026,
  title  = {Fusion Perception: a frozen-backbone retrieval descriptor for place recognition},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  note   = {Eximius Labs. DINOv2 backbone (Apache-2.0); head CC-BY-NC-4.0.},
  url    = {https://huggingface.co/EximiusLabs/fusion-perception-1-preview}
}
```

Backbone: **DINOv2** (Oquab et al., 2023). Benchmarks: **Revisited Oxford and Paris**
(Radenović et al., CVPR 2018) with the `revisitop1m` distractor set. Reranking: **AMES**
(Suma et al., ECCV 2024).