REPRODUCE.md

Reproducing the numbers

Everything in results.json comes from one pipeline. This file records what to run, what the evaluation harness is, where each input comes from, and what the full run costs.

Evaluation harness

compute_map from filipradenovic/revisitop (python/evaluate.py), vendored verbatim. A stage-0 audit scored the vendored copy against the file fetched byte-for-byte from that repository and recomputed mAP for three descriptor sets on both datasets: maximum absolute difference 0.0. The audit also verified that ROxford and RParis queries are absent from their own galleries (70 queries, 4,993 and 6,322 database images), that ground-truth bounding boxes are present, and that queries are cropped to those boxes before embedding while gallery images are not. The crop check is a recompute: the cached query descriptor matches a crop-then-embed at cosine 1.0 and diverges from an uncropped embed at cosine 0.49 to 0.97 depending on how much of the frame the box covers.

Medium uses ok = easy + hard, junk = junk. Hard uses ok = hard, junk = junk + easy.

Stages

All stages run as Modal functions against the fusion-data volume. Scripts live in the fusion-embeddings repository under scripts/.

Stage	Script	What it produces
0. Protocol audit	fp_phase3a_audit.py	fp_phase3a/audit_verdict.json
1a. GLDv2 ingest plan and image streaming	fp_phase3a_data.py --action {plan-full,stream,delta}	fp_phase3a/plan/plan_full.json, plan/leak_class_ids.json, packed image shards
1b. Feature extraction	fp_phase2_extract.py	fp_phase2/gld_feats/feats_*.pt (float16 CLS at three scales)
2. Head training, both protocols	fp_phase3a_retrain.py --action train --protocol {standard,decon}	fp_phase3a/full_head_{protocol}.pt, .json, eval descriptors, first-stage nn_*.pkl
2b. Seed spread for stage 2	fp_phase3a_seeds.py (3 seeds per protocol)	fp_phase3a/seeds/seed_results.json, seed_analysis.json, per-run heads
3. AMES reranking, no distractors	fp_phase3a_ames.py --action ours	fp_phase3a/ames_nn_full_{protocol}_dinov2_ames.json
4. Distractor extraction	fp_phase3a_r1m.py --action pipeline	fp_phase3a/r1m/locals_*.hdf5, cls_*.pt
5. Distractor finalize	fp_phase3a_r1m.py --action finalize	fp_phase3a/r1m/r1m_order.json, r1m_desc_{protocol}.pt
6. +1M evaluation	fp_phase3a_r1m.py --action eval1m --protocol {standard,decon}	fp_phase3a/r1m_eval_{protocol}_dinov2_ames.json
7. Semantic-embedding comparison	fp_fe2_placerec.py --action {extract,score}	fp_fe2_placerec/fe2_{roxford,rparis}.pt, fp_fe2_placerec/results.json

Feature extraction (stages 1 and 4)

facebook/dinov2-large, frozen, float16. For each of three scales (1.0, 1.414, 2.0) the short side is set to round(224 * scale / 14) * 14, the long side follows the aspect ratio, both are snapped to a multiple of 14, and the long side is clamped to 1022. Bicubic resize, ImageNet mean and standard deviation. The CLS token is L2-normalized per scale, the three are averaged, and the result is re-normalized. inference.py in this repository implements exactly this path and reproduces the cached evaluation descriptors to cosine 0.9996 or better.

Head training (stage 2)

Linear(1024, 2048) -> GELU -> Linear(2048, 512, bias=False) -> BatchNorm1d(512), ArcFace with margin 0.3 and scale 32, label smoothing 0.1, class-balanced sampling with weight 1/sqrt(class_count), AdamW at learning rate 1e-3 and weight decay 5e-4, cosine schedule, batch 4096, 40 epochs, early stop after 8 epochs without improvement. Classes with fewer than three images are dropped. Checkpoint selection is on ROxford Medium per epoch; RParis is not consulted until the final table. The decon protocol additionally masks out the 87 classes listed in fp_phase3a/plan/leak_class_ids.json before the class filter runs.

Best epoch was 12 for standard and 0 for decon.

Reranking (stages 3 and 6)

AMES is used as published. The image is built from a shallow clone of pavelsuma/ames; the local descriptors (dinov2_gallery_local.hdf5, dinov2_query_local.hdf5) and the dinov2_ames.pt checkpoint are downloaded from the authors' host. The only substitution is the first-stage ranking file nn_*.pkl, which is generated from our global descriptors.

Before trusting the transplant, fp_phase3a_ames.py --action repro runs the authors' own nn_superglobal.pkl through the same harness and lands at ROxford 92.70 / 83.75 and RParis 95.26 / 90.66 without distractors, against their published 92.4 +/- 0.9 / 83.1 +/- 1.1 and 95.2 +/- 0.1 / 90.2 +/- 0.4 (supplementary Table 9, three seeds). That is the harness check.

The shortlist is junk-aware, matching the AMES and DELG evaluation code: ground-truth junk ids are moved behind the shortlist before reranking. Every published two-stage row in the comparison does the same.

The reported cell is k1600_l0.55_t0.3, the AMES paper default: top-1600 shortlist, score = 0.55 * global_cosine + 0.45 * sigmoid(0.3 * ames_logit). A 15-cell lambda and temperature grid was also measured. Those cells are diagnostics, they are not reported as results, and they are not in results.json. For reference, the best grid cell exceeds the paper-default cell by between 0.07 and 1.90 mAP depending on the dataset, difficulty and protocol, which is the size of the effect avoided by fixing the cell in advance.

Two-stage reproduction from this repository

rerank.py in this repository is the second stage as a standalone script, so the two-stage rows can be reproduced without the Modal pipeline. It builds the junk-aware shortlist from our global descriptors, reranks it with the authors' AMES model over their local descriptors, fuses the two scores and reports Medium and Hard mAP with the same vendored compute_map. It is the logic of stage 3 and stage 6 in one file; the Modal scripts remain the record of how the published artifacts were produced.

Third-party assets

None of the following is redistributed with this model.

Asset	Where it comes from	Terms
AMES code	github.com/pavelsuma/ames, fetched by torch.hub on first use, or --ames-repo <clone>	Apache-2.0
dinov2_ames.pt	downloaded by the authors' model class from ptak.felk.cvut.cz/personal/sumapave/public/ames/networks/, cached under TORCH_HOME	authors' host, no separate license stated
dinov2_query_local.hdf5, dinov2_gallery_local.hdf5, per dataset	ptak.felk.cvut.cz/personal/sumapave/public/ames/data/<dataset>/, downloaded by --fetch	same
gnd_<dataset>.pkl	revisited Oxford / Paris release, mirrored on the same host	github.com/filipradenovic/revisitop

Cite AMES (Suma et al., ECCV 2024, arXiv 2408.03282) for any use of the two-stage numbers. The local descriptors are DINOv2-B features, which is why the pretraining caveat in README_hf.md applies to the two-stage rows as well as the global-only ones.

Without distractors

One-time asset download. The gallery local descriptors are 5.44 GB for ROxford and 6.89 GB for RParis, plus 0.08 GB of query descriptors each:

python rerank.py --dataset roxford5k --ames-dir ./ames_assets --fetch-only
python rerank.py --dataset rparis6k  --ames-dir ./ames_assets --fetch-only

--fetch-only writes to a .tmp name and renames, so an interrupted download is never mistaken for a complete one, but it does not resume. For a resumable download use wget -c against the same host, which is also what the AMES README suggests.

Then, per dataset, computing our descriptors from the benchmark images with inference.py:

python rerank.py --dataset roxford5k --ames-dir ./ames_assets \
    --images-root /data/roxford5k/jpg --head standard \
    --save-descriptors roxford5k_standard_desc.pt --out roxford5k_standard.json

python rerank.py --dataset rparis6k --ames-dir ./ames_assets \
    --images-root /data/rparis6k/jpg --head standard \
    --save-descriptors rparis6k_standard_desc.pt --out rparis6k_standard.json

--images-root is the flat jpg directory of the benchmark; file names come from the ground-truth pickle. Queries are cropped to the ground-truth box, gallery images are not. --descriptors <file>.pt skips the embedding pass on a rerun, and --head decon selects the decontaminated head. Defaults are --topk 1600 --lambdas 0.55 --temps 0.3, the reported cell; the arguments take comma-separated lists if you want the diagnostic grid.

Each dataset is one pass of 70 x 1600 query-candidate pairs per difficulty setting, and Medium and Hard are separate passes because their junk sets differ. Measured on an A10G: 12.4 minutes per pass, so about 25 minutes per dataset and just under an hour for both, a little over a dollar, plus roughly 10 minutes per dataset if the descriptors are being computed from images rather than supplied.

What the verification run produced

rerank.py was run against both benchmarks with the standard head and the default cell. Two descriptor sources were tried, because they answer different questions.

Feeding in the descriptor set the reported cell was computed from, which is the shipped head weights applied to the cached multi-scale CLS features in float32:

Source	ROxf M	ROxf H	RPar M	RPar H
results.json	91.00	80.49	95.53	91.43
rerank.py, supplied descriptors	91.00	80.49	95.53	91.43

Every cell matches. The global-only rows match too, at 76.51 / 58.09 and 92.85 / 84.88. The stored artifact holds 0.80488 for ROxford Hard and 0.91429 for RParis Hard where rerank.py reports 0.8049 and 0.9143, because the two paths round at different points: the Modal stage used the AMES metric wrapper, which rounds to three decimals of a percent, while rerank.py rounds the fraction to four places. Nothing else differs.

Recomputing the descriptors from the JPEGs with --images-root, which is the path a user without our cached features takes, on ROxford:

Source	ROxf M global	ROxf H global	ROxf M two stage	ROxf H two stage
results.json	76.51	58.09	91.00	80.49
rerank.py, --images-root	76.64	58.26	91.01	80.50

The first stage moves by 0.13 and 0.17 mAP and the two-stage cells by 0.01. This is the descriptor-level difference already recorded above: inference.py reproduces the cached evaluation descriptors at cosine 0.9996 or better, with the residual concentrated in the bounding-box-cropped queries, and 0.9996 on 70 queries is worth about a tenth of a mAP point. The reranker absorbs almost all of it, because a shortlist of 1600 out of 4993 is insensitive to a reordering that small. Expect a tenth of a point either way from this path; the exact figures come from supplying the descriptors. This second run covers ROxford only, since it is the tighter of the two cells and one dataset is enough to size the effect.

With the +1M distractors

--distractor-locals and --distractor-desc extend the gallery. Both are required together:

python rerank.py --dataset roxford5k --ames-dir ./ames_assets \
    --descriptors roxford5k_standard_desc.pt --head standard \
    --distractor-locals /data/r1m --distractor-desc /data/r1m/r1m_desc_standard.pt \
    --out roxford5k_standard_1m.json

--distractor-locals is a directory holding r1m_order.json and the locals_XXXX.hdf5 shards it names; --distractor-desc is {"desc": [1001001, 512]} in exactly that order. The gallery becomes [benchmark database ; distractor shards] and the shortlist is built over all of it. Both files come from the pipeline above:

fp_phase3a_r1m.py --action pipeline    # -> r1m/locals_XXXX.hdf5 and cls_XXXX.pt
fp_phase3a_r1m.py --action finalize    # -> r1m/r1m_order.json and r1m_desc_{protocol}.pt

The extraction, not the reranking, is what makes this expensive: about 1.08 TB of float16 local descriptors, roughly $100 of GPU time, and the whole set has to be resident because the shards are opened lazily across the full gallery. The cost table below is the measured projection. Nothing about the +1M setting changes the reranker or the reported cell; it only changes what is in the gallery.

What was verified through rerank.py is the no-distractor setting, above. The +1M cells in results.json stand on the fp_phase3a_r1m.py --action eval1m run recorded in r1m_eval_{protocol}_dinov2_ames.json; rerank.py is the same shortlist, model, fusion and scorer with a longer gallery, and it has not been re-run at +1M, because that run is the expensive one. A user who has the shards can run it. The gallery alignment is checked before the model loads, so a mismatch between the number of distractor descriptors and the rows in the shards fails immediately rather than producing a wrong number.

Distractor set (stages 4 to 6)

revisitop1m from http://ptak.felk.cvut.cz/revisitop/revisitop1m, 100 gzipped archives. Each archive is downloaded to container-local scratch, unpacked, run through (a) the AMES authors' extract_descriptors.py verbatim to produce 700 local descriptors per image, stored as float16 HDF5, and (b) our DINOv2-L three-scale CLS extraction. The raw JPEGs are deleted immediately after. The step is resumable, skips archives whose two outputs both exist, and is capped at 10 concurrent containers.

revisitop1m contains a small number of truncated JPEGs. ImageFile.LOAD_TRUNCATED_IMAGES is enabled so those decode with padding instead of raising, which keeps our CLS rows aligned with the AMES local-descriptor rows.

finalize concatenates the per-archive CLS shards in canonical archive order, writes r1m_order.json (100 shards, 1,001,001 images total), and applies each head to produce r1m_desc_{protocol}.pt. eval1m builds the gallery as [benchmark database ; R1M shards] in that order, scores global-only mAP, then reranks the top-1600.

Before the full run, fp_phase3a_r1m.py --action valcheck reproduces the authors' local extraction on the first 50 ROxford gallery images and cosine-compares against their published HDF5, and --action measure times one archive and writes a cost projection to fp_phase3a/r1m_projection.json. The projection had a hard pause threshold of $200.

Semantic-embedding comparison (stage 7)

EximiusLabs/fusion-embedding-2-2b-preview at revision 1720d8b16af578d794d7b21ee7b829281941899d, bfloat16 on an A10G. Every vector comes from that model's own released image path: the base chat template with the system instruction, the <|vision_start|><|image_pad|><|vision_end|> user turn, last-token pooling, Matryoshka prefix truncation and L2 normalization. The extractor is checked against the released embed_image API on sampled images from each dataset before extraction starts and aborts if the maximum absolute difference exceeds 1e-6; measured 1.5e-8. Image geometry is whatever the base processor does natively, aspect ratio preserved, no fixed square crop and no multi-scale averaging, which is the documented path for that model. Gallery images carry the document instruction and are never cropped; queries are cropped to the ground-truth box, exactly as in every other row of the table.

Twelve read-outs are scored per dataset from the stored 2048-d pooled vectors, at no additional GPU cost: document instruction on both sides against query instruction on the query side, 1024-d against 2048-d, mean-centering on against off. The reported row is the strongest configuration with protocol-cropped queries, selected on the sum of Medium mAP over both datasets, and the full grid stays in the artifact. Extraction is 5,206 and 6,535 forward passes and took 37 and 31 minutes on one A10G, under $2.

Cost and storage

From fp_phase3a/r1m_projection.json, measured on one archive of 510 images and extrapolated:

Item	Measured rate	Projected for 1M
Download and unpack	11.7 MB/s	11.5 h serial, ~$12 CPU across parallel containers
AMES local extraction	4.4 img/s on A10G	63.1 A10G-hours, ~$69
Our global extraction	18.0 img/s on A10G	15.4 A10G-hours, ~$17
Reranking		~$2
Total		~$100

Storage is the part that surprises people: the float16 local descriptors for 1M images occupy 1.083 TB across the 100 shards, measured on the volume. All of it has to be resident at once, because the evaluation opens shards lazily across the whole gallery.

Everything before the distractor run (audit, ingest, feature extraction, head training, no-distractor reranking, preflight) is estimated at $80 for this round and $110 cumulative including earlier phases, in fp_phase3a/final_report.json. Adding the +1M run puts the project near $200 all in. Treat all of these as estimates; the authoritative number is the Modal dashboard.

Artifact index

Every value in results.json traces to one of these.

Artifact	Contains
fusion-data:/fp_phase3a/r1m_eval_standard_dinov2_ames.json	+1M global-only and reranked, standard head
fusion-data:/fp_phase3a/r1m_eval_decon_dinov2_ames.json	+1M global-only and reranked, decon head
fusion-data:/fp_phase3a/ames_nn_full_standard_dinov2_ames.json	no-distractor reranked grid, standard head
fusion-data:/fp_phase3a/ames_nn_full_decon_dinov2_ames.json	no-distractor reranked grid, decon head
fusion-data:/fp_phase3a/full_head_standard.json	no-distractor global-only, training rows and classes
fusion-data:/fp_phase3a/full_head_decon.json	same, decon head
fusion-data:/fp_phase3a/full_head_{protocol}.pt	the shipped head weights
fusion-data:/fp_phase3a/r1m/r1m_order.json	distractor gallery composition, 1,001,001 images
fusion-data:/fp_phase3a/plan/plan_full.json	GLDv2 audit counts: 87 leak classes, 2,529 leak images
fusion-data:/fp_phase3a/plan/leak_class_ids.json	the 87 class ids
fusion-data:/fp_phase2/plan/leakage_report.json	per-class match records: landmark id, Wikimedia category, matching rule
leak_class_ids.json (in this package)	the 87 classes as published, merged from the three artifacts above
fusion-data:/fp_phase3a/audit_verdict.json	stage-0 protocol audit
fusion-data:/fp_phase3a/r1m_projection.json	measured rates and cost projection
fusion-data:/fp_fe2_placerec/results.json	the fusion-embedding-2 image-tower comparison, every read-out variant measured
fusion-data:/fp_fe2_placerec/fe2_{roxford,rparis}.pt	the raw fusion-embedding-2 gallery and query vectors those numbers are scored from