Semantically Retrieved Imagery
System idle
Upload an image or enter a text query to trigger cross-modal alignment.
SatFetch Performance Benchmarks
Quantitative evaluation on the EuroSAT cross-modal validation subset (3,000 paired image channels)
Same-Modal vs Cross-Modal Accuracy (Recall@5)
Average Query Latency (ms)
Model Comparison Matrix
| Model Strategy | Same R@1 | Same R@5 | Same R@10 | Cross R@1 | Cross R@5 | Cross R@10 | Latency |
|---|
Domain Projection Simulator
Tweak calibration weight ($\alpha$) and noise level ($\sigma$).
0.0 (Baseline)
0.75
1.0 (Full Centering)
0.00 (Pure)
0.10
0.50 (Noisy)
Recall@1
48.2%
Recall@5
64.5%
MAP@10
55.1%
Latency
31 ms
Core Pipeline Architecture & ZS-MC Workflow
Systematic data flow showing zero-shot domain-centering alignment and multi-sensor retrieval.
Zero-Shot Modality Centering (ZS-MC)
Satellite sensors operate in vastly different electromagnetic domains. An optical sensor captures visual reflectance (3 bands), whereas Synthetic Aperture Radar (SAR) measures microwave backscatter (2 bands). This results in a massive spectral domain gap when projected into joint CLIP embedding space.
To align the spaces without training parameters, ZS-MC computes the centroids of the source domain \(\mu_{src}\) and target domain \(\mu_{tgt}\) from the EuroSAT calibration split: \[\mu_{mod} = \frac{1}{N_{mod}}\sum_{i=1}^{N_{mod}} z_i\] The query vector \(z_0\) is calibrated by translating the centroid: \[z_c = z_0 - \mu_{src} + \mu_{tgt}\] This centers the query vector directly in the target representation space, correcting domain drift and restoring matching accuracy.
Hybrid Spatial-Spectral Indexing (H3)
Searching millions of satellite image tiles globally requires spatial constraint. Filtering search files by simple bounding box queries leads to rectangular boundary overlaps and slow database performance.
SatFetch integrates Uber's **H3 Hierarchical Hexagonal Index** to solve this. Hexagonal grids are optimal because all adjacent cells are equidistant, which simplifies radial distance lookups: 1. Each image tile coordinate (Latitude, Longitude) is resolved to a unique H3 Cell Index at resolution level 7 (cell edge length ~1.22km). 2. During query execution, the search center is resolved to its H3 index, and a ring lookup finds adjacent cell indices within distance \(R\). 3. The query is executed ONLY against database records belonging to these H3 hexagons, reducing candidate vector counts by 99.4% before running FAISS matrix multiplication.
SatFetch System Limitations & Edge Cases
1. Electromagnetic Centroid Calibration Drift ($\mu_{mod}$): ZS-MC aligns optical and radar domains via static centroid translation. Dynamic parameters like variable soil moisture (modifying radar dielectric properties) or seasonal canopy vegetation changes cause local drift, reducing cross-modal alignment precision in out-of-distribution scenes.
2. H3 Grid Edge Boundary Dropouts: Queries close to coordinate boundary vertices of an H3 cell can fail to retrieve adjacent cell images unless the ring lookup distance ($R$) is explicitly set to $\ge 1$ cell radius. Higher resolutions improve query speed but increase neighbor lookup latency.
3. Frozen Text Embeddings OOV Limits: The text encoder relies on frozen OpenAI CLIP weights. Highly specialized technical geological terms (e.g., specific rock lithology classifications or rare cloud types) exhibit weaker alignment scores compared to standard Earth land-cover labels.
4. Signal Attenuation & Cloud Masking: Heavy cloud cover blocks visual sensors completely. While SAR acts as a cloud-penetrating sensor, retrieving optical images from a cloudy query tile is physically restricted unless pre-processed cloud-masking layers are applied.
Team Flight Crew
Developed by Team 4MISTAKES • Rajiv Gandhi Institute of Petroleum Technology (An Institute of National Importance)
