---
library_name: pytorch
pipeline_tag: image-feature-extraction
tags:
  - geo-localization
  - visual-place-recognition
  - thermal
  - cross-modal-retrieval
  - swin-transformer
  - netvlad
license: other   # TODO: set after checking the licences listed under "Licence and data terms"
---

# SAMF (Swin-T + DSEM + MARM + NetVLAD) trained on Boson-nighttime

A retrieval model that matches a **thermal UAV image** to the **satellite tile** showing the same place. Both images are turned into one
8192-dimensional descriptor; the nearest satellite descriptor to the thermal descriptor is the predicted location.

This is a reproduction of the SAMF model ([paper](https://arxiv.org/abs/2610.12835), [code](https://github.com/YuxinWang879/SAMF)) trained with the
authors' released recipe, **not an official release**. This repository holds **only the weights** (`samf_final_best.pth`). The model code comes from the SAMF code repository.

## How to use it (weights only)

**1. Get the model code and the packages.** The weights have no code, so clone the SAMF repository next to your script (Python 3.10+):

```bash
git clone https://github.com/YuxinWang879/SAMF.git
pip install torch torchvision timm==1.0.22 numpy h5py einops faiss-cpu scikit-learn pillow huggingface_hub transformers
```

SAMF pins torch 2.9.1, torchvision 0.24.1 and timm 1.0.22; other versions may work but are untested.

**2. Load the weights and embed images.** Copy this into a script or notebook:

```python
import sys
import torch
import torch.nn.functional as F
import torchvision.transforms as T
import torchvision.transforms.functional as TF
from PIL import Image
from huggingface_hub import hf_hub_download

SAMF_ROOT = "./SAMF"                       # the cloned code repository
device = "cuda" if torch.cuda.is_available() else "cpu"

# --- build the model with the paper settings (they are SAMF's defaults) ---
sys.path.insert(0, SAMF_ROOT)
import parser as samf_parser               # SAMF's own parser.py
from model import network

sys.argv = [sys.argv[0], "--datasets_folder", ".", "--device", device]   # the folder is not used for inference
args = samf_parser.parse_arguments()
args.pretrain = "none"                     # the weights come from the checkpoint, no ImageNet download
model = network.GeoLocalizationNet(args)

# --- load the weights (the file is a pickle: only load checkpoints from a source you trust) ---
path = hf_hub_download("ElectroPython/samf", "samf_final_best.pth")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
state = {(k[7:] if k.startswith("module.") else k): v for k, v in ckpt["model_state_dict"].items()}
model.load_state_dict(state)
model = model.to(device).eval()

# --- preprocessing, exactly as SAMF's dataset code does it ---
norm = T.Compose([T.ToTensor(), T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

def prep(img, thermal):
    img = img.convert("RGB")
    if thermal:
        img = TF.adjust_contrast(img, 3)   # contrast factor 3 for thermal queries only
    return TF.resize(norm(img), [256, 256])

@torch.no_grad()
def embed(images, thermal, batch_size=16):
    """list of PIL images -> (N, 8192) L2-normalised descriptors. thermal=True for the query, False for satellite tiles."""
    out = []
    for i in range(0, len(images), batch_size):
        batch = images[i:i + batch_size]
        x = torch.stack([prep(im, thermal) for im in batch]).to(device)
        modality = torch.full((len(batch),), 1 if thermal else 0, dtype=torch.long, device=device)   # 1 = thermal, 0 = satellite
        out.append(F.normalize(model(x, modality=modality).float(), dim=1).cpu())
    return torch.cat(out)
```

**3. Rank satellite tiles for a thermal image:**

```python
tiles = [Image.open(p) for p in tile_paths]            # your satellite tiles
q = embed([Image.open("thermal.png")], thermal=True)   # (1, 8192)
t = embed(tiles, thermal=False)                        # (N, 8192): compute once per map and keep it
scores, idx = (q @ t.T).topk(5, dim=1)                 # cosine similarity (descriptors are L2-normalised)
print([tile_paths[i] for i in idx[0]])
```

Things that matter: images are read as RGB (a grayscale thermal image is replicated to 3 channels); inputs are resized to **256 x 256**;
and the modality id must be **1 for thermal, 0 for satellite**, since using the wrong one gives worse descriptors.

## Results

Dataset: Boson-nighttime (Xiao et al., STGL), splits of 10,256 / 13,011 / 26,568 query-tile pairs (train / val / test). A prediction counts as correct if the retrieved tile
is within 50 m of the query. Search is over the whole map of the split (no position prior).

| Split | R@1 | R@5 | R@10 | R@20 |
|---|---|---|---|---|
| validation (epoch 28, the saved checkpoint) | 96.0 | 98.3 | 98.8 | 99.3 |
| **test** | **89.3** | **94.8** | **96.5** | **97.8** |

The SAMF paper reports 93.6% test R@1. **This reproduction is 4.3 points lower**, and I have not found out why (seed, recipe details and the resumes during training
are untested possibilities). The validation set is easier than the test set: its database is half the size.

## Training

* Backbone Swin-T (timm) from ImageNet weights, fine-tuned end to end (SAMF's "freeze stages" code is a no-op, so nothing is frozen); DSEM and MARM modules; NetVLAD with 64 clusters.
* Triplet margin loss (margin 0.1), 10 hard negatives per query, Adam, learning rate 3e-5, 4 queries per batch, 256 x 256 input, 30 epochs of 10,000 queries.
* The checkpoint is the epoch with the best validation R@5 (epoch 28). Training ran on a single A100 and was **resumed several times** after runtime disconnects
  (model and optimizer state were restored each time; the data order after a resume differs from an uninterrupted run).

## Intended use and limitations

* Intended for research on thermal-to-satellite geo-localization.
* **Domain:** thermal tiles at about 1 m/px (512 m footprint), mostly desert at night, matched to Bing satellite imagery. Different sensors, altitudes, seasons or satellite sources
  are untested and may degrade a lot.
* It is a **retrieval** model: the answer is the centre of the best tile, with a tile-spacing resolution. It does not give a confidence or a "no match" decision.
* One training run, one seed: differences of about a point between runs are not meaningful.

## Licence and data terms

* The Boson-nighttime dataset card says it is "provided solely for non-commercial research and educational purposes" and is based on Bing satellite imagery
  (Bing Maps Print Rights); its metadata also lists "mit". Weights trained on it should be treated accordingly. **TODO: confirm and set the licence above.**
* Check the SAMF code repository's licence before redistributing anything derived from it. **TODO.**

## Citation

* SAMF: https://arxiv.org/abs/2610.12835 (code: https://github.com/YuxinWang879/SAMF)
* Dataset and baseline: Xiao, Tortei, Roura, Loianno, "Long-range UAV Thermal Geo-localization with Satellite Imagery", https://arxiv.org/abs/2306.02994