| --- |
| library_name: pytorch |
| pipeline_tag: image-feature-extraction |
| tags: |
| - geo-localization |
| - visual-place-recognition |
| - thermal |
| - cross-modal-retrieval |
| - swin-transformer |
| - netvlad |
| license: other |
| --- |
| |
| # SAMF (Swin-T + DSEM + MARM + NetVLAD) trained on Boson-nighttime |
|
|
| A retrieval model that matches a **thermal UAV image** to the **satellite tile** showing the same place. Both images are turned into one |
| 8192-dimensional descriptor; the nearest satellite descriptor to the thermal descriptor is the predicted location. |
|
|
| This is a reproduction of the SAMF model ([paper](https://arxiv.org/abs/2610.12835), [code](https://github.com/YuxinWang879/SAMF)) trained with the |
| authors' released recipe, **not an official release**. This repository holds **only the weights** (`samf_final_best.pth`). The model code comes from the SAMF code repository. |
|
|
| ## How to use it (weights only) |
|
|
| **1. Get the model code and the packages.** The weights have no code, so clone the SAMF repository next to your script (Python 3.10+): |
|
|
| ```bash |
| git clone https://github.com/YuxinWang879/SAMF.git |
| pip install torch torchvision timm==1.0.22 numpy h5py einops faiss-cpu scikit-learn pillow huggingface_hub transformers |
| ``` |
|
|
| SAMF pins torch 2.9.1, torchvision 0.24.1 and timm 1.0.22; other versions may work but are untested. |
|
|
| **2. Load the weights and embed images.** Copy this into a script or notebook: |
|
|
| ```python |
| import sys |
| import torch |
| import torch.nn.functional as F |
| import torchvision.transforms as T |
| import torchvision.transforms.functional as TF |
| from PIL import Image |
| from huggingface_hub import hf_hub_download |
| |
| SAMF_ROOT = "./SAMF" # the cloned code repository |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| |
| # --- build the model with the paper settings (they are SAMF's defaults) --- |
| sys.path.insert(0, SAMF_ROOT) |
| import parser as samf_parser # SAMF's own parser.py |
| from model import network |
| |
| sys.argv = [sys.argv[0], "--datasets_folder", ".", "--device", device] # the folder is not used for inference |
| args = samf_parser.parse_arguments() |
| args.pretrain = "none" # the weights come from the checkpoint, no ImageNet download |
| model = network.GeoLocalizationNet(args) |
| |
| # --- load the weights (the file is a pickle: only load checkpoints from a source you trust) --- |
| path = hf_hub_download("ElectroPython/samf", "samf_final_best.pth") |
| ckpt = torch.load(path, map_location="cpu", weights_only=False) |
| state = {(k[7:] if k.startswith("module.") else k): v for k, v in ckpt["model_state_dict"].items()} |
| model.load_state_dict(state) |
| model = model.to(device).eval() |
| |
| # --- preprocessing, exactly as SAMF's dataset code does it --- |
| norm = T.Compose([T.ToTensor(), T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]) |
| |
| def prep(img, thermal): |
| img = img.convert("RGB") |
| if thermal: |
| img = TF.adjust_contrast(img, 3) # contrast factor 3 for thermal queries only |
| return TF.resize(norm(img), [256, 256]) |
| |
| @torch.no_grad() |
| def embed(images, thermal, batch_size=16): |
| """list of PIL images -> (N, 8192) L2-normalised descriptors. thermal=True for the query, False for satellite tiles.""" |
| out = [] |
| for i in range(0, len(images), batch_size): |
| batch = images[i:i + batch_size] |
| x = torch.stack([prep(im, thermal) for im in batch]).to(device) |
| modality = torch.full((len(batch),), 1 if thermal else 0, dtype=torch.long, device=device) # 1 = thermal, 0 = satellite |
| out.append(F.normalize(model(x, modality=modality).float(), dim=1).cpu()) |
| return torch.cat(out) |
| ``` |
|
|
| **3. Rank satellite tiles for a thermal image:** |
|
|
| ```python |
| tiles = [Image.open(p) for p in tile_paths] # your satellite tiles |
| q = embed([Image.open("thermal.png")], thermal=True) # (1, 8192) |
| t = embed(tiles, thermal=False) # (N, 8192): compute once per map and keep it |
| scores, idx = (q @ t.T).topk(5, dim=1) # cosine similarity (descriptors are L2-normalised) |
| print([tile_paths[i] for i in idx[0]]) |
| ``` |
|
|
| Things that matter: images are read as RGB (a grayscale thermal image is replicated to 3 channels); inputs are resized to **256 x 256**; |
| and the modality id must be **1 for thermal, 0 for satellite**, since using the wrong one gives worse descriptors. |
|
|
| ## Results |
|
|
| Dataset: Boson-nighttime (Xiao et al., STGL), splits of 10,256 / 13,011 / 26,568 query-tile pairs (train / val / test). A prediction counts as correct if the retrieved tile |
| is within 50 m of the query. Search is over the whole map of the split (no position prior). |
|
|
| | Split | R@1 | R@5 | R@10 | R@20 | |
| |---|---|---|---|---| |
| | validation (epoch 28, the saved checkpoint) | 96.0 | 98.3 | 98.8 | 99.3 | |
| | **test** | **89.3** | **94.8** | **96.5** | **97.8** | |
|
|
| The SAMF paper reports 93.6% test R@1. **This reproduction is 4.3 points lower**, and I have not found out why (seed, recipe details and the resumes during training |
| are untested possibilities). The validation set is easier than the test set: its database is half the size. |
|
|
| ## Training |
|
|
| * Backbone Swin-T (timm) from ImageNet weights, fine-tuned end to end (SAMF's "freeze stages" code is a no-op, so nothing is frozen); DSEM and MARM modules; NetVLAD with 64 clusters. |
| * Triplet margin loss (margin 0.1), 10 hard negatives per query, Adam, learning rate 3e-5, 4 queries per batch, 256 x 256 input, 30 epochs of 10,000 queries. |
| * The checkpoint is the epoch with the best validation R@5 (epoch 28). Training ran on a single A100 and was **resumed several times** after runtime disconnects |
| (model and optimizer state were restored each time; the data order after a resume differs from an uninterrupted run). |
|
|
| ## Intended use and limitations |
|
|
| * Intended for research on thermal-to-satellite geo-localization. |
| * **Domain:** thermal tiles at about 1 m/px (512 m footprint), mostly desert at night, matched to Bing satellite imagery. Different sensors, altitudes, seasons or satellite sources |
| are untested and may degrade a lot. |
| * It is a **retrieval** model: the answer is the centre of the best tile, with a tile-spacing resolution. It does not give a confidence or a "no match" decision. |
| * One training run, one seed: differences of about a point between runs are not meaningful. |
|
|
| ## Licence and data terms |
|
|
| * The Boson-nighttime dataset card says it is "provided solely for non-commercial research and educational purposes" and is based on Bing satellite imagery |
| (Bing Maps Print Rights); its metadata also lists "mit". Weights trained on it should be treated accordingly. **TODO: confirm and set the licence above.** |
| * Check the SAMF code repository's licence before redistributing anything derived from it. **TODO.** |
|
|
| ## Citation |
|
|
| * SAMF: https://arxiv.org/abs/2610.12835 (code: https://github.com/YuxinWang879/SAMF) |
| * Dataset and baseline: Xiao, Tortei, Roura, Loianno, "Long-range UAV Thermal Geo-localization with Satellite Imagery", https://arxiv.org/abs/2306.02994 |
|
|
