Spectral Coverage

Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images

arXiv HuggingFace License Python 3.10 PyTorch 2.6.0 Website

πŸš€ Introduction

Spectral Coverage

From optical sensors to microwave radars, leveraging the complementary strengths of remote sensing (RS) sensors is crucial for achieving dense spatio-temporal monitoring of our planet, but recent foundation models (FMs) are often specific to single sensors or to fixed combinations.

SMARTIES is a generic and versatile FM lifting sensor-dependent efforts and enabling scalability and generalization to diverse RS sensors: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space, enabling the use of arbitrary combinations of bands both for training and inference. To obtain sensor-agnostic representations, SMARTIES was trained as a single, unified transformer model reconstructing masked multi-sensor data with cross-sensor token mixup, while modulating its feature representations to accept diverse sensors as input.

✨ Key Features

  • πŸ›°οΈ Multi-Sensor Representations: SMARTIES enables sensor-agnostic processing of Earth observation data, including optical (e.g., Sentinel-2), radar (e.g., Sentinel-1), and sub-meter resolution RGB (e.g., Maxar) imagery and unseen ones in a zero-shot manner.
  • 🌈 Spectrum-Aware Projections: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space: given a specific sensor, each one of its bands is projected by projection layers specific to wavelength ranges.
  • ⚑ Lightweight and Scalable: SMARTIES is designed to be lightweight and scalable, making it suitable for a wide range of remote sensing applications.
  • πŸ”€ Flexible Band Combinations: SMARTIES can handle arbitrary combinations of spectral bands from different sensors, enabling flexible remote sensing applications.
  • πŸ”„ Downstream Transfer: SMARTIES enables downstream transfer using a unified model across a diverse set of sensors and tasks, including scene classification, semantic segmentation, and multi-label classification.

SMARTIES Model Architecture

This repository contains the model weights of SMARTIES (ViT-L).

🧩 Using SMARTIES

SMARTIES is designed to be flexible and can be easily adapted to new datasets and sensors. You can easily use SMARTIES with a single line of code with Hugging Face transformer interface:

model = transformers.AutoModel.from_pretrained(
    "gsumbul/SMARTIES-v1-ViT-L",
    trust_remote_code=True
)

A Jupyter notebook is provided in SMARTIES GitHub page to show in detail how to use pretrained model weights. The details of SMARTIES are described in our paper, available on arXiv.

πŸ“£ Attribution

If you use SMARTIES, please cite the paper:

@article{smarties,
  title={{SMARTIES}: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images},
  author={Gencer Sumbul and Chang Xu and Emanuele Dalsasso and Devis Tuia},
  journal={arXiv preprint arXiv:2506.19585},
  year={2025}
}

πŸ“„ License

This repository is released under the Apache v2 License.

πŸ™ Acknowledgements

SMARTIES is supported by the European Space Agency (ESA) through the Discovery and Preparation Program, and is part of the project Toward a Foundation Model for Multi-Sensor Earth Observation Data with Language Semantics.

Downloads last month
25
Safetensors
Model size
0.3B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including gsumbul/SMARTIES-v1-ViT-L

Paper for gsumbul/SMARTIES-v1-ViT-L