README.md

---
license: apache-2.0
language:
- en
tags:
- OneScience
- earth-science
- remote-sensing-representation-learning
- multiscale-remote-sensing
- multispectral-remote-sensing
- convolutional-upsampling
frameworks: PyTorch
datasets:
  - FMoW-RGB
  - FMoW-Sentinel
  - EuroSAT
  - UCMerced
  - RESISC-45
---

<p align="center">
  <strong>
    <span style="font-size: 30px;">SatMAE++</span>
  </strong>
</p>

# Model Introduction

SatMAE++ is a masked autoencoder for optical and multispectral satellite imagery. It uses a visible-token encoder to learn remote sensing representations and reconstructs native spatial-scale targets through a convolutional multiscale decoder. The model supports RGB and grouped Sentinel inputs, with a training objective composed of MSE+L1 for masked patches and MSE+L1 for multiscale reconstruction.

Paper: Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery  
https://arxiv.org/abs/2403.05419

# Model Description

SatMAE++ was proposed by the team of Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwar, Salman Khan, and Fahad Shahbaz Khan at Mohamed bin Zayed University of Artificial Intelligence in Abu Dhabi. The model is trained using FMoW-RGB and FMoW-Sentinel imagery. The model is suitable for representation learning and multiscale reconstruction tasks involving optical and multispectral satellite imagery.

# Applicable Scenarios

| Scenario | Description |
| :---: | :--- |
| Optical satellite imagery pre-training | RGB input using 1x and 2x native targets. |
| Multispectral satellite imagery pre-training | Grouped Sentinel input using 1x, 2x, and 4x native targets. |
| Remote sensing scene classification | Transfer and fine-tune model representations for scene-classification datasets such as EuroSAT, UCMerced, and RESISC-45. |
| Multi-label land-cover classification | Fine-tune on multispectral datasets such as BigEarthNet to identify multiple land-cover types within the same region. |
| Local quick validation | Use synthetic data to complete the training, inference, evaluation, and visualization workflow. |
| Multi-GPU training | Launch DDP training through `torchrun`. |

# Usage Instructions

## 1. OneCode Usage

Experience intelligent one-click AI4S programming through the OneCode online environment:

[Experience intelligent AI4S programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)

## 2. Manual Installation and Usage

**Hardware Requirements**

- GPU or DCU execution is recommended.
- CPU can be used for small-configuration connectivity validation.
- DCU users need to install DTK in advance. DTK 25.04.2 or later, or the OneScience-recommended version matching the current cluster, is recommended.

### Download the Model Package

```bash
hf download OneScience-Group/SatMAE-PP --local-dir ./SatMAE++
cd SatMAE++
```

### Install the Runtime Environment

**DCU Environment**

```bash
# Activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# Supports installation with uv
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

**GPU Environment**

```bash
# Activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# Supports installation with uv
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

### Training Data Introduction

The paper uses FMoW-RGB and FMoW-Sentinel for pre-training. In NPZ files, `images` is `[B,C,H,W]`, with native `images_2x` or `images_4x` targets consistent with the configured scales. RGB uses three channels, while Sentinel uses ten channels encoded by spectral groups.

Synthetic data is used by default:

```bash
python scripts/fake_data.py
```

Run the command above when using synthetic data. When using real data, do not run `fake_data.py`; organize the data as `data/train.npz` and `data/test.npz`, where RGB data contains at least:

```text
images:    float32 [N,3,H,W]
images_2x: float32 [N,3,2H,2W]
```

Sentinel data uses ten channels and provides `images_2x` and `images_4x` according to the configuration:

```text
images:    float32 [N,10,H,W]
images_2x: float32 [N,10,2H,2W]
images_4x: float32 [N,10,4H,4W]
```

The high-resolution fields should be native targets registered with the input scene and should not be replaced by temporary interpolation of low-resolution imagery. After preparing the data, modify `conf/config.yaml` according to the actual number of channels, input size, spectral grouping, and training scales.

### Training

```bash
python scripts/train.py
```

Multiple GPUs:

```bash
torchrun --nproc_per_node=8 scripts/train.py
```

Training outputs:

```text
result/checkpoints/satmae_pp.pt
result/training/metrics.json
```

Training outputs include a model checkpoint that can be used for subsequent inference and metric records reflecting the training process and loss convergence, facilitating training-state preservation and analysis of model optimization.

The default configuration is a quick synthetic-data protocol. The formal paper configuration for RGB is ViT-L, `224/16`, and 800 epochs; the Sentinel configuration is ViT-L, `96/8`, 10-channel three-group input after removing B1/B9/B10, and 50 epochs.

### Trained Weights

This repository provides weights trained on FMoW-RGB optical satellite imagery and FMoW-Sentinel multispectral satellite imagery in the `weight/` folder. The weight files will be uploaded soon and are expected to be completed in the near future.

### Inference

```bash
python scripts/inference.py
```

Inference results are output to:

```text
result/output/reconstruction.npz
```

### Evaluation and Visualization

```bash
python scripts/result.py
```

Evaluation and visualization outputs are saved to:

```text
result/evaluation/metrics.json
result/evaluation/multiscale_reconstruction.png
result/evaluation/scale_comparison.png
```

Evaluation results measure model performance in terms of overall reconstruction error, masked-region recovery quality, and reconstruction error at different native scales, and intuitively show the model's multiscale recovery capability through multiscale reconstruction results and cross-scale error comparisons.

# OneScience Official Information

| Platform | OneScience Main Repository | Skills Repository |
| --- | --- | --- |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |

# Citation and License

This repository is a reproduction of the original SatMAE++ paper.