README.md

---
license: cc-by-nc-4.0
language:
- en
tags:
- OneScience
- earth-science
- remote-sensing-representation-learning
- satellite-imagery
- temporal-remote-sensing
- multispectral-remote-sensing
frameworks: PyTorch
datasets: []
---

<p align="center">
  <strong>
    <span style="font-size: 30px;">SatMAE</span>
  </strong>
</p>

# Model Introduction

SatMAE is a masked autoencoder pre-training model for temporal and multispectral satellite imagery. It learns remote sensing image representations through temporal positional encoding, spectral group encoding, and independent masking across temporal or spectral dimensions, and is mainly used to improve performance on tasks such as satellite image classification, land-cover classification, multi-label classification, and semantic segmentation when labeled data is limited.

Paper: SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery  
https://arxiv.org/abs/2207.08051

# Model Description

SatMAE was proposed by a research team at Stanford University. The model is pre-trained using fMoW satellite imagery, temporal satellite imagery, and Sentinel-2 multispectral imagery. The model is suitable for remote sensing tasks such as satellite image classification, land-cover classification, multi-label classification, and semantic segmentation.

# Applicable Scenarios

| Scenario | Description |
| :---: | :--- |
| Temporal satellite image pre-training | Train SatMAE using timestamped `BTCHW` data. |
| Multispectral satellite image pre-training | Train SatMAE using `BCHW` data and a spectral grouping configuration. |
| Local quick validation | Use synthetic data to check data loading, training, inference, and evaluation. |
| ModelScope/OneCode operation | Run scripts after downloading it as a standalone model package. |
| Multi-GPU training | Launch multi-process training through `torchrun`. |

# Usage Instructions

## 1. OneCode Usage

Experience intelligent one-click AI4S programming through the OneCode online environment:

[Experience intelligent one-click AI4S programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)

## 2. Manual Installation and Usage

**Hardware Requirements**

- GPU or DCU execution is recommended.
- CPU can be used for imports and small-configuration connectivity validation, but full training and inference are slow.
- DCU users need to install DTK in advance. DTK 25.04.2 or later, or the OneScience-recommended version matching the current cluster, is recommended.

### Download the Model Package

```bash
hf download OneScience-Group/SatMAE --local-dir ./SatMAE
cd SatMAE
```

### Install the Runtime Environment

**DCU Environment**

```bash
# Activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# Supports installation with uv
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

**GPU Environment**

```bash
# Activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# Supports installation with uv
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

### Training Data Introduction

The temporal experiment in the paper uses fMoW RGB, with input consisting of RGB image sequences of length 3 from the same location; the multispectral experiment uses fMoW-Sentinel. In temporal-mode NPZ files, `images` is `[B,T,C,H,W]`, `timestamps` is `[B,T,3]`, with the three fields being `year - 2002`, `month - 1`, and `hour` in order, and `labels` is `[B]`. The model also accepts continuous scalar time in `[B,T]` format.

Small synthetic data following the same protocol is used by default:

```bash
python scripts/fake_data.py
```

Run the command above when using synthetic data. When using real data, do not run `fake_data.py`; save the data as `data/train.npz` and `data/test.npz`, and modify `conf/config.yaml` according to the actual protocol.

```text
images:     float32 [N,T,C,H,W]
timestamps: float32 [N,T,3]
labels:     int64   [N]
```

The three fields of `timestamps` are `year - 2002`, `month - 1`, and `hour` in order; continuous scalar time can also use the `[N,T]` format. The data should have completed size processing, channel ordering, temporal sorting, and numerical normalization.

### Training

Single GPU:

```bash
python scripts/train.py
```

Multiple GPUs:

```bash
torchrun --nproc_per_node=8 scripts/train.py
```

Training outputs:

```text
result/checkpoints/satmae.pt
result/training/metrics.json
```

The default configuration uses small synthetic data to validate the training workflow. Formal training should use the Base, Large, or Huge paper architecture presets provided by the code, together with real data and the paper's training budget.

### Trained Weights

This repository provides weights trained on fMoW RGB temporal satellite imagery and fMoW-Sentinel multispectral satellite imagery in the `weight/` folder. The weight files will be uploaded soon and are expected to be completed in the near future.

### Inference

```bash
python scripts/inference.py
```

Inference results are output to:

```text
result/output/reconstruction.npz
```

### Evaluation and Visualization

```bash
python scripts/result.py
```

Evaluation and visualization outputs are saved to:

```text
result/evaluation/metrics.json
result/evaluation/temporal_frame_reconstruction.png
result/evaluation/temporal_reconstruction_error.png
result/evaluation/spectral_band_reconstruction.png
```

Evaluation results include overall and masked-patch MSE, overall and masked-patch MSE for each temporal frame, and reconstruction MSE for each input channel. Synthetic-data results are only used to validate the engineering workflow and do not represent downstream transfer performance from the paper.

# OneScience Official Information

| Platform | OneScience Main Repository | Skills Repository |
| --- | --- | --- |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |

# Citation and License

This repository is a reproduction of the original SatMAE paper.