README.md

---
license: gpl-3.0
language:
- en
tags:
- OneScience
- Earth Science
- Hyperspectral Remote Sensing
- Masked Autoencoder
frameworks: PyTorch
datasets:
- fMoW-Sentinel
- BigEarthNet
---

<p align="center">
  <strong>
    <span style="font-size: 30px;">SpectralGPT</span>
  </strong>
</p>

# Model Introduction

SpectralGPT is a foundation model for spectral remote sensing imagery. It learns cross-band and spatial structures through spatial-spectral 3D patching, masked autoencoding, and progressive pretraining, and can provide representations for tasks such as classification, segmentation, and change detection.

Paper: SpectralGPT: Spectral Remote Sensing Foundation Model  
https://arxiv.org/abs/2311.07113

# Model Description

SpectralGPT was proposed by a research team from Northwestern Polytechnical University and other institutions. The model is first trained on `96x96` fMoW-Sentinel data and then performs second-stage progressive pretraining on `128x128` BigEarthNet data. It is suitable for multispectral image reconstruction, spectral remote sensing representation learning, and transfer to downstream remote sensing tasks.

# Use Cases

| Scenario | Description |
| :---: | :--- |
| Progressive pretraining | Sequentially perform `96x96` first-stage and `128x128` second-stage training. |
| Spatial-spectral reconstruction | Perform masked reconstruction of 12-band spatial-spectral Sentinel-2 patches. |
| Multispectral land-cover classification | Transfer spatial-spectral representations and fine-tune for land-cover classification on datasets such as EuroSAT and BigEarthNet. |
| Semantic segmentation and change detection | Adapt the pretrained encoder to pixel-level land-cover segmentation and bi-temporal remote sensing change detection tasks. |
| Local engineering validation | Use a small amount of synthetic data to check the training, inference, and evaluation workflows. |
| Multi-GPU training | Launch distributed training with `torchrun`. |

# Usage Guide

## 1. OneCode

Experience intelligent one-click AI4S programming through the OneCode online environment:

[Click to Experience Intelligent One-Click AI4S Programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)

## 2. Download and Installation

```bash
hf download OneScience-Group/SpectralGPT --local-dir ./SpectralGPT
cd SpectralGPT
```

### Environment Dependencies

**Hardware Requirements**

- A GPU or DCU is recommended.
- CPU can be used for small-configuration connectivity validation; full training and inference will be slow.
- DCU users must install DTK in advance. DTK 25.04.2 or above, or the OneScience recommended version matching the current cluster, is recommended.

**DCU Environment**

```bash
# Please activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# uv installation is supported
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

**GPU Environment**

```bash
# Please activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# uv installation is supported
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

### Training Data Introduction

By default, a small amount of synthetic data is used to validate the two-stage engineering workflow. The first stage uses fMoW-Sentinel-style samples, and the second stage uses BigEarthNet-style samples.

The synthetic data preserves the official progressive pretraining input specifications of 12 bands, `96x96` in the first stage, and `128x128` in the second stage.

Real data must be preprocessed and converted to the following NPZ training protocol. This protocol is consistent with the model input specification but is not the download format of the original datasets.

```text
stage1:
images: float32 [N,12,96,96]
band_order: string [12]
normalization: string scalar
scale_factors: float32 [N]
stage: string scalar = stage1

stage2:
images: float32 [N,12,128,128]
band_order: string [12]
normalization: string scalar
scale_factors: float32 [N]
stage: string scalar = stage2
```

`fake_data.py` automatically writes the `protocol` and `data_source` protocol metadata. These fields must be retained when using real data.

```bash
python scripts/fake_data.py
```

### Training

```bash
python scripts/train.py
```

For multi-GPU training, use:

```bash
torchrun --nproc_per_node=8 scripts/train.py
```

Training first completes the 96-size first stage, then interpolates the spatial positional encoding and completes the 128-size second stage, saving stage checkpoints, a final checkpoint, and aggregate training metrics. The default configuration is intended for quick workflow validation. Formal experiments should use the two-stage data scale, model configuration, and training duration corresponding to the paper.

```text
result/checkpoints/stage1.pth
result/checkpoints/stage2.pth
result/checkpoints/final.pth
result/training/metrics.json
```

### Training Weights

This repository will provide SpectralGPT training weights in the `weight/` folder. The weight files will be uploaded soon and are expected to be available in the near future.

### Inference

```bash
python scripts/inference.py
```

Inference loads the final second-stage checkpoint, performs masked reconstruction on the `128x128` test data, and saves the results to:

```text
result/output/reconstruction.npz
```

### Evaluation and Visualization

```bash
python scripts/result.py
```

Evaluation reports masked-region MSE, MAE, PSNR, spectral angle, and per-band RMSE, and generates a figure containing the input, visible region, prediction, and composite result. Results on synthetic data are only for engineering workflow validation and do not represent full-paper performance.

```text
result/output/metrics.json
result/output/reconstruction.png
```

# Official OneScience Resources

| Platform | OneScience Main Repository | Skills Repository |
| --- | --- | --- |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |

# Citation and License

This repository is a reproduction of the original SpectralGPT paper.

Use of the code and data in this repository remains subject to the licenses and terms of use of their respective projects.