README.md

---
license: apache-2.0
language:
- en
tags:
- OneScience
- Earth Science
- Multi-Source Earth Observation
- Remote Sensing Foundation Model
frameworks: PyTorch
---

<p align="center">
  <strong>
    <span style="font-size: 30px;">Clay Foundation Model</span>
  </strong>
</p>

# Model Introduction

Clay Foundation Model is an independent engineering reproduction of the public Clay v1.5 specification. It jointly encodes satellite imagery from different sensors, band center wavelengths, ground sample distance, time, and latitude/longitude into Earth observation embeddings, and reconstructs input bands with a Masked Autoencoder. It can serve as a representation backbone for land-cover classification, regression, change detection, and other downstream remote-sensing tasks.

Official documentation: Clay Foundation Model v1.5  
https://clay-foundation.github.io/model/release-notes/specification.html

# Model Description

Clay Foundation Model was proposed by the Clay Foundation. The model was trained on multi-sensor Earth observation data including Sentinel-2, Landsat, Sentinel-1, NAIP, LINZ, and MODIS. It is suitable for remote-sensing image representation, multispectral reconstruction, land-cover classification, regression, change detection, and related tasks.

# Use Cases

| Use Case | Description |
| :---: | :--- |
| Multi-sensor representation | Use one model to process imagery with different numbers of bands from sensors such as Sentinel-2, Landsat, and Sentinel-1. |
| Dynamic spectral encoding | Dynamically generate Patch Embeddings from the center wavelengths of input bands to validate integration methods for new sensors. |
| Remote-sensing image reconstruction | Use a Masked Autoencoder to reconstruct masked patches in multispectral imagery. |
| Local engineering validation | Use a small amount of synthetic data to validate training, inference, evaluation, visualization, and checkpoint workflows. |
| Multi-GPU training | Launch distributed data-parallel training with `torchrun`. |

# Usage Instructions

## 1.OneCode

Experience intelligent, one-click AI4S programming through the OneCode online environment:

[Try intelligent, one-click AI4S programming](https://web-2069360198568017922-iaaj.ksai.scnet.cn:58043/home)

## 2.Download and Installation

```bash
hf download OneScience-Group/ClayFoundation --local-dir ./ClayFoundation
cd ClayFoundation
```

### Environment Dependencies

**Hardware Requirements**

- A GPU or DCU is recommended.
- A CPU can be used to verify connectivity with the default small-sample configuration; training the official-size model requires large-scale acceleration resources.
- DCU users must install DTK in advance. DTK 25.04.2 or later, or the OneScience-recommended version matching the current cluster, is recommended.

**DCU Environment**

```bash
# Activate DTK and CONDA first
conda create -n onescience311 python=3.11 -y
conda activate onescience311
# uv installation is supported
pip install onescience[earth-dcu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

**GPU Environment**

```bash
# Activate CONDA first
conda create -n onescience311 python=3.11 -y libstdcxx-ng=12 libgcc-ng=12 gcc_linux-64=12 gxx_linux-64=12
conda activate onescience311
# uv installation is supported
pip install onescience[earth-gpu] -i http://mirrors.onescience.ai:3141/pypi/simple/ --trusted-host mirrors.onescience.ai
```

### Training Data

This repository uses a small number of synthetic samples to validate the engineering workflow. The training and test data are stored in `data/train.npz` and `data/test.npz`, respectively. The synthetic data contains 10-band Sentinel-2, 6-band Landsat, and 2-band Sentinel-1 imagery, with center wavelengths and ground sample distances corresponding to the official metadata for each sensor. Each sample also includes week, hour, latitude, and longitude metadata; a synthetic teacher representation target; and classification and regression targets for lightweight evaluation.

The default input size is 64×64, with four training samples and two test samples. This scale is intended only to validate the dynamic-band interface, spatiotemporal metadata encoding, MAE training, and multi-sensor inference workflow. It does not represent the data distribution or training scale of the approximately 70 million global remote-sensing chips used by the official model.

```bash
python scripts/fake_data.py
```

### Training

```bash
python scripts/train.py
```

For multi-GPU training, use:

```bash
torchrun --nproc_per_node=8 --nnodes=1 --rdzv_id=1000 --rdzv_backend=c10d --max_restarts=0 --master_addr="localhost" --master_port=29500 scripts/train.py
```

The default configuration is intended for rapid workflow validation. Formal experiments should use real remote-sensing data, the official-size configuration, a DINOv2 teacher, and a complete training schedule.

```text
result/checkpoints/clayfoundation.pt
result/training/metrics.json
```

### Trained Weights

This repository does not include synthetic or official weights in `weight/`. The Clay Foundation has released the official Clay v1.5 checkpoint:

https://huggingface.co/made-with-clay/Clay/resolve/main/v1.5/clay-v1.5.ckpt

This repository is a scaled-down independent engineering implementation. Its model parameter names and dimensions are incompatible with the official Encoder weights of approximately 1.25 GB. To use the official weights, use the model implementation and data preprocessing workflow provided by the official Clay repository.

### Inference

```bash
python scripts/inference.py
```

Inference loads the training checkpoint and separately processes Sentinel-2, Landsat, and Sentinel-1 test imagery. It generates Encoder embeddings, L2-normalized projected embeddings, complete multispectral MAE reconstructions, reconstruction loss, and teacher representation alignment loss, and saves them to:

```text
result/output/predictions.npz
```

### Evaluation and Visualization

```bash
python scripts/result.py
```

The evaluation measures image reconstruction and representation quality across sensors and reports reconstruction error, embedding norms, and representation similarity. It also generates comparison plots of the input imagery, reconstructed results, and absolute errors. Results on synthetic data are intended only to validate the engineering workflow and do not represent the official Clay v1.5 training loss, real downstream-task performance, or cross-region generalization capability.

```text
result/evaluation/metrics.json
result/evaluation/comparison.png
```

# Official OneScience Information

| Platform | OneScience Main Repository | Skills Repository |
| --- | --- | --- |
| Gitee | https://gitee.com/onescience-ai/onescience | https://gitee.com/onescience-ai/oneskills |
| GitHub | https://github.com/onescience-ai/OneScience | https://github.com/onescience-ai/oneskills |

# Citation and License

This repository is an independent engineering reproduction of the public Clay Foundation Model v1.5 specification. The official Clay source code and model weights are licensed under Apache-2.0.

Official Clay repository:  
https://github.com/Clay-foundation/model

Use of this repository's code, the official model weights, and the data remains subject to the licenses and terms of use of their respective projects.