README.md

---
license: cc-by-4.0
language:
- en
tags:
- remote-sensing
- earth-observation
- self-supervised-learning
- sentinel-2
- sentinel-1
- multispectral
- sar
- feature-extraction
- vision
- ssl4eo
- mae
- moco
- dino
- data2vec
- vit
- resnet
- transformers
library_name: transformers
pipeline_tag: feature-extraction
datasets:
- wangyi111/SSL4EO-S12
---

# SSL4EO-S12 Transformers Models

Hugging Face–compatible checkpoints converted from the official [SSL4EO-S12](https://arxiv.org/abs/2211.07044) pretrained weights. Each subfolder is a standalone model repo layout (`config.json`, `model.safetensors`, preprocessor, and optional remote code) for feature extraction on Earth observation imagery.

## Model Description

These models are self-supervised encoders pretrained on the [SSL4EO-S12 dataset](https://huggingface.co/datasets/wangyi111/SSL4EO-S12): a large-scale multimodal, multitemporal corpus of Sentinel-1 SAR and Sentinel-2 multispectral patches from 251k+ global locations.

This collection bundles **16 converted checkpoints** spanning:

- **Architectures:** ViT (S/B/L/H) and ResNet18/50
- **SSL methods:** MAE, MoCo, DINO, Data2vec
- **Input modalities:** S2-L1C 13-band (`s2c`), S1 SAR 2-band (`s1`), S2 RGB 3-band (`rgb`)

ViT MAE/MoCo/DINO and all ResNet folders ship self-contained remote code (`modeling_*.py`, processor, pipeline) and load with `trust_remote_code=True`. The Data2vec folder currently provides weights + config only.

**Developed by:** [zhu-xlab / SSL4EO-S12](https://github.com/zhu-xlab/SSL4EO-S12)  
**Converted for Hugging Face by:** BiliSakura  
**License (weights):** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/)  
**Original paper:** [SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation](https://arxiv.org/abs/2211.07044)

## Available checkpoints (16 models)

| Folder | SSL | Arch | Input |
|--------|-----|------|-------|
| `ssl4eo-vit-small-patch16-s2c-mae` | MAE | ViT-S/16 | S2-L1C 13-band |
| `ssl4eo-vit-base-patch16-s2c-mae` | MAE | ViT-B/16 | S2-L1C 13-band |
| `ssl4eo-vit-large-patch16-s2c-mae` | MAE | ViT-L/16 | S2-L1C 13-band |
| `ssl4eo-vit-huge-patch14-s2c-mae` | MAE | ViT-H/14 | S2-L1C 13-band |
| `ssl4eo-vit-small-patch16-s1-mae` | MAE | ViT-S/16 | S1 SAR 2-band |
| `ssl4eo-vit-base-patch16-s1-mae` | MAE | ViT-B/16 | S1 SAR 2-band |
| `ssl4eo-vit-large-patch16-s1-mae` | MAE | ViT-L/16 | S1 SAR 2-band |
| `ssl4eo-vit-huge-patch14-s1-mae` | MAE | ViT-H/14 | S1 SAR 2-band |
| `ssl4eo-vit-small-patch16-s2c-moco` | MoCo v3 | ViT-S/16 | S2-L1C 13-band |
| `ssl4eo-vit-small-patch16-s2c-dino` | DINO | ViT-S/16 | S2-L1C 13-band |
| `ssl4eo-vit-small-patch16-s2c-data2vec` | Data2vec | ViT-S/16 | S2-L1C 13-band |
| `ssl4eo-resnet18-rgb-moco` | MoCo v2 | ResNet18 | S2-L1C RGB |
| `ssl4eo-resnet18-s2c-moco` | MoCo v2 | ResNet18 | S2-L1C 13-band |
| `ssl4eo-resnet50-s2c-moco` | MoCo v2 | ResNet50 | S2-L1C 13-band |
| `ssl4eo-resnet50-s2c-dino` | DINO | ResNet50 | S2-L1C 13-band |
| `ssl4eo-resnet50-s1-moco` | MoCo v2 | ResNet50 | S1 SAR 2-band |

Legacy `.pth` filename mapping is in [`conversion_manifest.json`](conversion_manifest.json).

## Intended use

- Unsupervised / self-supervised **feature extraction** on Sentinel-1 or Sentinel-2 patches
- **Linear probing** or **fine-tuning** for EO downstream tasks (classification, segmentation, change detection)
- Research baselines comparable to the original SSL4EO-S12 benchmark

## Out-of-scope use

- Not trained for generative tasks, captioning, or general natural-image applications
- Not a drop-in replacement for ImageNet-pretrained models on RGB natural scenes
- Band count and preprocessing must match the checkpoint modality (`num_channels` in `config.json`)

## Usage

### ViT (self-contained remote code)

Processors default to **`do_resize: false`**. Pass native `(H, W, C)` patches; spatial token count scales with input size for ViT and ResNet backbones.

```python
from transformers import pipeline
import numpy as np

REPO = "BiliSakura/SSL4EO-S12-transformers"
SUBFOLDER = "ssl4eo-vit-base-patch16-s2c-mae"

pipe = pipeline(
    task="ssl4eo-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)

# S2-L1C: 13 bands at native size (e.g. 512×512)
image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # [1, hidden_size]
```

Opt in to 224×224 resize:

```python
features = pipe(image, pool=True, return_tensors=True, image_processor_kwargs={"do_resize": True})
```

Load components directly:

```python
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
```

### ResNet (self-contained remote code)

```python
from transformers import pipeline
import numpy as np

pipe = pipeline(
    task="ssl4eo-feature-extraction",
    model="BiliSakura/SSL4EO-S12-transformers",
    trust_remote_code=True,
    model_kwargs={"subfolder": "ssl4eo-resnet50-s2c-moco"},
)

image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # [1, 2048]
```

Or load via the [`ssl4eo`](https://github.com/zhu-xlab/SSL4EO-S12) package:

```python
from ssl4eo.models.ssl4eo_resnet import SSL4EOResNetModel

model = SSL4EOResNetModel.from_pretrained(
    "BiliSakura/SSL4EO-S12-transformers",
    subfolder="ssl4eo-resnet18-rgb-moco",
)
```

### Local paths

Replace `REPO` with a local directory, e.g. `/path/to/SSL4EO-S12-transformers/ssl4eo-vit-base-patch16-s2c-mae`, and omit `subfolder` when pointing at a single checkpoint folder.

## Training data

All weights were pretrained on **SSL4EO-S12** (Sentinel-1 + Sentinel-2 patch triplets, ~251k locations, four seasonal timestamps). See the [dataset card](https://huggingface.co/datasets/wangyi111/SSL4EO-S12) and [paper](https://arxiv.org/abs/2211.07044) for collection and preprocessing details.

Default ViT/ResNet pretraining used **100 epochs** on 13-band S2-L1C unless noted (MAE ViT-H uses 199 epochs). Inputs are clipped to `[0, 1]` by dividing reflectance by `10000`.

## Dependencies

- `transformers`, `timm`, `torch`, `torchvision`, `safetensors`
- `opencv-python` (multispectral resize with more than 4 channels)
- `ssl4eo` (optional; required for Data2vec loading until remote-code templates are added)


## Citation

```bibtex
@article{wang2022ssl4eo,
  title={SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation},
  author={Wang, Yi and Braham, Nassim Ait Ali and Xiong, Zhitong and Liu, Chenying and Albrecht, Conrad M and Zhu, Xiao Xiang},
  journal={arXiv preprint arXiv:2211.07044},
  year={2022}
}
```

## License

Pretrained **model weights** in this repository are released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/), consistent with the [SSL4EO-S12 project](https://github.com/zhu-xlab/SSL4EO-S12). Remote-code files derived from the integration layer may follow the upstream repository license (Apache-2.0).