--- license: cc-by-4.0 language: - en tags: - remote-sensing - earth-observation - self-supervised-learning - sentinel-2 - sentinel-1 - multispectral - sar - feature-extraction - vision - ssl4eo - mae - moco - dino - data2vec - vit - resnet - transformers library_name: transformers pipeline_tag: feature-extraction datasets: - wangyi111/SSL4EO-S12 --- # SSL4EO-S12 Transformers Models Hugging Face–compatible checkpoints converted from the official [SSL4EO-S12](https://arxiv.org/abs/2211.07044) pretrained weights. Each subfolder is a standalone model repo layout (`config.json`, `model.safetensors`, preprocessor, and optional remote code) for feature extraction on Earth observation imagery. ## Model Description These models are self-supervised encoders pretrained on the [SSL4EO-S12 dataset](https://huggingface.co/datasets/wangyi111/SSL4EO-S12): a large-scale multimodal, multitemporal corpus of Sentinel-1 SAR and Sentinel-2 multispectral patches from 251k+ global locations. This collection bundles **16 converted checkpoints** spanning: - **Architectures:** ViT (S/B/L/H) and ResNet18/50 - **SSL methods:** MAE, MoCo, DINO, Data2vec - **Input modalities:** S2-L1C 13-band (`s2c`), S1 SAR 2-band (`s1`), S2 RGB 3-band (`rgb`) ViT MAE/MoCo/DINO and all ResNet folders ship self-contained remote code (`modeling_*.py`, processor, pipeline) and load with `trust_remote_code=True`. The Data2vec folder currently provides weights + config only. **Developed by:** [zhu-xlab / SSL4EO-S12](https://github.com/zhu-xlab/SSL4EO-S12) **Converted for Hugging Face by:** BiliSakura **License (weights):** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) **Original paper:** [SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation](https://arxiv.org/abs/2211.07044) ## Available checkpoints (16 models) | Folder | SSL | Arch | Input | |--------|-----|------|-------| | `ssl4eo-vit-small-patch16-s2c-mae` | MAE | ViT-S/16 | S2-L1C 13-band | | `ssl4eo-vit-base-patch16-s2c-mae` | MAE | ViT-B/16 | S2-L1C 13-band | | `ssl4eo-vit-large-patch16-s2c-mae` | MAE | ViT-L/16 | S2-L1C 13-band | | `ssl4eo-vit-huge-patch14-s2c-mae` | MAE | ViT-H/14 | S2-L1C 13-band | | `ssl4eo-vit-small-patch16-s1-mae` | MAE | ViT-S/16 | S1 SAR 2-band | | `ssl4eo-vit-base-patch16-s1-mae` | MAE | ViT-B/16 | S1 SAR 2-band | | `ssl4eo-vit-large-patch16-s1-mae` | MAE | ViT-L/16 | S1 SAR 2-band | | `ssl4eo-vit-huge-patch14-s1-mae` | MAE | ViT-H/14 | S1 SAR 2-band | | `ssl4eo-vit-small-patch16-s2c-moco` | MoCo v3 | ViT-S/16 | S2-L1C 13-band | | `ssl4eo-vit-small-patch16-s2c-dino` | DINO | ViT-S/16 | S2-L1C 13-band | | `ssl4eo-vit-small-patch16-s2c-data2vec` | Data2vec | ViT-S/16 | S2-L1C 13-band | | `ssl4eo-resnet18-rgb-moco` | MoCo v2 | ResNet18 | S2-L1C RGB | | `ssl4eo-resnet18-s2c-moco` | MoCo v2 | ResNet18 | S2-L1C 13-band | | `ssl4eo-resnet50-s2c-moco` | MoCo v2 | ResNet50 | S2-L1C 13-band | | `ssl4eo-resnet50-s2c-dino` | DINO | ResNet50 | S2-L1C 13-band | | `ssl4eo-resnet50-s1-moco` | MoCo v2 | ResNet50 | S1 SAR 2-band | Legacy `.pth` filename mapping is in [`conversion_manifest.json`](conversion_manifest.json). ## Intended use - Unsupervised / self-supervised **feature extraction** on Sentinel-1 or Sentinel-2 patches - **Linear probing** or **fine-tuning** for EO downstream tasks (classification, segmentation, change detection) - Research baselines comparable to the original SSL4EO-S12 benchmark ## Out-of-scope use - Not trained for generative tasks, captioning, or general natural-image applications - Not a drop-in replacement for ImageNet-pretrained models on RGB natural scenes - Band count and preprocessing must match the checkpoint modality (`num_channels` in `config.json`) ## Usage ### ViT (self-contained remote code) Processors default to **`do_resize: false`**. Pass native `(H, W, C)` patches; spatial token count scales with input size for ViT and ResNet backbones. ```python from transformers import pipeline import numpy as np REPO = "BiliSakura/SSL4EO-S12-transformers" SUBFOLDER = "ssl4eo-vit-base-patch16-s2c-mae" pipe = pipeline( task="ssl4eo-feature-extraction", model=REPO, trust_remote_code=True, model_kwargs={"subfolder": SUBFOLDER}, ) # S2-L1C: 13 bands at native size (e.g. 512×512) image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8) features = pipe(image, pool=True, return_tensors=True) print(features.shape) # [1, hidden_size] ``` Opt in to 224×224 resize: ```python features = pipe(image, pool=True, return_tensors=True, image_processor_kwargs={"do_resize": True}) ``` Load components directly: ```python from transformers import AutoModel, AutoImageProcessor model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True) processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True) ``` ### ResNet (self-contained remote code) ```python from transformers import pipeline import numpy as np pipe = pipeline( task="ssl4eo-feature-extraction", model="BiliSakura/SSL4EO-S12-transformers", trust_remote_code=True, model_kwargs={"subfolder": "ssl4eo-resnet50-s2c-moco"}, ) image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8) features = pipe(image, pool=True, return_tensors=True) print(features.shape) # [1, 2048] ``` Or load via the [`ssl4eo`](https://github.com/zhu-xlab/SSL4EO-S12) package: ```python from ssl4eo.models.ssl4eo_resnet import SSL4EOResNetModel model = SSL4EOResNetModel.from_pretrained( "BiliSakura/SSL4EO-S12-transformers", subfolder="ssl4eo-resnet18-rgb-moco", ) ``` ### Local paths Replace `REPO` with a local directory, e.g. `/path/to/SSL4EO-S12-transformers/ssl4eo-vit-base-patch16-s2c-mae`, and omit `subfolder` when pointing at a single checkpoint folder. ## Training data All weights were pretrained on **SSL4EO-S12** (Sentinel-1 + Sentinel-2 patch triplets, ~251k locations, four seasonal timestamps). See the [dataset card](https://huggingface.co/datasets/wangyi111/SSL4EO-S12) and [paper](https://arxiv.org/abs/2211.07044) for collection and preprocessing details. Default ViT/ResNet pretraining used **100 epochs** on 13-band S2-L1C unless noted (MAE ViT-H uses 199 epochs). Inputs are clipped to `[0, 1]` by dividing reflectance by `10000`. ## Dependencies - `transformers`, `timm`, `torch`, `torchvision`, `safetensors` - `opencv-python` (multispectral resize with more than 4 channels) - `ssl4eo` (optional; required for Data2vec loading until remote-code templates are added) ## Citation ```bibtex @article{wang2022ssl4eo, title={SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observation}, author={Wang, Yi and Braham, Nassim Ait Ali and Xiong, Zhitong and Liu, Chenying and Albrecht, Conrad M and Zhu, Xiao Xiang}, journal={arXiv preprint arXiv:2211.07044}, year={2022} } ``` ## License Pretrained **model weights** in this repository are released under [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/), consistent with the [SSL4EO-S12 project](https://github.com/zhu-xlab/SSL4EO-S12). Remote-code files derived from the integration layer may follow the upstream repository license (Apache-2.0).