README.md

---
license: apache-2.0
language:
- en
tags:
- remote-sensing
- earth-observation
- self-supervised-learning
- sentinel-2
- multispectral
- sar
- feature-extraction
- vision
- softcon
- vit
- resnet
- dinov2
- transformers
library_name: transformers
pipeline_tag: feature-extraction
datasets:
- wangyi111/SSL4EO-S12
---

# SoftCon Transformers Models

Hugging Face–compatible checkpoints converted from the official [SoftCon](https://arxiv.org/abs/2405.20462) pretrained weights. Each subfolder is a standalone model repo layout (`config.json`, `model.safetensors`, preprocessor, and remote code) for feature extraction on Earth observation imagery.

## Model Description

These models are encoders pretrained with multi-label guided soft contrastive learning on [SSL4EO-S12](https://arxiv.org/abs/2211.07044) multispectral and SAR imagery, with DINOv2-style continual pretraining for ViT backbones.

This collection bundles **6 converted checkpoints**:

- **Architectures:** ResNet-50, ViT-S/14, ViT-B/14 (DINOv2-style)
- **Input modalities:** S2-L1C 13-band (`s2c`), S1 SAR 2-band (`s1`)

All folders ship self-contained remote code (`modeling_softcon.py`, processor, pipeline) and load with `trust_remote_code=True`. ViT backbones reuse transformers' built-in [`Dinov2Model`](https://huggingface.co/docs/transformers/model_doc/dinov2).

**Developed by:** [zhu-xlab / SoftCon](https://github.com/zhu-xlab/softcon)  
**Converted for Hugging Face by:** BiliSakura  
**License (weights):** [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)  
**Original paper:** [Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining](https://arxiv.org/abs/2405.20462)

## Available checkpoints (6 models)

| Folder | Backbone | Modality | Channels | Embedding dim | Legacy file |
|--------|----------|----------|----------|---------------|-------------|
| `softcon-resnet50-s2c` | RN50 | S2-L1C MS | 13 | 2048 | `B13_rn50_softcon.pth` |
| `softcon-vit-small-patch14-s2c` | ViT-S/14 | S2-L1C MS | 13 | 384 | `B13_vits14_softcon_enc.pth` |
| `softcon-vit-base-patch14-s2c` | ViT-B/14 | S2-L1C MS | 13 | 768 | `B13_vitb14_softcon_enc.pth` |
| `softcon-resnet50-s1` | RN50 | S1 SAR | 2 | 2048 | `B2_rn50_softcon.pth` |
| `softcon-vit-small-patch14-s1` | ViT-S/14 | S1 SAR | 2 | 384 | `B2_vits14_softcon_enc.pth` |
| `softcon-vit-base-patch14-s1` | ViT-B/14 | S1 SAR | 2 | 768 | `B2_vitb14_softcon_enc.pth` |


## Usage

Processors default to **`do_resize: false`**. Pass patches at native `(H, W, C)`; the processor rescales to `[0, 1]` without changing spatial size.

```python
from transformers import pipeline
import numpy as np

REPO = "BiliSakura/SOFTCON-transformers"
SUBFOLDER = "softcon-vit-small-patch14-s2c"

pipe = pipeline(
    task="softcon-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)

# S2-L1C: 13 bands at native resolution (e.g. 512×512)
image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # [1, 384]
```

Dense token features:

```python
tokens = pipe(image, pool=False, return_tensors=True)
```

Opt in to resize (patch-14 models were pretrained on 224×224):

```python
features = pipe(image, pool=True, return_tensors=True, image_processor_kwargs={"do_resize": True})
```

Load components directly:

```python
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
```

## Normalization

By default, the bundled image processor rescales inputs to `[0, 1]` by dividing by 255. SoftCon recommends mapping each channel to uint8 using per-channel mean/std from SSL4EO-S12 or the target dataset before inference. Enable `do_normalize=True` with per-channel `image_mean` and `image_std` on `SoftConImageProcessor` when using normalized inputs.

## Dependencies

- `transformers`, `torch`, `torchvision`, `safetensors`
- `opencv-python` (multispectral resize with more than 4 channels)

## Citation

```bibtex
@misc{wang2024multilabel,
  title={Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining},
  author={Wang, Yi and Albrecht, Conrad M and Zhu, Xiao Xiang},
  journal={arXiv preprint arXiv:2405.20462},
  year={2024}
}
```