README.md

---
license: apache-2.0
language:
- en
tags:
- remote-sensing
- earth-observation
- self-supervised-learning
- multispectral
- sar
- rgb
- depth
- feature-extraction
- decur
- resnet
- vit
- segformer
- transformers
library_name: transformers
pipeline_tag: feature-extraction
---

# DeCUR Transformers Models

Self-contained Hugging Face checkpoints converted from the official [DeCUR](https://arxiv.org/abs/2312.10132) pretrained weights. Each subfolder ships remote code (`modeling_decur.py`, processor, pipeline) and loads with `trust_remote_code=True`.

**Original paper:** [DeCUR: Decoupling Common and Unique Representations for Multimodal Self-Supervision](https://arxiv.org/abs/2312.10132)  
**Converted for Hugging Face by:** BiliSakura

## Available checkpoints (16 models)

| Folder | Backbone | Modality | Channels | RDA |
|--------|----------|----------|----------|-----|
| `decur-resnet50-s1` | ResNet-50 | Sentinel-1 SAR | 2 | no |
| `decur-resnet50-s2c` | ResNet-50 | Sentinel-2 L1C | 13 | no |
| `decur-resnet50-rda-s1` | ResNet-50 + RDA | Sentinel-1 SAR | 2 | yes |
| `decur-resnet50-rda-s2c` | ResNet-50 + RDA | Sentinel-2 L1C | 13 | yes |
| `decur-vit-small-patch16-s1` | ViT-S/16 | Sentinel-1 SAR | 2 | no |
| `decur-vit-small-patch16-s2c` | ViT-S/16 | Sentinel-2 L1C | 13 | no |
| `decur-resnet50-rgb` | ResNet-50 | GeoNRW RGB | 3 | no |
| `decur-resnet50-dem` | ResNet-50 | GeoNRW DEM | 3 | no |
| `decur-resnet50-rda-rgb` | ResNet-50 + RDA | GeoNRW RGB | 3 | yes |
| `decur-resnet50-rda-dem` | ResNet-50 + RDA | GeoNRW DEM | 3 | yes |
| `decur-vit-small-patch16-rgb` | ViT-S/16 | GeoNRW RGB | 3 | no |
| `decur-vit-small-patch16-dem` | ViT-S/16 | GeoNRW DEM | 3 | no |
| `decur-mit-b2-rgb` | SegFormer MiT-B2 | SUN RGB-D RGB | 3 | no |
| `decur-mit-b2-hha` | SegFormer MiT-B2 | SUN RGB-D HHA | 3 | no |
| `decur-mit-b5-rgb` | SegFormer MiT-B5 | SUN RGB-D RGB | 3 | no |
| `decur-mit-b5-hha` | SegFormer MiT-B5 | SUN RGB-D HHA | 3 | no |

Legacy `.pth` filename mapping is in [`conversion_manifest.json`](conversion_manifest.json).

## Usage

Processors default to **`do_resize: false`** — pass images at native `(H, W, C)`; only value rescaling (`/255`) is applied unless you enable normalization.

```python
from transformers import pipeline
import numpy as np

REPO = "/path/to/DECUR-transformers"
SUBFOLDER = "decur-resnet50-s2c"

pipe = pipeline(
    task="decur-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)

# Native Sentinel-2 patch (e.g. 512×512, 13 bands)
image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 2048])

# Dense token map (spatial grid scales with input size)
tokens = pipe(image, pool=False, return_tensors=True)
print(tokens.shape)    # ResNet: [1, 256, 2048] for 512×512
```

Sentinel-1 ViT (2 channels):

```python
SUBFOLDER = "decur-vit-small-patch16-s1"
pipe = pipeline(
    task="decur-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)
image = np.random.randint(0, 255, (448, 448, 2), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 384])
```

Load components directly:

```python
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
inputs = processor(image, return_tensors="pt")
out = model(**inputs)
```

Each checkpoint folder can also be used as a standalone local path (omit `subfolder`):

```python
pipe = pipeline(
    task="decur-feature-extraction",
    model="/path/to/DECUR-transformers/decur-mit-b2-rgb",
    trust_remote_code=True,
)
```

## Native resolution notes

| Backbone | Native larger input | Pooled output | Sequence output |
|----------|---------------------|---------------|-----------------|
| ResNet-50 | Yes (fully conv) | `[B, 2048]` | `[B, H'×W', 2048]` |
| ViT-S/16 | Yes (`interpolate_pos_encoding=True` by default) | `[B, 384]` (CLS) | `[B, N, 384]` |
| MiT-B2/B5 | Yes | `[B, 512]` (spatial mean) | `[B, H'×W', 512]` |

- **`size` in `preprocessor_config.json`** (224×224) is the pretraining reference, not a forced input size.
- **ViT** uses interpolated positional embeddings by default for non-224 inputs. Disable with `model(..., interpolate_pos_encoding=False)`.
- **Opt in to resize** (e.g. match 224×224 pretraining):

```python
features = pipe(
    image,
    pool=True,
    return_tensors=True,
    image_processor_kwargs={"do_resize": True},
)
```

The custom pipeline is registered in each checkpoint's `config.json` under `custom_pipelines.decur-feature-extraction` (see [Adding a new pipeline](https://huggingface.co/docs/transformers/add_new_pipeline#upload-to-the-hub)).

## Test CLI

```bash
conda activate rsgen
python test_decur.py --model decur-mit-b2-rgb
python test_decur.py --all
```

## Dependencies

- `transformers`
- `torch`
- `torchvision` (ResNet backbones)
- `einops` (RDA modules)
- `opencv-python` (only when resizing multispectral inputs with >4 channels)

## Notes

- Pooled features: ResNet `[B, 2048]`, ViT `[B, 384]` (CLS token), MiT `[B, 512]` (spatial mean).
- GeoNRW DEM checkpoints use **3 input channels** in the released weights (not 1-channel DSM).
- RDA ResNet models include deformable attention modules (`da_l3`, `da_l4`).