BiliSakura/DECUR-transformers

DeCUR Transformers Models

Self-contained Hugging Face checkpoints converted from the official DeCUR pretrained weights. Each subfolder ships remote code (modeling_decur.py, processor, pipeline) and loads with trust_remote_code=True.

Original paper: DeCUR: Decoupling Common and Unique Representations for Multimodal Self-Supervision
Converted for Hugging Face by: BiliSakura

Available checkpoints (16 models)

Folder	Backbone	Modality	Channels	RDA
decur-resnet50-s1	ResNet-50	Sentinel-1 SAR	2	no
decur-resnet50-s2c	ResNet-50	Sentinel-2 L1C	13	no
decur-resnet50-rda-s1	ResNet-50 + RDA	Sentinel-1 SAR	2	yes
decur-resnet50-rda-s2c	ResNet-50 + RDA	Sentinel-2 L1C	13	yes
decur-vit-small-patch16-s1	ViT-S/16	Sentinel-1 SAR	2	no
decur-vit-small-patch16-s2c	ViT-S/16	Sentinel-2 L1C	13	no
decur-resnet50-rgb	ResNet-50	GeoNRW RGB	3	no
decur-resnet50-dem	ResNet-50	GeoNRW DEM	3	no
decur-resnet50-rda-rgb	ResNet-50 + RDA	GeoNRW RGB	3	yes
decur-resnet50-rda-dem	ResNet-50 + RDA	GeoNRW DEM	3	yes
decur-vit-small-patch16-rgb	ViT-S/16	GeoNRW RGB	3	no
decur-vit-small-patch16-dem	ViT-S/16	GeoNRW DEM	3	no
decur-mit-b2-rgb	SegFormer MiT-B2	SUN RGB-D RGB	3	no
decur-mit-b2-hha	SegFormer MiT-B2	SUN RGB-D HHA	3	no
decur-mit-b5-rgb	SegFormer MiT-B5	SUN RGB-D RGB	3	no
decur-mit-b5-hha	SegFormer MiT-B5	SUN RGB-D HHA	3	no

Legacy .pth filename mapping is in conversion_manifest.json.

Usage

Processors default to do_resize: false — pass images at native (H, W, C); only value rescaling (/255) is applied unless you enable normalization.

from transformers import pipeline
import numpy as np

REPO = "/path/to/DECUR-transformers"
SUBFOLDER = "decur-resnet50-s2c"

pipe = pipeline(
    task="decur-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)

# Native Sentinel-2 patch (e.g. 512×512, 13 bands)
image = np.random.randint(0, 255, (512, 512, 13), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 2048])

# Dense token map (spatial grid scales with input size)
tokens = pipe(image, pool=False, return_tensors=True)
print(tokens.shape)    # ResNet: [1, 256, 2048] for 512×512

Sentinel-1 ViT (2 channels):

SUBFOLDER = "decur-vit-small-patch16-s1"
pipe = pipeline(
    task="decur-feature-extraction",
    model=REPO,
    trust_remote_code=True,
    model_kwargs={"subfolder": SUBFOLDER},
)
image = np.random.randint(0, 255, (448, 448, 2), dtype=np.uint8)
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 384])

Load components directly:

from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(REPO, subfolder=SUBFOLDER, trust_remote_code=True)
inputs = processor(image, return_tensors="pt")
out = model(**inputs)

Each checkpoint folder can also be used as a standalone local path (omit subfolder):

pipe = pipeline(
    task="decur-feature-extraction",
    model="/path/to/DECUR-transformers/decur-mit-b2-rgb",
    trust_remote_code=True,
)

Native resolution notes

Backbone	Native larger input	Pooled output	Sequence output
ResNet-50	Yes (fully conv)	[B, 2048]	[B, H'×W', 2048]
ViT-S/16	Yes (interpolate_pos_encoding=True by default)	[B, 384] (CLS)	[B, N, 384]
MiT-B2/B5	Yes	[B, 512] (spatial mean)	[B, H'×W', 512]

• size in preprocessor_config.json (224×224) is the pretraining reference, not a forced input size.
• ViT uses interpolated positional embeddings by default for non-224 inputs. Disable with model(..., interpolate_pos_encoding=False).
• Opt in to resize (e.g. match 224×224 pretraining):

features = pipe(
    image,
    pool=True,
    return_tensors=True,
    image_processor_kwargs={"do_resize": True},
)

The custom pipeline is registered in each checkpoint's config.json under custom_pipelines.decur-feature-extraction (see Adding a new pipeline).

Test CLI

conda activate rsgen
python test_decur.py --model decur-mit-b2-rgb
python test_decur.py --all

Dependencies

• transformers
• torch
• torchvision (ResNet backbones)
• einops (RDA modules)
• opencv-python (only when resizing multispectral inputs with >4 channels)

Notes

• Pooled features: ResNet [B, 2048], ViT [B, 384] (CLS token), MiT [B, 512] (spatial mean).
• GeoNRW DEM checkpoints use 3 input channels in the released weights (not 1-channel DSM).
• RDA ResNet models include deformable attention modules (da_l3, da_l4).