README.md

---
license: apache-2.0
tags:
  - remote-sensing
  - earth-observation
  - skysensepp
  - feature-extraction
pipeline_tag: feature-extraction
---

# SkySense++ Transformers

HuggingFace-compatible checkpoints for SkySense++ zero-shot MSL backbones, converted from the official release weights.

## Checkpoints

| Directory | Modality | Architecture | Source |
|-----------|----------|--------------|--------|
| `skysensepp-swinv2-msl-hr` | High-res optical | SwinV2 Huge + MSL | `skysensepp_release_hr.pth` |
| `skysensepp-vit-msl-s2` | Sentinel-2 | ViT-Large + MSL | `skysensepp_release_s2.pth` |
| `skysensepp-vit-msl-s1` | Sentinel-1 | ViT-Large + MSL | `skysensepp_release_s1.pth` |
| `skysensepp-fusion-neck` | Multi-modal fusion (optional) | TransformerEncoder | `fusion.*` from `skysensepp_release.ckpt` |
| `skysensepp-fewshot-release` | Full 1-shot segmentation | HR + S2 + S1 + fusion + VAE + UPerHead | `skysensepp_release.ckpt` |

Each subdirectory is a self-contained HuggingFace model repo with remote code (`trust_remote_code=True`).

The fusion neck is an **optional** component — backbone checkpoints do not include or require it by default.

The few-shot release bundles all submodules into one end-to-end model (~6.8 GB).

## Usage

```python
from transformers import pipeline
import torch

MODEL = "/path/to/SkySensePlusPlus-transformers/skysensepp-swinv2-msl-hr"

pipe = pipeline(
    task="image-feature-extraction",
    model=MODEL,
    trust_remote_code=True,
    device="cpu",
)

hr_img = torch.randn(1, 3, 512, 512)
annotation = torch.zeros(1, 512, 512, dtype=torch.long)  # semantic class indices

features = pipe(hr_img, annotation=annotation)
print(features["last_hidden_state"].shape)  # (1, 2816, 16, 16)
```

Sentinel-2 / Sentinel-1 backbones use the same pipeline pattern:

```python
s2_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/skysensepp-vit-msl-s2",
    trust_remote_code=True,
    device="cpu",
)

s2_img = torch.randn(1, 10, 16, 16)
s2_anno = torch.zeros(1, 16, 16, dtype=torch.long)
features = s2_pipe(s2_img, annotation=s2_anno)
print(features["last_hidden_state"].shape)
```

SkySense++ MSL models require both imagery and a semantic annotation map. Use class index `0` for background/unlabeled regions during zero-shot feature extraction.

### Optional fusion neck

```python
fusion_pipe = pipeline(
    task="skysensepp-fusion",
    model="/path/to/skysensepp-fusion-neck",
    trust_remote_code=True,
    device="cpu",
)

# Concatenated HR + S2 + S1 stage-3 tokens per spatial location
hidden_states = torch.randn(256, 3, 2816)
fused = fusion_pipe(hidden_states)

print(fused["pooler_output"].shape)  # (256, 1024)
```

### Few-shot / 1-shot segmentation

The full release model expects vertically stacked prompt+query inputs (prompt on top, query on bottom):

```python
from transformers import pipeline
import torch

MODEL = "/path/to/SkySensePlusPlus-transformers/skysensepp-fewshot-release"

pipe = pipeline(
    task="skysensepp-fewshot",
    model=MODEL,
    trust_remote_code=True,
    device=0,  # GPU recommended (~24 GB); CPU OOMs at 1024×512 HR
)

# Stacked HR (3, 1024, 512), S2/S1 with seq=2, RGB targets (ImageNet-normalized)
hr = torch.randn(1, 3, 1024, 512)
s2 = torch.randn(1, 10, 2, 32, 32)
s1 = torch.randn(1, 2, 2, 32, 32)
targets = torch.randn(1, 3, 1024, 512)  # use real RGB annotation maps in practice
anno_mask = torch.zeros(1, 8, 4, dtype=torch.long)
anno_mask[:, 4:, :] = 1  # mask query (bottom) half

result = pipe(hr, s2_img=s2, s1_img=s1, targets=targets, anno_mask=anno_mask)
print(result["logits"].shape)  # (1, 65, 512, 512) — query region only
```

## Conversion

Source project: `/home/czy/local/projects/SkySensePlusPlus-transformers`

```bash
conda activate rsgen
python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysensepp_release_hr.pth \
  --modality hr \
  --output-dir /path/to/skysensepp-swinv2-msl-hr \
  --clean-output

# Full few-shot release (~6.8 GB)
python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysensepp_release.ckpt \
  --modality fewshot \
  --output-dir /path/to/skysensepp-fewshot-release \
  --clean-output
```

## Notes

- HR conversion skips Swin relative-position buffers (`relative_position_index`, `relative_coords_table`). These are **deterministically recomputed** at init from window geometry — not randomly initialized. Learned CPB weights (`cpb_mlp`, `logit_scale`) are loaded.
- The few-shot model uses the same 62 skipped HR buffers; all 1522 learned tensors load with 0 unexpected keys.