File size: 4,584 Bytes
8d93654 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | ---
license: apache-2.0
tags:
- remote-sensing
- earth-observation
- skysensepp
- feature-extraction
pipeline_tag: feature-extraction
---
# SkySense++ Transformers
HuggingFace-compatible checkpoints for SkySense++ zero-shot MSL backbones, converted from the official release weights.
## Checkpoints
| Directory | Modality | Architecture | Source |
|-----------|----------|--------------|--------|
| `skysensepp-swinv2-msl-hr` | High-res optical | SwinV2 Huge + MSL | `skysensepp_release_hr.pth` |
| `skysensepp-vit-msl-s2` | Sentinel-2 | ViT-Large + MSL | `skysensepp_release_s2.pth` |
| `skysensepp-vit-msl-s1` | Sentinel-1 | ViT-Large + MSL | `skysensepp_release_s1.pth` |
| `skysensepp-fusion-neck` | Multi-modal fusion (optional) | TransformerEncoder | `fusion.*` from `skysensepp_release.ckpt` |
| `skysensepp-fewshot-release` | Full 1-shot segmentation | HR + S2 + S1 + fusion + VAE + UPerHead | `skysensepp_release.ckpt` |
Each subdirectory is a self-contained HuggingFace model repo with remote code (`trust_remote_code=True`).
The fusion neck is an **optional** component — backbone checkpoints do not include or require it by default.
The few-shot release bundles all submodules into one end-to-end model (~6.8 GB).
## Usage
```python
from transformers import pipeline
import torch
MODEL = "/path/to/SkySensePlusPlus-transformers/skysensepp-swinv2-msl-hr"
pipe = pipeline(
task="image-feature-extraction",
model=MODEL,
trust_remote_code=True,
device="cpu",
)
hr_img = torch.randn(1, 3, 512, 512)
annotation = torch.zeros(1, 512, 512, dtype=torch.long) # semantic class indices
features = pipe(hr_img, annotation=annotation)
print(features["last_hidden_state"].shape) # (1, 2816, 16, 16)
```
Sentinel-2 / Sentinel-1 backbones use the same pipeline pattern:
```python
s2_pipe = pipeline(
task="image-feature-extraction",
model="/path/to/skysensepp-vit-msl-s2",
trust_remote_code=True,
device="cpu",
)
s2_img = torch.randn(1, 10, 16, 16)
s2_anno = torch.zeros(1, 16, 16, dtype=torch.long)
features = s2_pipe(s2_img, annotation=s2_anno)
print(features["last_hidden_state"].shape)
```
SkySense++ MSL models require both imagery and a semantic annotation map. Use class index `0` for background/unlabeled regions during zero-shot feature extraction.
### Optional fusion neck
```python
fusion_pipe = pipeline(
task="skysensepp-fusion",
model="/path/to/skysensepp-fusion-neck",
trust_remote_code=True,
device="cpu",
)
# Concatenated HR + S2 + S1 stage-3 tokens per spatial location
hidden_states = torch.randn(256, 3, 2816)
fused = fusion_pipe(hidden_states)
print(fused["pooler_output"].shape) # (256, 1024)
```
### Few-shot / 1-shot segmentation
The full release model expects vertically stacked prompt+query inputs (prompt on top, query on bottom):
```python
from transformers import pipeline
import torch
MODEL = "/path/to/SkySensePlusPlus-transformers/skysensepp-fewshot-release"
pipe = pipeline(
task="skysensepp-fewshot",
model=MODEL,
trust_remote_code=True,
device=0, # GPU recommended (~24 GB); CPU OOMs at 1024×512 HR
)
# Stacked HR (3, 1024, 512), S2/S1 with seq=2, RGB targets (ImageNet-normalized)
hr = torch.randn(1, 3, 1024, 512)
s2 = torch.randn(1, 10, 2, 32, 32)
s1 = torch.randn(1, 2, 2, 32, 32)
targets = torch.randn(1, 3, 1024, 512) # use real RGB annotation maps in practice
anno_mask = torch.zeros(1, 8, 4, dtype=torch.long)
anno_mask[:, 4:, :] = 1 # mask query (bottom) half
result = pipe(hr, s2_img=s2, s1_img=s1, targets=targets, anno_mask=anno_mask)
print(result["logits"].shape) # (1, 65, 512, 512) — query region only
```
## Conversion
Source project: `/home/czy/local/projects/SkySensePlusPlus-transformers`
```bash
conda activate rsgen
python scripts/convert_checkpoint_to_hf.py \
--input-path /path/to/skysensepp_release_hr.pth \
--modality hr \
--output-dir /path/to/skysensepp-swinv2-msl-hr \
--clean-output
# Full few-shot release (~6.8 GB)
python scripts/convert_checkpoint_to_hf.py \
--input-path /path/to/skysensepp_release.ckpt \
--modality fewshot \
--output-dir /path/to/skysensepp-fewshot-release \
--clean-output
```
## Notes
- HR conversion skips Swin relative-position buffers (`relative_position_index`, `relative_coords_table`). These are **deterministically recomputed** at init from window geometry — not randomly initialized. Learned CPB weights (`cpb_mlp`, `logit_scale`) are loaded.
- The few-shot model uses the same 62 skipped HR buffers; all 1522 learned tensors load with 0 unexpected keys.
|