File size: 3,106 Bytes
2ada71f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | ---
license: apache-2.0
tags:
- remote-sensing
- earth-observation
- skysense
- feature-extraction
pipeline_tag: feature-extraction
---
# SkySense Transformers
HuggingFace-compatible checkpoints for the SkySense (CVPR 2024) foundation model backbones.
## Checkpoints
| Directory | Modality | Architecture | Source |
|-----------|----------|--------------|--------|
| `skysense-swinv2-huge-rgb` | High-res optical (RGB) | SwinV2 Huge | `skysense_model_backbone_hr.pth` |
| `skysense-vit-large-s2` | Sentinel-2 | ViT-Large | `skysense_model_backbone_s2.pth` |
| `skysense-vit-large-s1` | Sentinel-1 SAR | ViT-Large | `skysense_model_backbone_s1.pth` |
Each subdirectory is a self-contained HuggingFace model repo with remote code (`trust_remote_code=True`).
## Usage
```python
from transformers import pipeline
import torch
# HR RGB backbone — input 224×224
hr_pipe = pipeline(
task="image-feature-extraction",
model="/path/to/SkySense-transformers/skysense-swinv2-huge-rgb",
trust_remote_code=True,
device="cpu",
)
hr_img = torch.randn(1, 3, 224, 224)
features = hr_pipe(hr_img)
print(features["last_hidden_state"].shape) # (1, 2816, 7, 7)
# Sentinel-2 — 10 bands, 64×64
s2_pipe = pipeline(
task="image-feature-extraction",
model="/path/to/SkySense-transformers/skysense-vit-large-s2",
trust_remote_code=True,
device="cpu",
)
s2_img = torch.randn(1, 10, 64, 64)
features = s2_pipe(s2_img)
print(features["last_hidden_state"].shape) # (1, 1024, 16, 16)
# Sentinel-1 — VV/VH, 64×64
s1_pipe = pipeline(
task="image-feature-extraction",
model="/path/to/SkySense-transformers/skysense-vit-large-s1",
trust_remote_code=True,
device="cpu",
)
s1_img = torch.randn(1, 2, 64, 64)
features = s1_pipe(s1_img)
print(features["last_hidden_state"].shape)
```
## Conversion
Source project: `/home/czy/local/projects/SkySense-transformers`
```bash
conda activate rsgen
python scripts/convert_checkpoint_to_hf.py \
--input-path /path/to/skysense_model_backbone_hr.pth \
--modality hr \
--output-dir /path/to/skysense-swinv2-huge-rgb \
--clean-output
python scripts/convert_checkpoint_to_hf.py \
--input-path /path/to/skysense_model_backbone_s2.pth \
--modality s2 \
--output-dir /path/to/skysense-vit-large-s2 \
--clean-output
python scripts/convert_checkpoint_to_hf.py \
--input-path /path/to/skysense_model_backbone_s1.pth \
--modality s1 \
--output-dir /path/to/skysense-vit-large-s1 \
--clean-output
```
The converter also accepts unified pretraining checkpoints with `backbone_gep.*` / `backbone_s2.*` / `backbone_s1.*` prefixes.
## Notes
- HR conversion skips Swin relative-position buffers (`relative_position_index`, `relative_coords_table`) and `mask_token`. Buffers are **deterministically recomputed** at init; learned CPB weights are loaded.
- ViT checkpoints use per-layer `ln1`/`ln2` keys remapped to `norm1`/`norm2`.
- Swin FFN keys `ffn.layers.0.0` → `ffn.layers.0`, `ffn.layers.1` → `ffn.layers.3`.
- HR Swin uses `pad_small_map=True` so 224×224 inputs work with window size 8 at deep stages.
|