README.md

---
license: apache-2.0
tags:
  - remote-sensing
  - earth-observation
  - skysense
  - feature-extraction
pipeline_tag: feature-extraction
---

# SkySense Transformers

HuggingFace-compatible checkpoints for the SkySense (CVPR 2024) foundation model backbones.

## Checkpoints

| Directory | Modality | Architecture | Source |
|-----------|----------|--------------|--------|
| `skysense-swinv2-huge-rgb` | High-res optical (RGB) | SwinV2 Huge | `skysense_model_backbone_hr.pth` |
| `skysense-vit-large-s2` | Sentinel-2 | ViT-Large | `skysense_model_backbone_s2.pth` |
| `skysense-vit-large-s1` | Sentinel-1 SAR | ViT-Large | `skysense_model_backbone_s1.pth` |

Each subdirectory is a self-contained HuggingFace model repo with remote code (`trust_remote_code=True`).

## Usage

```python
from transformers import pipeline
import torch

# HR RGB backbone — input 224×224
hr_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/SkySense-transformers/skysense-swinv2-huge-rgb",
    trust_remote_code=True,
    device="cpu",
)
hr_img = torch.randn(1, 3, 224, 224)
features = hr_pipe(hr_img)
print(features["last_hidden_state"].shape)  # (1, 2816, 7, 7)

# Sentinel-2 — 10 bands, 64×64
s2_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/SkySense-transformers/skysense-vit-large-s2",
    trust_remote_code=True,
    device="cpu",
)
s2_img = torch.randn(1, 10, 64, 64)
features = s2_pipe(s2_img)
print(features["last_hidden_state"].shape)  # (1, 1024, 16, 16)

# Sentinel-1 — VV/VH, 64×64
s1_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/SkySense-transformers/skysense-vit-large-s1",
    trust_remote_code=True,
    device="cpu",
)
s1_img = torch.randn(1, 2, 64, 64)
features = s1_pipe(s1_img)
print(features["last_hidden_state"].shape)
```

## Conversion

Source project: `/home/czy/local/projects/SkySense-transformers`

```bash
conda activate rsgen

python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysense_model_backbone_hr.pth \
  --modality hr \
  --output-dir /path/to/skysense-swinv2-huge-rgb \
  --clean-output

python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysense_model_backbone_s2.pth \
  --modality s2 \
  --output-dir /path/to/skysense-vit-large-s2 \
  --clean-output

python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysense_model_backbone_s1.pth \
  --modality s1 \
  --output-dir /path/to/skysense-vit-large-s1 \
  --clean-output
```

The converter also accepts unified pretraining checkpoints with `backbone_gep.*` / `backbone_s2.*` / `backbone_s1.*` prefixes.

## Notes

- HR conversion skips Swin relative-position buffers (`relative_position_index`, `relative_coords_table`) and `mask_token`. Buffers are **deterministically recomputed** at init; learned CPB weights are loaded.
- ViT checkpoints use per-layer `ln1`/`ln2` keys remapped to `norm1`/`norm2`.
- Swin FFN keys `ffn.layers.0.0` → `ffn.layers.0`, `ffn.layers.1` → `ffn.layers.3`.
- HR Swin uses `pad_small_map=True` so 224×224 inputs work with window size 8 at deep stages.