---
language:
- en
license: mit
library_name: pytorch
tags:
- remote-sensing
- change-detection
- image-captioning
- multimodal
- retrieval
pipeline_tag: image-to-text
datasets:
- lcybuaa/LEVIR-CC
---

# RSICRC: Multimodal Remote Sensing Image Change Retrieval and Captioning

**RSICRC** is a multimodal foundation model designed for **bi-temporal remote sensing images**. It jointly performs **change captioning** (describing changes between two images) and **text-image retrieval** (finding image pairs that match a text description).

The model leverages Contrastive Learning and a decoupled decoder architecture to handle both tasks simultaneously.

## 📄 Paper
**Towards a Multimodal Framework for Remote Sensing Image Change Retrieval and Captioning** *Roger Ferrod, Luigi Di Caro, Dino Ienco* Published at **Discovery Science 2024**

[**Read the Paper**](https://doi.org/10.1007/978-3-031-78980-9_15) | [**GitHub Repository**](https://github.com/rogerferrod/RSICRC)

## 🏗️ Model Architecture
The framework is inspired by **CoCa** but adapted for bi-temporal remote sensing data.
* **Encoder:** A Siamese network (ResNet-50 or ViT via OpenCLIP) that encodes "before" and "after" images. A Hierarchical Self-Attention (HSA) block and a residual block with a cosine mask fuse the bi-temporal features.
* **Decoder:** A decoupled Transformer decoder split into:
    * **Unimodal Layers:** Encode text only (used for contrastive alignment).
    * **Multimodal Layers:** Apply cross-attention between visual and textual features to generate captions.

## 💻 Usage
To use this model use the custom source code.

### Inference Code

```python 
import torch
import json
import open_clip
from huggingface_hub import hf_hub_download

from src.model import ICCModel 

# 1. Download necessary files from Hugging Face
repo_id = "rogerferrod/RSICRC"
config_path = hf_hub_download(repo_id=repo_id, filename="config.json")
vocab_path = hf_hub_download(repo_id=repo_id, filename="levir_vocab.json") 
weights_path = hf_hub_download(repo_id=repo_id, filename="pytorch_model.bin")

# 2. Load Configuration and Vocabulary
with open(config_path, 'r') as f:
    config = json.load(f)

with open(vocab_path, 'r') as f:
    vocab = json.load(f)

# 3. Setup Device and Backbone (OpenCLIP)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
clip_model, _, preprocess = open_clip.create_model_and_transforms(config['backbone'])

# 4. Initialize the Model
model = ICCModel(
    device=device,
    clip=clip_model,
    backbone=config['backbone'],
    d_model=config['d_model'],
    vocab_size=len(vocab),
    max_len=config['max_len'],
    num_heads=config['num_heads'],
    h_dim=config['h_dim'],
    a_dim=config['a_dim'],
    encoder_layers=config['encoder_layers'],
    decoder_layers=config['decoder_layers'],
    dropout=config['dropout'],
    learnable=config['learnable'],
    fine_tune=config['fine_tune'],
    tie_embeddings=config['tie_embeddings'],
    prenorm=config['prenorm']
)

# 5. Load Weights
model.load_state_dict(torch.load(weights_path, map_location=device))
model = model.to(device)
model.eval()

print("Model loaded successfully!")
```

## 📚 Citation

If you use this model or code in your research, please cite our paper:

```bibtext
@InProceedings{10.1007/978-3-031-78980-9_15,
author       = {Roger Ferrod and
                  Luigi Di Caro and
                  Dino Ienco},
editor       = {Dino Pedreschi and
                Anna Monreale and
                Riccardo Guidotti and
                Roberto Pellungrini and
                Francesca Naretto},
title        = {Towards a Multimodal Framework for Remote Sensing Image Change Retrieval
                and Captioning},
booktitle    = {Discovery Science - 27th International Conference, {DS} 2024, Pisa,
                Italy, October 14-16, 2024, Proceedings, Part {II}},
series       = {Lecture Notes in Computer Science},
volume       = {15244},
pages        = {231--245},
publisher    = {Springer},
year         = {2024},
url          = {https://doi.org/10.1007/978-3-031-78980-9\_15},
doi          = {10.1007/978-3-031-78980-9\_15}
}
```