README.md

---
library_name: open_clip
base_model: apple/DFN2B-CLIP-ViT-L-14
tags:
  - vision-language
  - image-text-retrieval
  - planetary-science
  - remote-sensing
  - mars
  - geospatial
  - open-clip
---

# MarsScope-v1
MarsScope-v1 is a planetary vision-language model and aligns orbital imagery and scientific language in a shared embedding space, supporting text-to-image, image-to-image, and multimodal retrieval for label-free exploration of planetary surfaces. It is based on `ViT-L-14-quickgelu`, initialized from [CLIP-DFN2B](https://huggingface.co/apple/DFN2B-CLIP-ViT-L-14), and fine-tuned at 512 x 512 resolution on more than 200,000 curated pairs of planetary surface images and geomorphological descriptions.



## Model Details

- **Architecture:** `ViT-L-14-quickgelu`
- **Framework:** [OpenCLIP](https://github.com/mlfoundations/open_clip)
- **Initialization:** [CLIP-DFN2B](https://huggingface.co/apple/DFN2B-CLIP-ViT-L-14)
- **Input resolution:** 512 x 512
- **Parameters:** approximately 0.4B
- **Training objective:** contrastive image-text alignment

## Applications

The MarScope framework was introduced in [Natural Language-Driven Global Mapping of Martian Landforms](https://arxiv.org/abs/2601.15949) for natural language-driven global mapping, process-oriented geomorphological retrieval, and visual search for rare or previously unmapped features.

It was subsequently evaluated in [MarsRetrieval](https://arxiv.org/abs/2602.13961), which covers paired Martian image-text retrieval, fine-grained landform retrieval, and planetary-scale geo-localization.

## Usage

For large-scale retrieval, encode and normalize the image and text queries separately, then rank images by cosine similarity. To illustrate the effect of planetary-domain fine-tuning, we compare the image-text cosine similarities produced by the CLIP-DFN2B base model and MarsScope-v1:

| Central peak crater | Yardangs |
|:---:|:---:|
| ![Central peak crater](central_peak_crater.png) | ![Yardangs](yardangs.png) |


```bash
pip install torch open_clip_torch pillow huggingface_hub
```

```python
import open_clip
import torch
from huggingface_hub import hf_hub_download
from PIL import Image

repo_id = "claytonwang/MarsScope-v1"
model_id = f"hf-hub:{repo_id}"
device = "cuda" if torch.cuda.is_available() else "cpu"

model, _, preprocess = open_clip.create_model_and_transforms(model_id)
tokenizer = open_clip.get_tokenizer(model_id)
model = model.to(device).eval()

image_names = [
    "central_peak_crater.png",
    "yardangs.png",
]
text_prompts = [
    "a satellite image of a central peak crater on Mars",
    "a satellite image of yardangs on Mars",
]

image_paths = [
    hf_hub_download(repo_id=repo_id, filename=name)
    for name in image_names
]
images = torch.stack([
    preprocess(Image.open(path).convert("RGB"))
    for path in image_paths
]).to(device)
text = tokenizer(text_prompts).to(device)

with torch.inference_mode():
    image_features = model.encode_image(images, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    similarity = image_features @ text_features.T

# Rows: central_peak_crater.png, yardangs.png
# Columns: crater prompt, yardangs prompt
print(similarity)
```

## Examples

### CLIP-DFN2B

```text
tensor([[0.3569, 0.2822],
        [0.2729, 0.2723]])
```

### MarsScope-v1

```text
tensor([[0.4516, 0.0585],
        [0.0963, 0.3537]])
```

Compared with the base CLIP-DFN2B, MarsScope-v1 shows stronger alignment between Martian landform images and their matching descriptions while more clearly separating mismatched pairs. For comprehensive evaluation, refer to [MarsRetrieval](https://arxiv.org/abs/2602.13961).

## Limitations

MarsScope-v1 is intended for planetary-science research. Retrieval results may be affected by image resolution, illumination, spatial scale, geographic coverage, and the terminology used in a query. Model outputs should be validated by domain experts before scientific interpretation.

## Citation

```bibtex
@article{wang2026natural,
  title   = {Natural Language-Driven Global Mapping of Martian Landforms},
  author  = {Wang, Yiran and Wang, Shuoyuan and Wei, Zhaoran and Zhao, Jiannan
             and Yao, Zhonghua and Xie, Zejian and Zhang, Songxin and Huang, Jun
             and Jing, Bingyi and Wei, Hongxin},
  journal = {arXiv preprint arXiv:2601.15949},
  year    = {2026}
}

@article{wang2026marsretrieval,
  title   = {MarsRetrieval: Benchmarking Vision-Language Models for
             Planetary-Scale Geospatial Retrieval on Mars},
  author  = {Wang, Shuoyuan and Wang, Yiran and Wei, Hongxin},
  journal = {arXiv preprint arXiv:2602.13961},
  year    = {2026}
}
```