README.md

---
license: mit
language:
- en
tags:
- remote-sensing
- earth-observation
- self-supervised-learning
- satellite
- multispectral
- feature-extraction
- convnext
- mae
- mmearth
- mp-mae
- transformers
library_name: transformers
pipeline_tag: feature-extraction
---

# MMEarth Transformers Models

Hugging Face–compatible checkpoints converted from the official [MMEarth](https://arxiv.org/abs/2405.02771) MP-MAE pretrained weights. Each subfolder is a standalone model repo layout (`config.json`, `model.safetensors`, preprocessor, and remote code) for geospatial feature extraction.

## Model Description

These models are ConvNeXt V2 encoders pretrained with Multi Pretext Masked Autoencoding (MP-MAE) on the [MMEarth](https://github.com/vishalned/MMEarth-data) multi-modal geospatial dataset. Checkpoints cover different pretext task configurations (all modalities, S2-only, RGB/BGR, image-level, pixel-level) and model sizes (atto, tiny).

All folders ship self-contained remote code (`modeling_mmearth.py`, processor, pipeline) and load with `trust_remote_code=True`.

**Developed by:** [MMEarth Authors](https://github.com/vishalned/MMEarth-train)  
**Converted for Hugging Face by:** BiliSakura  
**License (weights):** MIT  
**Original paper:** [MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning](https://arxiv.org/abs/2405.02771) (ECCV 2024)

## Available checkpoints (10 models)

| Folder | Input | Size | Dataset | Loss | Image | Patch | Ch |
|--------|-------|------|---------|------|-------|-------|----|
| `mmearth-convnextv2-atto-all-mod-1m-64-uncertainty-56x8` | all_mod | atto | 1M_64 | uncertainty | 56 | 8 | 12 |
| `mmearth-convnextv2-atto-all-mod-1m-64-unweighted-56x8` | all_mod | atto | 1M_64 | unweighted | 56 | 8 | 12 |
| `mmearth-convnextv2-atto-all-mod-1m-128-uncertainty-112x16` | all_mod | atto | 1M_128 | uncertainty | 112 | 16 | 12 |
| `mmearth-convnextv2-atto-all-mod-100k-128-uncertainty-112x16` | all_mod | atto | 100k_128 | uncertainty | 112 | 16 | 12 |
| `mmearth-convnextv2-tiny-all-mod-1m-64-uncertainty-56x8` | all_mod | tiny | 1M_64 | uncertainty | 56 | 8 | 12 |
| `mmearth-convnextv2-atto-s2-1m-64-uncertainty-56x8` | S2 | atto | 1M_64 | uncertainty | 56 | 8 | 12 |
| `mmearth-convnextv2-atto-rgb-1m-64-uncertainty-56x8` | rgb (BGR) | atto | 1M_64 | uncertainty | 56 | 8 | 3 |
| `mmearth-convnextv2-atto-rgb-1m-128-uncertainty-112x16` | rgb (BGR) | atto | 1M_128 | uncertainty | 112 | 16 | 3 |
| `mmearth-convnextv2-atto-img-mod-1m-64-uncertainty-56x8` | img_mod | atto | 1M_64 | uncertainty | 56 | 8 | 12 |
| `mmearth-convnextv2-atto-pix-mod-1m-64-uncertainty-56x8` | pix_mod | atto | 1M_64 | uncertainty | 56 | 8 | 12 |

Legacy `.pth` filename mapping is in [`conversion_manifest.json`](conversion_manifest.json).

## Usage

Processors default to **`do_resize: false`**. Inputs keep native height and width. Apply per-band MMEarth normalization when you have dataset statistics (`image_mean` / `image_std`).

```python
from transformers import pipeline
import numpy as np

MODEL = "/path/to/MMEarth-transformers/mmearth-convnextv2-atto-rgb-1m-64-uncertainty-56x8"

pipe = pipeline(
    task="mmearth-feature-extraction",
    model=MODEL,
    trust_remote_code=True,
)

# RGB/BGR: 3 bands at native size (56×56 for this checkpoint)
image = np.random.rand(56, 56, 3).astype(np.float32) * 1000
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 320])
```

12-band Sentinel-2 (all_mod / S2 checkpoints):

```python
MODEL = "/path/to/MMEarth-transformers/mmearth-convnextv2-atto-all-mod-1m-64-uncertainty-56x8"
pipe = pipeline(task="mmearth-feature-extraction", model=MODEL, trust_remote_code=True)

image = np.random.rand(56, 56, 12).astype(np.float32) * 1000
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 320])
```

Dense spatial token map:

```python
tokens = pipe(image, pool=False, return_tensors=True)
print(tokens.shape)  # [1, num_patches, hidden_size]
```

To resize to the pretraining reference size:

```python
features = pipe(image, pool=True, return_tensors=True, image_processor_kwargs={"do_resize": True})
```

Load components directly:

```python
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(MODEL, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(MODEL, trust_remote_code=True)
```

## Custom pipeline

Each checkpoint registers a custom pipeline in `config.json`:

```json
"custom_pipelines": {
  "mmearth-feature-extraction": {
    "impl": "pipeline_mmearth.MMEarthImageFeatureExtractionPipeline",
    "pt": ["AutoModel"]
  }
}
```

This follows the [HuggingFace custom pipeline pattern](https://huggingface.co/docs/transformers/add_new_pipeline): remote code ships with the model folder, and `trust_remote_code=True` loads `MMEarthImageFeatureExtractionPipeline`, which extends the standard `ImageFeatureExtractionPipeline` with numpy array and file path support.

The built-in `image-feature-extraction` task also works:

```python
pipe = pipeline(task="image-feature-extraction", model=MODEL, trust_remote_code=True)
```

## Normalization

MMEarth pretraining normalizes each band with dataset-specific mean/std from `data_*_band_stats.json`. The converted preprocessor defaults to `do_normalize: false` because band statistics are not embedded in the legacy checkpoints. Provide your own `image_mean` / `image_std` when preprocessing:

```python
features = pipe(
    image,
    pool=True,
    return_tensors=True,
    image_processor_kwargs={
        "do_normalize": True,
        "image_mean": [...],  # one value per channel
        "image_std": [...],
    },
)
```

RGB checkpoints were trained with **BGR** channel order (bands B4, B3, B2). The processor swaps RGB→BGR when `channel_order="bgr"`.

## Dependencies

- `transformers`, `torch`, `timm`, `safetensors`
- `opencv-python` (multispectral resize with more than 4 channels when `do_resize=True`)

## Citation

```bibtex
@inproceedings{nedungadi2024mmearth,
  title={MMEarth: Exploring multi-modal pretext tasks for geospatial representation learning},
  author={Nedungadi, Vishal and Kariryaa, Ankit and Oehmcke, Stefan and Belongie, Serge and Igel, Christian and Lang, Nico},
  booktitle={European Conference on Computer Vision},
  pages={164--182},
  year={2024},
  organization={Springer}
}
```