README.md

metadata

license: mit
language:
  - en
tags:
  - remote-sensing
  - earth-observation
  - self-supervised-learning
  - satellite
  - multispectral
  - feature-extraction
  - convnext
  - mae
  - mmearth
  - mp-mae
  - transformers
library_name: transformers
pipeline_tag: feature-extraction

MMEarth Transformers Models

Hugging Face–compatible checkpoints converted from the official MMEarth MP-MAE pretrained weights. Each subfolder is a standalone model repo layout (config.json, model.safetensors, preprocessor, and remote code) for geospatial feature extraction.

Model Description

These models are ConvNeXt V2 encoders pretrained with Multi Pretext Masked Autoencoding (MP-MAE) on the MMEarth multi-modal geospatial dataset. Checkpoints cover different pretext task configurations (all modalities, S2-only, RGB/BGR, image-level, pixel-level) and model sizes (atto, tiny).

All folders ship self-contained remote code (modeling_mmearth.py, processor, pipeline) and load with trust_remote_code=True.

Developed by: MMEarth Authors
Converted for Hugging Face by: BiliSakura
License (weights): MIT
Original paper: MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning (ECCV 2024)

Available checkpoints (10 models)

Folder	Input	Size	Dataset	Loss	Image	Patch	Ch
mmearth-convnextv2-atto-all-mod-1m-64-uncertainty-56x8	all_mod	atto	1M_64	uncertainty	56	8	12
mmearth-convnextv2-atto-all-mod-1m-64-unweighted-56x8	all_mod	atto	1M_64	unweighted	56	8	12
mmearth-convnextv2-atto-all-mod-1m-128-uncertainty-112x16	all_mod	atto	1M_128	uncertainty	112	16	12
mmearth-convnextv2-atto-all-mod-100k-128-uncertainty-112x16	all_mod	atto	100k_128	uncertainty	112	16	12
mmearth-convnextv2-tiny-all-mod-1m-64-uncertainty-56x8	all_mod	tiny	1M_64	uncertainty	56	8	12
mmearth-convnextv2-atto-s2-1m-64-uncertainty-56x8	S2	atto	1M_64	uncertainty	56	8	12
mmearth-convnextv2-atto-rgb-1m-64-uncertainty-56x8	rgb (BGR)	atto	1M_64	uncertainty	56	8	3
mmearth-convnextv2-atto-rgb-1m-128-uncertainty-112x16	rgb (BGR)	atto	1M_128	uncertainty	112	16	3
mmearth-convnextv2-atto-img-mod-1m-64-uncertainty-56x8	img_mod	atto	1M_64	uncertainty	56	8	12
mmearth-convnextv2-atto-pix-mod-1m-64-uncertainty-56x8	pix_mod	atto	1M_64	uncertainty	56	8	12

Legacy .pth filename mapping is in conversion_manifest.json.

Usage

Processors default to do_resize: false. Inputs keep native height and width. Apply per-band MMEarth normalization when you have dataset statistics (image_mean / image_std).

from transformers import pipeline
import numpy as np

MODEL = "/path/to/MMEarth-transformers/mmearth-convnextv2-atto-rgb-1m-64-uncertainty-56x8"

pipe = pipeline(
    task="mmearth-feature-extraction",
    model=MODEL,
    trust_remote_code=True,
)

# RGB/BGR: 3 bands at native size (56×56 for this checkpoint)
image = np.random.rand(56, 56, 3).astype(np.float32) * 1000
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 320])

12-band Sentinel-2 (all_mod / S2 checkpoints):

MODEL = "/path/to/MMEarth-transformers/mmearth-convnextv2-atto-all-mod-1m-64-uncertainty-56x8"
pipe = pipeline(task="mmearth-feature-extraction", model=MODEL, trust_remote_code=True)

image = np.random.rand(56, 56, 12).astype(np.float32) * 1000
features = pipe(image, pool=True, return_tensors=True)
print(features.shape)  # torch.Size([1, 320])

Dense spatial token map:

tokens = pipe(image, pool=False, return_tensors=True)
print(tokens.shape)  # [1, num_patches, hidden_size]

To resize to the pretraining reference size:

features = pipe(image, pool=True, return_tensors=True, image_processor_kwargs={"do_resize": True})

Load components directly:

from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained(MODEL, trust_remote_code=True)
processor = AutoImageProcessor.from_pretrained(MODEL, trust_remote_code=True)

Custom pipeline

Each checkpoint registers a custom pipeline in config.json:

"custom_pipelines": {
  "mmearth-feature-extraction": {
    "impl": "pipeline_mmearth.MMEarthImageFeatureExtractionPipeline",
    "pt": ["AutoModel"]
  }
}

This follows the HuggingFace custom pipeline pattern: remote code ships with the model folder, and trust_remote_code=True loads MMEarthImageFeatureExtractionPipeline, which extends the standard ImageFeatureExtractionPipeline with numpy array and file path support.

The built-in image-feature-extraction task also works:

pipe = pipeline(task="image-feature-extraction", model=MODEL, trust_remote_code=True)

Normalization

MMEarth pretraining normalizes each band with dataset-specific mean/std from data_*_band_stats.json. The converted preprocessor defaults to do_normalize: false because band statistics are not embedded in the legacy checkpoints. Provide your own image_mean / image_std when preprocessing:

features = pipe(
    image,
    pool=True,
    return_tensors=True,
    image_processor_kwargs={
        "do_normalize": True,
        "image_mean": [...],  # one value per channel
        "image_std": [...],
    },
)

RGB checkpoints were trained with BGR channel order (bands B4, B3, B2). The processor swaps RGB→BGR when channel_order="bgr".

Dependencies

• transformers, torch, timm, safetensors
• opencv-python (multispectral resize with more than 4 channels when do_resize=True)

Citation

@inproceedings{nedungadi2024mmearth,
  title={MMEarth: Exploring multi-modal pretext tasks for geospatial representation learning},
  author={Nedungadi, Vishal and Kariryaa, Ankit and Oehmcke, Stefan and Belongie, Serge and Igel, Christian and Lang, Nico},
  booktitle={European Conference on Computer Vision},
  pages={164--182},
  year={2024},
  organization={Springer}
}