Update README.md

README.md

@@ -1,65 +1,113 @@
 ---
-license: etalab-2.0
+license: apache-2.0
 pipeline_tag: image-segmentation
 tags:
-- semantic segmentation
 - pytorch
-- landcover
+- pytorch lightning
+- remote sensing
+- earth observation
+- self-supervised learning
+- masked autoencoders
+- transformers
+- multimodal
+- multitemporal
 library_name: pytorch
 datasets:
-- IGNF/FLAIR-HUB
+- ignf/flair-hub
 ---
 
-We introduce **MAESTRO**, a tailored adaptation of the Masked Autoencoder (MAE) framework that effectively orchestrates the use of multimodal, multitemporal, and multispectral Earth Observation (EO) data. Evaluated on four EO datasets, MAESTRO sets a new state-of-the-art on tasks that strongly rely on multitemporal dynamics, while remaining highly competitive on tasks dominated by a single monotemporal modality.
+## Download
 
-Our contributions are as follows:
+⚖️ [**Model weights**](checkpoints/pretrain-epoch=99.ckpt) <br>
+⚙️ [**Model configuration**](.hydra/config_resolved.yaml) <br>
+📂 [**Dataset splits**](dataset_splits) <br>
+
+## Abstract
+
+**MAESTRO** is a tailored adaptation of the Masked Autoencoder (MAE) framework that effectively orchestrates the use of multimodal, multitemporal, and multispectral Earth Observation (EO) data. Evaluated on four EO datasets, MAESTRO sets a new state-of-the-art on tasks that strongly rely on multitemporal dynamics, while remaining highly competitive on tasks dominated by a single monotemporal modality.
+
+MAESTRO's contributions are as follows:
 - **Extensive benchmarking of multimodal and multitemporal SSL:** Impact evaluation of various fusion strategies for multimodal and multitemporal SSL.
 - **Patch-group-wise normalization:** Novel normalization scheme that normalizes reconstruction targets patch-wise within groups of highly correlated spectral bands.
 - **MAESTRO:** Novel adaptation of the MAE that combines optimized fusion strategies with our tailored patch-group-wise normalization..
 
 <div style="position: relative; text-align: center;">
-    <img src="./media/Maestro_Overview.png" alt="Classes distribution." style="width: 100%; display: block; margin: 0 auto;"/>
+    <img src="./media/Maestro_Overview.png" style="width: 100%; display: block; margin: 0 auto;"/>
 </div>
 
 
+📃 **Paper:** https://arxiv.org/abs/2508.10894 <br>
+💻 **Code repository:** https://github.com/IGNF/MAESTRO <br>
 
 
 
-💻 **Code repository:** https://github.com/IGNF/MAESTRO<br>
-📃 **Paper:** https://arxiv.org/abs/2508.10894
+## Pre-training
 
-<hr>
 
+This model is pre-trained on [FLAIR-HUB](https://huggingface.co/datasets/IGNF/FLAIR-HUB)
+
+FLAIR-HUB contains 241,100 tiles of size 102.4 × 102.4 m, covering a total area of 2,528 km² across France.
 
-## Pre-training Dataset
+We retain six distinct modalities:
+  - Aerial imagery RGB + NIR (0.2 m resolution)
+  - DEM/DSM imagery (0.2 m resolution)
+  - SPOT 6–7 imagery
+  - Sentinel-1 time series in ascending orbit
+  - Sentinel-1 time series in descending orbit
+  - Sentinel-2 time series
 
+Below is the reconstruction loss during pre-training on the combined training and validation ensembles, using patch-group-wise normalization and modality-weighted averaging proportional to token counts.
+
+<div style="position: relative; text-align: center;">
+    <img src="./media/Reconstruction_Loss.png" style="width: 100%; display: block; margin: 0 auto;"/>
+</div>
 
 <hr>
 
-## 🔎 Cross-dataset Evaluation
 
-Benchmark results on 4 datasets :
+## Fine-tuning
+
+For optimal fine-tuning results with this model:
+- Ensure that patch sizes and channels match between pre-training and fine-tuning for each modality:
+  - Modality "aerial":
+    - Patch size: 16
+    - Channels: NIR, RED, GREEN, BLUE
+  - Modality "spot":
+    - Patch size: 16
+    - Channels: RED, GREEN, BLUE
+  - Modality "s1":
+    - Patch size: 2
+    - Channels: VV, VH
+  - Modality "s2":
+    - Patch size: 2
+    - Channels: B02, B03, B04, B05, B06, B07, B08, B8A, B11, B12
+- Use fixed cross-dataset grids for positional encodings proportional to ground sampling distance: `grid_pos_enc` ≈ 1.6 * `crop_meters`
+
+Note that modality names must match between pre-training and fine-tuning. 
+
+Below are cross-dataset evaluation results obtained with these guidelines on TreeSatAI-TS, PASTIS-HD.
 
 <p align="center">
 
-| Model              | Pre-training dataset | TreeSatAI-TS | PASTIS-HD | FLAIR#2 | FLAIR-HUB |
-|--------------------|-----------------------|--------------|-----------|---------|-----------|
-| MAESTRO (ours)     | FLAIR-HUB             | **79.6**     | **68.0**  | -       | -         |
-| MAESTRO (ours)     | S2-NAIP urban         | 78.8         | 67.4      | 62.6    | 64.6      |
-| DINO-v2            | LVD-142M              | 76.7         | 64.4      | **64.2**| 66.0      |
-| DINO-v2 sat.       | Maxar Vivid2          | 76.3         | 64.0      | 63.5    | **66.0**  |
-| DOFA               | DOFA MM               | 76.0         | 62.9      | 62.3    | 65.1      |
-| CROMA              | SSL4EO                | 70.5         | 65.0      | 39.0    | 44.3      |
-| Prithvi-EO-2.0     | HLS                   | 75.6         | 66.2      | 41.8    | 44.9      |
-| SatMAE             | fMoW RGB+S            | 76.9         | 66.6      | 42.5    | 45.0      |
+| Model              | Pre-training dataset | TreeSatAI-TS | PASTIS-HD | 
+|--------------------|-----------------------|--------------|-----------|
+| MAESTRO (ours)     | FLAIR-HUB             | **79.6**     | **68.0**  |
+| DINO-v2            | LVD-142M              | 76.7         | 64.4      |
+| DINO-v2 sat.       | Maxar Vivid2          | 76.3         | 64.0      |
+| DOFA               | DOFA MM               | 76.0         | 62.9      |
+| CROMA              | SSL4EO                | 70.5         | 65.0      |
+| Prithvi-EO-2.0     | HLS                   | 75.6         | 66.2      |
+| SatMAE             | fMoW RGB+S            | 76.9         | 66.6      |
 </p>
 
 
-<hr>
+## 🚀 Getting started
 
-## 🚀 Getting Started
+Prerequisites: 
+- Fetch [Dataset splits](dataset_splits) and move them to each dataset directory
+- Fetch [model weights](checkpoints/pretrain-epoch=99.ckpt) and [model configuration](.hydra/config_resolved.yaml) and move them to `/path/to/experiments/`
 
-First, set up the module with [Poetry](https://python-poetry.org/).
+The module is setup with [Poetry](https://python-poetry.org/).
 
 ```bash
 # 1. Change directory
@@ -69,67 +117,111 @@ cd MAESTRO
 poetry install
 ```
 
-Then, you can start from the following minimal examples.
-
-Intra-dataset MAESTRO on TreeSatAI-TS:
+Pre-training on FLAIR-HUB is performed using:
 ```bash
-# pre-train, probe and finetune on TreeSatAI-TS
 poetry run python main.py \
-        model.model=mae model.model_size=medium \
-        opt_pretrain.epochs=100 opt_probe.epochs=10 opt_finetune.epochs=50 \
-        datasets.name_dataset=treesatai_ts \
-        datasets.root_dir=/path/to/dataset/dir datasets.treesatai_ts.rel_dir=TreeSatAI-TS \
-        run.exp_dir=/path/to/experiments/dir run.exp_name=mae-m_treesat
+  model.model=mae model.model_size=medium \
+  model.fusion_mode=group model.inter_depth=3 \
+  opt_pretrain.epochs=100 opt_probe.epochs=0 opt_finetune.epochs=0 \
+  opt_pretrain.batch_size=144 \  # batch size 9 on 4 nodes with 4 GPUs per node
+  datasets.name_dataset=flair \
+  datasets.flair.filter_inputs=[aerial,dem,spot,s2,s1_asc,s1_des] \
+  datasets.flair.crop_meters=102.4 datasets.flair.grid_pos_enc=160 \ 
+  datasets.flair.aerial.image_size=512 datasets.flair.aerial.patch_size.mae=16 \
+  datasets.flair.dem.image_size=512 datasets.flair.dem.patch_size.mae=32 \
+  datasets.flair.spot.image_size=128 datasets.flair.spot.patch_size.mae=16 datasets.flair.spot.bands=3 \
+  datasets.flair.s2.image_size=10 datasets.flair.s2.patch_size.mae=2 \
+  datasets.flair.s1_asc.image_size=10 datasets.flair.s1_asc.patch_size.mae=2 \
+  datasets.flair.s1_des.image_size=10 datasets.flair.s1_des.patch_size.mae=2 \
+  datasets.root_dir=/path/to/dataset/dir datasets.flair.csv_dir=/path/to/dataset/dir/FLAIR-HUB datasets.flair.rel_dir=FLAIR-HUB \
+  run.exp_dir=/path/to/experiments/dir run.exp_name=MAESTRO_FLAIR-HUB_base \
 ```
 
-Intra-dataset MAESTRO on PASTIS-HD:
+Fine-tuning on TreeSatAI-TS:
 ```bash
-# pre-train, probe and finetune on PASTIS-HD
 poetry run python main.py \
-        model.model=mae model.model_size=medium \
-        opt_pretrain.epochs=100 opt_probe.epochs=10 opt_finetune.epochs=50 \
-        datasets.name_dataset=pastis_hd \
-        datasets.root_dir=/path/to/dataset/dir datasets.pastis_hd.rel_dir=PASTIS-HD \
-        run.exp_dir=/path/to/experiments/dir run.exp_name=mae-m_pastis
+  model.model=mae model.model_size=medium \
+  model.fusion_mode=group model.inter_depth=3 \
+  opt_pretrain.epochs=0 opt_probe.epochs=10 opt_finetune.epochs=50 \
+  opt_probe.batch_size=96 opt_finetune.batch_size=96 \  # batch size 32 on 1 node with 3 GPUs per node
+  opt_finetune.monitor=treesat_mlc_thresh/weighted_f1_val \
+  datasets.name_dataset=treesatai_ts \
+  datasets.treesatai_ts.filter_inputs=[aerial,s2,s1_asc,s1_des] \
+  datasets.treesatai_ts.crop_meters=60 datasets.treesatai_ts.grid_pos_enc=96 \ 
+  datasets.treesatai_ts.aerial.image_size=240 datasets.treesatai_ts.aerial.patch_size.mae=16 \
+  datasets.treesatai_ts.s2.image_size=6 datasets.treesatai_ts.s2.patch_size.mae=2 \
+  datasets.treesatai_ts.s1_asc.image_size=6 datasets.treesatai_ts.s1_asc.patch_size.mae=2 \
+  datasets.treesatai_ts.s1_des.image_size=6 datasets.treesatai_ts.s1_des.patch_size.mae=2 \
+  datasets.root_dir=/path/to/dataset/dir datasets.treesatai_ts.rel_dir=TreeSatAI-TS \
+  run.exp_dir=/path/to/experiments/dir run.exp_name=MAESTRO_FLAIR-HUB-x-TSAI-TS_base \
+  run.load_name=MAESTRO_FLAIR-HUB_base  # load pre-trained model
 ```
 
-Intra-dataset MAESTRO on FLAIR-HUB:
+Fine-tuning on PASTIS-HD:
 ```bash
-# pre-train, probe and finetune on FLAIR-HUB
 poetry run python main.py \
-        model.model=mae model.model_size=medium \
-        opt_pretrain.epochs=100 opt_probe.epochs=15 opt_finetune.epochs=100 \
-        datasets.name_dataset=flair \
-        datasets.root_dir=/path/to/dataset/dir datasets.flair.rel_dir=FLAIR-HUB \
-        run.exp_dir=/path/to/experiments/dir run.exp_name=mae-m_flair
+  model.model=mae model.model_size=medium \
+  model.fusion_mode=group model.inter_depth=3 \
+  opt_pretrain.epochs=0 opt_probe.epochs=10 opt_finetune.epochs=50 \
+  opt_probe.batch_size=48 opt_finetune.batch_size=48 \  # batch size 16 on 1 node with 3 GPUs per node
+  opt_finetune.monitor=pastis_seg/average_iou_val \
+  datasets.name_dataset=pastis_hd \
+  datasets.pastis_hd.filter_inputs=[spot,s2,s1_asc,s1_des] \
+  datasets.pastis_hd.crop_meters=160 datasets.pastis_hd.grid_pos_enc=256 datasets.pastis_hd.repeats=8 \ 
+  datasets.pastis_hd.spot.image_size=160 datasets.pastis_hd.spot.patch_size.mae=16 \
+  datasets.pastis_hd.s2.image_size=16 datasets.pastis_hd.s2.patch_size.mae=2 \
+  datasets.pastis_hd.s1_asc.image_size=16 datasets.pastis_hd.s1_asc.patch_size.mae=2 \
+  datasets.pastis_hd.s1_des.image_size=16 datasets.pastis_hd.s1_des.patch_size.mae=2 \
+  datasets.root_dir=/path/to/dataset/dir datasets.pastis_hd.rel_dir=PASTIS-HD \
+  run.exp_dir=/path/to/experiments/dir run.exp_name=MAESTRO_FLAIR-HUB-x-PASTIS-HD_base \
+  run.load_name=MAESTRO_FLAIR-HUB_base  # load pre-trained model
 ```
 
-Cross-dataset MAESTRO from S2-NAIP urban to TreeSatAI-TS:
+
+Fine-tuning on FLAIR#2:
 ```bash
-# pre-train on S2-NAIP urban
 poetry run python main.py \
-        model.model=mae model.model_size=medium \
-        opt_pretrain.epochs=15 opt_probe.epochs=0 opt_finetune.epochs=0 \
-        datasets.name_dataset=s2_naip \
-        datasets.root_dir=/path/to/dataset/dir datasets.s2_naip.rel_dir=s2-naip-urban \
-        run.exp_dir=/path/to/experiments/dir run.exp_name=mae-m_s2-naip && \
-# probe and finetune on TreeSatAI-TS
-poetry run python main.py \
-        model.model=mae model.model_size=medium \
-        opt_pretrain.epochs=0 opt_probe.epochs=10 opt_finetune.epochs=50 \
-        datasets.name_dataset=treesatai_ts \
-        datasets.treesatai_ts.aerial.image_size=240 datasets.treesatai_ts.aerial.patch_size.mae=16 \
-        datasets.treesatai_ts.s1_asc.name_embed=s1 datasets.treesatai_ts.s1_des.name_embed=s1 \
-        datasets.root_dir=/path/to/dataset/dir datasets.treesatai_ts.rel_dir=TreeSatAI-TS \
-        run.exp_dir=/path/to/experiments/dir run.load_name=mae-m_s2-naip run.exp_name=mae-m_s2-naip-x-treesat
+  model.model=mae model.model_size=medium \
+  model.fusion_mode=group model.inter_depth=3 \
+  opt_pretrain.epochs=0 opt_probe.epochs=15 opt_finetune.epochs=100 \
+  opt_probe.batch_size=48 opt_finetune.batch_size=48 \  # batch size 6 on 2 nodes with 4 GPUs per node
+  opt_finetune.monitor=cosia/average_iou_val \
+  datasets.name_dataset=flair \
+  datasets.flair.version=flair2 \
+  datasets.flair.filter_inputs=[aerial,s2] \
+  datasets.flair.crop_meters=102.4 datasets.flair.grid_pos_enc=160 \ 
+  datasets.flair.aerial.image_size=512 datasets.flair.aerial.patch_size.mae=16 \
+  datasets.flair.s2.image_size=10 datasets.flair.s2.patch_size.mae=2 \
+  datasets.root_dir=/path/to/dataset/dir datasets.flair.csv_dir=/path/to/dataset/dir/FLAIR-HUB datasets.flair.rel_dir=FLAIR-HUB \
+  run.exp_dir=/path/to/experiments/dir run.exp_name=MAESTRO_FLAIR-HUB-x-FLAIR2_base \
+  run.load_name=MAESTRO_FLAIR-HUB_base  # load pre-trained model
+```
 
+Fine-tuning on FLAIR-HUB:
+```bash
+poetry run python main.py \
+  model.model=mae model.model_size=medium \
+  model.fusion_mode=group model.inter_depth=3 \
+  opt_pretrain.epochs=0 opt_probe.epochs=15 opt_finetune.epochs=100 \
+  opt_probe.batch_size=96 opt_finetune.batch_size=96 \  # batch size 6 on 4 nodes with 4 GPUs per node
+  opt_finetune.monitor=cosia/average_iou_val \
+  datasets.name_dataset=flair \
+  datasets.flair.filter_inputs=[aerial,s2,s1_asc,s1_des] \
+  datasets.flair.crop_meters=102.4 datasets.flair.grid_pos_enc=160 \ 
+  datasets.flair.aerial.image_size=512 datasets.flair.aerial.patch_size.mae=16 \
+  datasets.flair.s2.image_size=10 datasets.flair.s2.patch_size.mae=2 \
+  datasets.flair.s1_asc.image_size=10 datasets.flair.s1_asc.patch_size.mae=2 \
+  datasets.flair.s1_des.image_size=10 datasets.flair.s1_des.patch_size.mae=2 \
+  datasets.root_dir=/path/to/dataset/dir datasets.flair.csv_dir=/path/to/dataset/dir/FLAIR-HUB datasets.flair.rel_dir=FLAIR-HUB \
+  run.exp_dir=/path/to/experiments/dir run.exp_name=MAESTRO_FLAIR-HUB-x-FLAIR-HUB_base \
+  run.load_name=MAESTRO_FLAIR-HUB_base  # load pre-trained model
 ```
 
 <hr>
 
 ## Reference
 
-If you use this code, please cite:
+If you use this model, please cite:
 
 ```bibtex
 @article{labatie2025maestro,
@@ -144,4 +236,4 @@ If you use this code, please cite:
 
 ## Acknowledgement
 
-The experiments in the paper were conducted using HPC/AI resources from GENCI-IDRIS (allocations A0181013803, A0161013803, and AD010114597R1).
+The experiments in the paper were conducted using HPC/AI resources from GENCI-IDRIS (allocations A0181013803, A0161013803, AD010114597R1, and AD011014690R1).