metadata
license: mit
datasets:
- rsi/PixelsPointsPolygons
language:
- en
metrics:
- accuracy
base_model:
- timm/vit_small_patch8_224.dino
pipeline_tag: object-detection
tags:
- building
- vectorization
- polygon
- aerial
- image
- pointcloud
- multimodal
The P3 dataset: Pixels, Points and Polygons
for Multimodal Building Vectorization
Raphael Sulzer1,2 Liuyun Duan1 Nicolas Girard1 Florent Lafarge2
1LuxCarta Technology2Centre Inria d'Université Côte d'Azur
Figure 1: A view of our dataset of Zurich, Switzerland
Abstract
We present the P3 dataset, a large-scale multimodal benchmark for building vectorization, constructed from aerial LiDAR point clouds, high-resolution aerial imagery, and vectorized 2D building outlines, collected across three continents. The dataset contains over 10 billion LiDAR points with decimeter-level accuracy and RGB images at a ground sampling distance of 25 cm. While many existing datasets primarily focus on the image modality, P3 offers a complementary perspective by also incorporating dense 3D information. We demonstrate that LiDAR point clouds serve as a robust modality for predicting building polygons, both in hybrid and end-to-end learning frameworks. Moreover, fusing aerial LiDAR and imagery further improves accuracy and geometric quality of predicted polygons. The P3 dataset is publicly available, along with code and pretrained weights of three state-of-the-art models for building polygon prediction at https://github.com/raphaelsulzer/PixelsPointsPolygons.
Highlights
- A global, multimodal dataset of aerial images, aerial LiDAR point clouds and building outline polygons, available at huggingface.co/datasets/rsi/PixelsPointsPolygons
- A library for training and evaluating state-of-the-art deep learning methods on the dataset, available at github.com/raphaelsulzer/PixelsPointsPolygons
- Pretrained model weights, available at huggingface.co/rsi/PixelsPointsPolygons
Dataset
Overview
Download
git lfs install
git clone https://huggingface.co/datasets/rsi/PixelsPointsPolygons $DATA_ROOT
Structure
📁 Click to expand folder structure
PixelsPointsPolygons/data/224
├── annotations
│ ├── annotations_all_test.json
│ ├── annotations_all_train.json
│ └── annotations_all_val.json
│ ... (24 files total)
├── images
│ ├── train
│ │ ├── CH
│ │ │ ├── 0
│ │ │ │ ├── image0_CH_train.tif
│ │ │ │ ├── image1000_CH_train.tif
│ │ │ │ └── image1001_CH_train.tif
│ │ │ │ ... (5000 files total)
│ │ │ ├── 5000
│ │ │ │ ├── image5000_CH_train.tif
│ │ │ │ ├── image5001_CH_train.tif
│ │ │ │ └── image5002_CH_train.tif
│ │ │ │ ... (5000 files total)
│ │ │ └── 10000
│ │ │ ├── image10000_CH_train.tif
│ │ │ ├── image10001_CH_train.tif
│ │ │ └── image10002_CH_train.tif
│ │ │ ... (5000 files total)
│ │ │ ... (11 dirs total)
│ │ ├── NY
│ │ │ ├── 0
│ │ │ │ ├── image0_NY_train.tif
│ │ │ │ ├── image1000_NY_train.tif
│ │ │ │ └── image1001_NY_train.tif
│ │ │ │ ... (5000 files total)
│ │ │ ├── 5000
│ │ │ │ ├── image5000_NY_train.tif
│ │ │ │ ├── image5001_NY_train.tif
│ │ │ │ └── image5002_NY_train.tif
│ │ │ │ ... (5000 files total)
│ │ │ └── 10000
│ │ │ ├── image10000_NY_train.tif
│ │ │ ├── image10001_NY_train.tif
│ │ │ └── image10002_NY_train.tif
│ │ │ ... (5000 files total)
│ │ │ ... (11 dirs total)
│ │ └── NZ
│ │ ├── 0
│ │ │ ├── image0_NZ_train.tif
│ │ │ ├── image1000_NZ_train.tif
│ │ │ └── image1001_NZ_train.tif
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── image5000_NZ_train.tif
│ │ │ ├── image5001_NZ_train.tif
│ │ │ └── image5002_NZ_train.tif
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── image10000_NZ_train.tif
│ │ ├── image10001_NZ_train.tif
│ │ └── image10002_NZ_train.tif
│ │ ... (5000 files total)
│ │ ... (11 dirs total)
│ ├── val
│ │ ├── CH
│ │ │ └── 0
│ │ │ ├── image0_CH_val.tif
│ │ │ ├── image100_CH_val.tif
│ │ │ └── image101_CH_val.tif
│ │ │ ... (529 files total)
│ │ ├── NY
│ │ │ └── 0
│ │ │ ├── image0_NY_val.tif
│ │ │ ├── image100_NY_val.tif
│ │ │ └── image101_NY_val.tif
│ │ │ ... (529 files total)
│ │ └── NZ
│ │ └── 0
│ │ ├── image0_NZ_val.tif
│ │ ├── image100_NZ_val.tif
│ │ └── image101_NZ_val.tif
│ │ ... (529 files total)
│ └── test
│ ├── CH
│ │ ├── 0
│ │ │ ├── image0_CH_test.tif
│ │ │ ├── image1000_CH_test.tif
│ │ │ └── image1001_CH_test.tif
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── image5000_CH_test.tif
│ │ │ ├── image5001_CH_test.tif
│ │ │ └── image5002_CH_test.tif
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── image10000_CH_test.tif
│ │ ├── image10001_CH_test.tif
│ │ └── image10002_CH_test.tif
│ │ ... (4400 files total)
│ ├── NY
│ │ ├── 0
│ │ │ ├── image0_NY_test.tif
│ │ │ ├── image1000_NY_test.tif
│ │ │ └── image1001_NY_test.tif
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── image5000_NY_test.tif
│ │ │ ├── image5001_NY_test.tif
│ │ │ └── image5002_NY_test.tif
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── image10000_NY_test.tif
│ │ ├── image10001_NY_test.tif
│ │ └── image10002_NY_test.tif
│ │ ... (4400 files total)
│ └── NZ
│ ├── 0
│ │ ├── image0_NZ_test.tif
│ │ ├── image1000_NZ_test.tif
│ │ └── image1001_NZ_test.tif
│ │ ... (5000 files total)
│ ├── 5000
│ │ ├── image5000_NZ_test.tif
│ │ ├── image5001_NZ_test.tif
│ │ └── image5002_NZ_test.tif
│ │ ... (5000 files total)
│ └── 10000
│ ├── image10000_NZ_test.tif
│ ├── image10001_NZ_test.tif
│ └── image10002_NZ_test.tif
│ ... (4400 files total)
├── lidar
│ ├── train
│ │ ├── CH
│ │ │ ├── 0
│ │ │ │ ├── lidar0_CH_train.copc.laz
│ │ │ │ ├── lidar1000_CH_train.copc.laz
│ │ │ │ └── lidar1001_CH_train.copc.laz
│ │ │ │ ... (5000 files total)
│ │ │ ├── 5000
│ │ │ │ ├── lidar5000_CH_train.copc.laz
│ │ │ │ ├── lidar5001_CH_train.copc.laz
│ │ │ │ └── lidar5002_CH_train.copc.laz
│ │ │ │ ... (5000 files total)
│ │ │ └── 10000
│ │ │ ├── lidar10000_CH_train.copc.laz
│ │ │ ├── lidar10001_CH_train.copc.laz
│ │ │ └── lidar10002_CH_train.copc.laz
│ │ │ ... (5000 files total)
│ │ │ ... (11 dirs total)
│ │ ├── NY
│ │ │ ├── 0
│ │ │ │ ├── lidar0_NY_train.copc.laz
│ │ │ │ ├── lidar10_NY_train.copc.laz
│ │ │ │ └── lidar1150_NY_train.copc.laz
│ │ │ │ ... (1071 files total)
│ │ │ ├── 5000
│ │ │ │ ├── lidar5060_NY_train.copc.laz
│ │ │ │ ├── lidar5061_NY_train.copc.laz
│ │ │ │ └── lidar5062_NY_train.copc.laz
│ │ │ │ ... (2235 files total)
│ │ │ └── 10000
│ │ │ ├── lidar10000_NY_train.copc.laz
│ │ │ ├── lidar10001_NY_train.copc.laz
│ │ │ └── lidar10002_NY_train.copc.laz
│ │ │ ... (4552 files total)
│ │ │ ... (11 dirs total)
│ │ └── NZ
│ │ ├── 0
│ │ │ ├── lidar0_NZ_train.copc.laz
│ │ │ ├── lidar1000_NZ_train.copc.laz
│ │ │ └── lidar1001_NZ_train.copc.laz
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── lidar5000_NZ_train.copc.laz
│ │ │ ├── lidar5001_NZ_train.copc.laz
│ │ │ └── lidar5002_NZ_train.copc.laz
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── lidar10000_NZ_train.copc.laz
│ │ ├── lidar10001_NZ_train.copc.laz
│ │ └── lidar10002_NZ_train.copc.laz
│ │ ... (4999 files total)
│ │ ... (11 dirs total)
│ ├── val
│ │ ├── CH
│ │ │ └── 0
│ │ │ ├── lidar0_CH_val.copc.laz
│ │ │ ├── lidar100_CH_val.copc.laz
│ │ │ └── lidar101_CH_val.copc.laz
│ │ │ ... (529 files total)
│ │ ├── NY
│ │ │ └── 0
│ │ │ ├── lidar0_NY_val.copc.laz
│ │ │ ├── lidar100_NY_val.copc.laz
│ │ │ └── lidar101_NY_val.copc.laz
│ │ │ ... (529 files total)
│ │ └── NZ
│ │ └── 0
│ │ ├── lidar0_NZ_val.copc.laz
│ │ ├── lidar100_NZ_val.copc.laz
│ │ └── lidar101_NZ_val.copc.laz
│ │ ... (529 files total)
│ └── test
│ ├── CH
│ │ ├── 0
│ │ │ ├── lidar0_CH_test.copc.laz
│ │ │ ├── lidar1000_CH_test.copc.laz
│ │ │ └── lidar1001_CH_test.copc.laz
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── lidar5000_CH_test.copc.laz
│ │ │ ├── lidar5001_CH_test.copc.laz
│ │ │ └── lidar5002_CH_test.copc.laz
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── lidar10000_CH_test.copc.laz
│ │ ├── lidar10001_CH_test.copc.laz
│ │ └── lidar10002_CH_test.copc.laz
│ │ ... (4400 files total)
│ ├── NY
│ │ ├── 0
│ │ │ ├── lidar0_NY_test.copc.laz
│ │ │ ├── lidar1000_NY_test.copc.laz
│ │ │ └── lidar1001_NY_test.copc.laz
│ │ │ ... (4964 files total)
│ │ ├── 5000
│ │ │ ├── lidar5000_NY_test.copc.laz
│ │ │ ├── lidar5001_NY_test.copc.laz
│ │ │ └── lidar5002_NY_test.copc.laz
│ │ │ ... (4953 files total)
│ │ └── 10000
│ │ ├── lidar10000_NY_test.copc.laz
│ │ ├── lidar10001_NY_test.copc.laz
│ │ └── lidar10002_NY_test.copc.laz
│ │ ... (4396 files total)
│ └── NZ
│ ├── 0
│ │ ├── lidar0_NZ_test.copc.laz
│ │ ├── lidar1000_NZ_test.copc.laz
│ │ └── lidar1001_NZ_test.copc.laz
│ │ ... (5000 files total)
│ ├── 5000
│ │ ├── lidar5000_NZ_test.copc.laz
│ │ ├── lidar5001_NZ_test.copc.laz
│ │ └── lidar5002_NZ_test.copc.laz
│ │ ... (5000 files total)
│ └── 10000
│ ├── lidar10000_NZ_test.copc.laz
│ ├── lidar10001_NZ_test.copc.laz
│ └── lidar10002_NZ_test.copc.laz
│ ... (4400 files total)
└── ffl
├── train
│ ├── CH
│ │ ├── 0
│ │ │ ├── image0_CH_train.pt
│ │ │ ├── image1000_CH_train.pt
│ │ │ └── image1001_CH_train.pt
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── image5000_CH_train.pt
│ │ │ ├── image5001_CH_train.pt
│ │ │ └── image5002_CH_train.pt
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── image10000_CH_train.pt
│ │ ├── image10001_CH_train.pt
│ │ └── image10002_CH_train.pt
│ │ ... (5000 files total)
│ │ ... (11 dirs total)
│ ├── NY
│ │ ├── 0
│ │ │ ├── image0_NY_train.pt
│ │ │ ├── image1000_NY_train.pt
│ │ │ └── image1001_NY_train.pt
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── image5000_NY_train.pt
│ │ │ ├── image5001_NY_train.pt
│ │ │ └── image5002_NY_train.pt
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── image10000_NY_train.pt
│ │ ├── image10001_NY_train.pt
│ │ └── image10002_NY_train.pt
│ │ ... (5000 files total)
│ │ ... (11 dirs total)
│ ├── NZ
│ │ ├── 0
│ │ │ ├── image0_NZ_train.pt
│ │ │ ├── image1000_NZ_train.pt
│ │ │ └── image1001_NZ_train.pt
│ │ │ ... (5000 files total)
│ │ ├── 5000
│ │ │ ├── image5000_NZ_train.pt
│ │ │ ├── image5001_NZ_train.pt
│ │ │ └── image5002_NZ_train.pt
│ │ │ ... (5000 files total)
│ │ └── 10000
│ │ ├── image10000_NZ_train.pt
│ │ ├── image10001_NZ_train.pt
│ │ └── image10002_NZ_train.pt
│ │ ... (5000 files total)
│ │ ... (11 dirs total)
│ ├── processed-flag-all
│ ├── processed-flag-CH
│ └── processed-flag-NY
│ ... (8 files total)
├── val
│ ├── CH
│ │ └── 0
│ │ ├── image0_CH_val.pt
│ │ ├── image100_CH_val.pt
│ │ └── image101_CH_val.pt
│ │ ... (529 files total)
│ ├── NY
│ │ └── 0
│ │ ├── image0_NY_val.pt
│ │ ├── image100_NY_val.pt
│ │ └── image101_NY_val.pt
│ │ ... (529 files total)
│ ├── NZ
│ │ └── 0
│ │ ├── image0_NZ_val.pt
│ │ ├── image100_NZ_val.pt
│ │ └── image101_NZ_val.pt
│ │ ... (529 files total)
│ ├── processed-flag-all
│ ├── processed-flag-CH
│ └── processed-flag-NY
│ ... (8 files total)
└── test
├── CH
│ ├── 0
│ │ ├── image0_CH_test.pt
│ │ ├── image1000_CH_test.pt
│ │ └── image1001_CH_test.pt
│ │ ... (5000 files total)
│ ├── 5000
│ │ ├── image5000_CH_test.pt
│ │ ├── image5001_CH_test.pt
│ │ └── image5002_CH_test.pt
│ │ ... (5000 files total)
│ └── 10000
│ ├── image10000_CH_test.pt
│ ├── image10001_CH_test.pt
│ └── image10002_CH_test.pt
│ ... (4400 files total)
├── NY
│ ├── 0
│ │ ├── image0_NY_test.pt
│ │ ├── image1000_NY_test.pt
│ │ └── image1001_NY_test.pt
│ │ ... (5000 files total)
│ ├── 5000
│ │ ├── image5000_NY_test.pt
│ │ ├── image5001_NY_test.pt
│ │ └── image5002_NY_test.pt
│ │ ... (5000 files total)
│ └── 10000
│ ├── image10000_NY_test.pt
│ ├── image10001_NY_test.pt
│ └── image10002_NY_test.pt
│ ... (4400 files total)
├── NZ
│ ├── 0
│ │ ├── image0_NZ_test.pt
│ │ ├── image1000_NZ_test.pt
│ │ └── image1001_NZ_test.pt
│ │ ... (5000 files total)
│ ├── 5000
│ │ ├── image5000_NZ_test.pt
│ │ ├── image5001_NZ_test.pt
│ │ └── image5002_NZ_test.pt
│ │ ... (5000 files total)
│ └── 10000
│ ├── image10000_NZ_test.pt
│ ├── image10001_NZ_test.pt
│ └── image10002_NZ_test.pt
│ ... (4400 files total)
├── processed-flag-all
├── processed-flag-CH
└── processed-flag-NY
... (8 files total)
Pretrained model weights
Download
git lfs install
git clone https://huggingface.co/rsi/PixelsPointsPolygons $MODEL_ROOT
Code
Download
git clone https://github.com/raphaelsulzer/PixelsPointsPolygons
Installation
To create a conda environment named p3 and install the repository as a python package with all dependencies run
bash install.sh
or, if you want to manage the environment yourself run
pip install -r requirements-torch-cuda.txt
pip install .
⚠️ Warning: The implementation of the LiDAR point cloud encoder uses Open3D-ML. Currently, Open3D-ML officially only supports the PyTorch version specified in requirements-torch-cuda.txt.
Setup
The project supports hydra configuration which allows to modify any parameter either from a .yaml file of directly from the command line.
To setup the project structure we recommend to specify your $DATA_ROOT and $MODEL_ROOT in config/host/default.yaml.
To view all available configuration options run
python scripts/train.py --help
Predict a single tile
TODO
python scripts/predict_demo.py
Reproduce paper results
To reproduce the results from the paper you can run any of the following commands
python scripts/modality_ablation.py
python scripts/lidar_density_ablation.py
python scripts/all_countries.py
Custom training, prediction and evaluation
We recommend to first setup a custom $EXP_FILE in config/experiment following the structure of one of the existing experiment files, e.g. ffl_fusion.yaml. You can then run:
# train your model (on multiple GPUs)
torchrun --nproc_per_node=$NUM_GPU scripts/train.py experiment=$EXP_FILE
# predict the test set with your model (on multiple GPUs)
torchrun --nproc_per_node=$NUM_GPU scripts/predict.py evaluation=test checkpoint=best_val_iou
# evaluate your prediction of the test set
python scripts/evaluate.py model=<model> evaluation=test checkpoint=best_val_iou
You could also continue training from a provided pretrained model with
# train your model (on a single GPU)
python scripts/train.py experiment=p2p_fusion checkpoint=latest
Citation
If you find our work useful, please consider citing:
TODO
Acknowledgements
This repository benefits from the following open-source work. We thank the authors for their great work.