---
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: image-segmentation
tags:
- semantic-segmentation
- drone
- rgb
- thermal
- infrared
- dinov3
- aerial
datasets:
- markus-42/SegFly
model-index:
- name: Firefly-RGB
results: []
- name: Firefly-Thermal
results: []
---
# Introduction
[](https://github.com/markus-42/SegFly)
[](https://markus-42.github.io/publications/2026/segfly/)
[](https://arxiv.org/abs/2603.17920)
[](https://huggingface.co/datasets/markus-42/SegFly)
Following the acceptance of [SegFly](https://markus-42.github.io/publications/2026/segfly/) at the ECCV 2026 computer vision conference, we not only release the [SegFly dataset](https://huggingface.co/datasets/markus-42/SegFly) on HuggingFace, but also our `Firefly` semantic segmentation model, specialized for RGB and thermal imagery from aerial perspectives. Firefly was trained on our SegFly dataset.
#### Key Features:
- **Aerial-specialized**: Fine-tuned on diverse aerial imagery from urban, industrial, and rural environments.
- **Multi-altitude performance**: Trained on data from 50m, 40m, and 30m altitudes.
- **Seasonal robustness**: Covers data across all seasons for improved generalization.
- **Semantic Categories**: Both models segment 15 semantic classes: road, walkway, dirt, gravel, grass, vegetation, tree, ground obstacles, vehicle, water, building, roof, parking lot, constructions, and truck.
# Checkpoints
Two pretrained checkpoints are provided:
| Variant | Subfolder | Modality | Architecture |
| :--- | :--- | :--- | :--- |
| Firefly-RGB | `Firefly_RGB/` | RGB drone imagery | DINOv3 ViT-B/16 + MLP head |
| Firefly-Thermal | `Firefly_Thermal/` | Thermal drone imagery | DINOv3 ViT-B/16 + Rein adapter + MLP head |
# Setup
```bash
git clone https://huggingface.co/markus-42/SegFly-Firefly
cd SegFly-Firefly
# clone DINOv3 backbone inside the repository
git clone https://github.com/facebookresearch/dinov3.git
# create virtual environment and install dependencies
uv venv && uv pip install -r requirements.txt
# activate virtual environment
source .venv/bin/activate
```
# Inference
Run single-image inference with `infer.py`. Weights are auto-detected from the modality:
```bash
# RGB model
python infer.py --image example.jpg --modality rgb
# Thermal model
python infer.py --image example_thermal.jpg --modality thermal
```
The colorized segmentation map is saved to `./infer_output/` by default.
| Argument | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `--image` | `str` | *(required)* | Path to the input image. |
| `--modality` | `str` | `rgb` | Model to use: `rgb` or `thermal`. |
| `--output` | `str` | `./infer_output` | Directory for the colorized segmentation output. |
| `--weights_path` | `str` | `""` | Path to model weights. Auto-detected from modality if not set. |
| `--dinov3_repo_dir` | `str` | `./dinov3` | Path to the local DINOv3 repository. |
| `--image_size` | `int` | `640` | Resolution at which the image is fed to the model. |
# Batch Evaluation
`eval.py` computes segmentation metrics (mIoU, Frequency-Weighted IoU, Pixel Accuracy) against ground truth masks for a set of images.
```bash
# RGB model
python eval.py --data_dir /path/to/dataset --modality rgb
# Thermal model
python eval.py --data_dir /path/to/dataset --modality thermal
```
`--data_dir` can point to any directory. The loader walks it recursively and collects images and masks by subfolder name — no specific top-level structure is required:
```
data_dir/
└── /
├── src/ ← RGB images (.png / .jpg / .jpeg) [--modality rgb]
├── thermal_src/ ← thermal images (.png / .jpg / .jpeg) [--modality thermal]
└── gt/ ← grayscale segmentation masks (.png, raw SegFly class IDs)
```
Images and masks are matched by alphabetical sort order, so filenames must correspond 1-to-1.
Append `--visualize` to save colorized segmentation maps to `/visualizations`.
| Argument | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `--data_dir` | `str` | `./data` | Root directory of the dataset. Walked recursively; images are collected from `src/` or `thermal_src/` subfolders and masks from `gt/` subfolders. |
| `--weights_path` | `str` | `""` | Path to the saved weights. Auto-detected from modality if not set. |
| `--class_dict_path` | `str` | `./classes_segfly.csv` | Path to the CSV file defining the dataset's class mapping. |
| `--modality` | `str` | `rgb` | Modality of the dataset: `rgb` or `thermal`. |
| `--output_dir` | `str` | `./eval_output` | Directory to save `per_image_iou.csv`, `per_class_scores.txt`, and visualizations. |
| `--visualize` | `flag` | `False` | If provided, generates side-by-side visualizations of Image, Ground Truth, and Prediction. |
| `--image_size` | `int` | `640` | Input resolution for evaluation. |
| `--dinov3_repo_dir` | `str` | `./dinov3` | Path to the local DINOv3 repository. |
# Reference
If our work was helpful to you, we would appreciate citing our papers and giving the repository a like ❤️
```bibtex
@inproceedings{gross2026segfly,
title={{SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale}},
author={Markus Gross and Sai Bharadhwaj Matha and Rui Song and Viswanathan Muthuveerappan and Conrad Christoph and Julius Huber and Daniel Cremers},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year={2026},
}
```
Since SegFly is based on the [OccuFly dataset](https://markus-42.github.io/publications/2026/occufly/), consider citing this work as well:
```bibtex
@inproceedings{gross2026occufly,
title={{OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective}},
author={Markus Gross and Sai B. Matha and Aya Fahmy and Rui Song and Daniel Cremers and Henri Meess},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2026},
}
```
# License
This work is licensed under the [CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/). See the LICENSE file for the full legal terms.