README.md

---
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: image-segmentation
tags:
  - semantic-segmentation
  - drone
  - rgb
  - thermal
  - infrared
  - dinov3
  - aerial
datasets:
  - markus-42/SegFly
model-index:
  - name: Firefly-RGB
    results: []
  - name: Firefly-Thermal
    results: []
---

# Introduction

<div align="center">

[![GitHub Documentation](https://img.shields.io/badge/GitHub-Documentation-green?logo=github&logoColor=white&labelColor=555)](https://github.com/markus-42/SegFly)&nbsp;&nbsp;
[![Project](https://img.shields.io/badge/Project-Page-blue.svg)](https://markus-42.github.io/publications/2026/segfly/)&nbsp;&nbsp;
[![arXiv](https://img.shields.io/badge/arXiv-Paper-red.svg)](https://arxiv.org/abs/2603.17920)&nbsp;&nbsp;
[![SegFly Dataset](https://img.shields.io/badge/HuggingFace-SegFly%20Dataset%20-e58f00?logo=huggingface&logoColor=white&labelColor=555)](https://huggingface.co/datasets/markus-42/SegFly)

</div>

Following the acceptance of [SegFly](https://markus-42.github.io/publications/2026/segfly/) at the ECCV 2026 computer vision conference, we not only release the [SegFly dataset](https://huggingface.co/datasets/markus-42/SegFly) on HuggingFace, but also our `Firefly` semantic segmentation model, specialized for RGB and thermal imagery from aerial perspectives. Firefly was trained on our SegFly dataset.

#### Key Features:
- **Aerial-specialized**: Fine-tuned on diverse aerial imagery from urban, industrial, and rural environments.
- **Multi-altitude performance**: Trained on data from 50m, 40m, and 30m altitudes.
- **Seasonal robustness**: Covers data across all seasons for improved generalization.
- **Semantic Categories**: Both models segment 15 semantic classes: road, walkway, dirt, gravel, grass, vegetation, tree, ground obstacles, vehicle, water, building, roof, parking lot, constructions, and truck.


# Checkpoints

Two pretrained checkpoints are provided:

| Variant | Subfolder | Modality | Architecture |
| :--- | :--- | :--- | :--- |
| Firefly-RGB | `Firefly_RGB/` | RGB drone imagery | DINOv3 ViT-B/16 + MLP head |
| Firefly-Thermal | `Firefly_Thermal/` | Thermal drone imagery | DINOv3 ViT-B/16 + Rein adapter + MLP head |


# Setup

```bash
git clone https://huggingface.co/markus-42/SegFly-Firefly
cd SegFly-Firefly

# clone DINOv3 backbone inside the repository
git clone https://github.com/facebookresearch/dinov3.git

# create virtual environment and install dependencies
uv venv && uv pip install -r requirements.txt

# activate virtual environment
source .venv/bin/activate
```

# Inference

Run single-image inference with `infer.py`. Weights are auto-detected from the modality:

```bash
# RGB model
python infer.py --image example.jpg --modality rgb

# Thermal model
python infer.py --image example_thermal.jpg --modality thermal
```

The colorized segmentation map is saved to `./infer_output/` by default.

| Argument | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `--image` | `str` | *(required)* | Path to the input image. |
| `--modality` | `str` | `rgb` | Model to use: `rgb` or `thermal`. |
| `--output` | `str` | `./infer_output` | Directory for the colorized segmentation output. |
| `--weights_path` | `str` | `""` | Path to model weights. Auto-detected from modality if not set. |
| `--dinov3_repo_dir` | `str` | `./dinov3` | Path to the local DINOv3 repository. |
| `--image_size` | `int` | `640` | Resolution at which the image is fed to the model. |

# Batch Evaluation

`eval.py` computes segmentation metrics (mIoU, Frequency-Weighted IoU, Pixel Accuracy) against ground truth masks for a set of images.

```bash
# RGB model
python eval.py --data_dir /path/to/dataset --modality rgb

# Thermal model
python eval.py --data_dir /path/to/dataset --modality thermal
```

`--data_dir` can point to any directory. The loader walks it recursively and collects images and masks by subfolder name — no specific top-level structure is required:

```
data_dir/
└── <any nesting>/
    ├── src/           ← RGB images (.png / .jpg / .jpeg)       [--modality rgb]
    ├── thermal_src/   ← thermal images (.png / .jpg / .jpeg)   [--modality thermal]
    └── gt/            ← grayscale segmentation masks (.png, raw SegFly class IDs)
```

Images and masks are matched by alphabetical sort order, so filenames must correspond 1-to-1.

Append `--visualize` to save colorized segmentation maps to `<output_dir>/visualizations`.

| Argument | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `--data_dir` | `str` | `./data` | Root directory of the dataset. Walked recursively; images are collected from `src/` or `thermal_src/` subfolders and masks from `gt/` subfolders. |
| `--weights_path` | `str` | `""` | Path to the saved weights. Auto-detected from modality if not set. |
| `--class_dict_path` | `str` | `./classes_segfly.csv` | Path to the CSV file defining the dataset's class mapping. |
| `--modality` | `str` | `rgb` | Modality of the dataset: `rgb` or `thermal`. |
| `--output_dir` | `str` | `./eval_output` | Directory to save `per_image_iou.csv`, `per_class_scores.txt`, and visualizations. |
| `--visualize` | `flag` | `False` | If provided, generates side-by-side visualizations of Image, Ground Truth, and Prediction. |
| `--image_size` | `int` | `640` | Input resolution for evaluation. |
| `--dinov3_repo_dir` | `str` | `./dinov3` | Path to the local DINOv3 repository. |


# Reference
If our work was helpful to you, we would appreciate citing our papers and giving the repository a like ❤️

```bibtex
@inproceedings{gross2026segfly,
    title={{SegFly: A Dataset and 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale}}, 
    author={Markus Gross and Sai Bharadhwaj Matha and Rui Song and Viswanathan Muthuveerappan and Conrad Christoph and Julius Huber and Daniel Cremers},
    booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
    year={2026},
}
```

Since SegFly is based on the [OccuFly dataset](https://markus-42.github.io/publications/2026/occufly/), consider citing this work as well:

```bibtex
@inproceedings{gross2026occufly,
    title={{OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective}}, 
    author={Markus Gross and Sai B. Matha and Aya Fahmy and Rui Song and Daniel Cremers and Henri Meess},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2026},
}
```

# License
This work is licensed under the [CC BY-NC-SA 4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/). See the LICENSE file for the full legal terms.