README.md

---
tags:
  - mask-generation
  - sam2
  - segmentation
  - instance-segmentation
  - video-segmentation
library_name: sam2
pipeline_tag: mask-generation
license: apache-2.0
datasets:
  - iloncka/mosquito-species-segmentation-dataset
base_model:
  - facebook/sam2-hiera-tiny
---
Model Card for `culico-net-segm-v1-nano`

`culico-net-segm-v1-nano` - instance segmentation model focused on segmenting mosquito specimens in images and videos. This model is a result of the CulicidaeLab project and was developed by fine-tuning the SAM2 (Segment Anything Model 2) architecture.

The `culico-net-segm-v1-nano` is based on SAM2's efficient Hiera-Tiny backbone and was specifically adapted for mosquito specimen segmentation using a dedicated dataset. This model can handle both image segmentation and video segmentation tasks with prompt-based interaction.

**Model Details:**
- **Model Type:** Instance segmentation / Mask generation
- **Architecture:** SAM2 with Hiera-Tiny backbone
- **Input:** Images or video frames with optional prompts (points, boxes)
- **Output:** Binary masks for mosquito specimens
- **Backbone:** Hiera-Tiny (efficient transformer architecture)

**Papers:**
- SAM 2: Segment Anything in Images and Videos: https://arxiv.org/abs/2408.00714
- Original GitHub Repository: https://github.com/facebookresearch/sam2

**Dataset:** The model was trained on the `iloncka/mosquito-species-segmentation-dataset`. This is one of a suite of datasets which also includes iloncka/mosquito-species-detection-dataset and iloncka/mosquito-species-classification-dataset. These datasets contain images of various mosquito species with detailed segmentation annotations, crucial for training accurate segmentation models.

**Pretrain Dataset:** ImageNet-1k, SA-1B (Segment Anything 1B dataset)

**Model Usage:**

The model can be used for both image and video segmentation tasks. Below are code snippets demonstrating how to use the model:

**For image prediction:**
```python
import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained("iloncka/culico-net-segm-v1-nano")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)
```

**For video prediction:**
```python
import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained("iloncka/culico-net-segm-v1-nano")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...
```

Refer to the [official SAM2 demo notebooks](https://github.com/facebookresearch/sam2) for details.

**The CulicidaeLab Project:**

The `culico-net-segm-v1-nano model` is a component of the larger `CulicidaeLab` project. This project aims to provide a comprehensive suite of tools for mosquito monitoring and research. Other parts of the project include:

- **Related Models:**
  - [iloncka/culico-net-cls-v1](https://huggingface.co/iloncka/culico-net-cls-v1) - Classification model
  - [iloncka/culico-net-det-v1](https://huggingface.co/iloncka/culico-net-det-v1) - Detection model

- **Datasets:**
  - [iloncka/mosquito-species-detection-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-detection-dataset)
  - [iloncka/mosquito-species-segmentation-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-segmentation-dataset)
  - [iloncka/mosquito-species-classification-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-classification-dataset)

- **Python Library:** [https://github.com/iloncka-ds/culicidaelab](https://github.com/iloncka-ds/culicidaelab)

- **Mobile Applications:**
  - [https://gitlab.com/mosquitoscan/mosquitoscan-app](https://gitlab.com/mosquitoscan/mosquitoscan-app)
  - [https://github.com/iloncka-ds/culicidaelab-mobile](https://github.com/iloncka-ds/culicidaelab-mobile)

- **Web Application:** [https://github.com/iloncka-ds/culicidaelab-server](https://github.com/iloncka-ds/culicidaelab-server)

**Practical Applications:**

The `culico-net-segm-v1-nano` model and the broader `CulicidaeLab` project have several practical applications:

- **Precise Morphological Analysis:** Enables detailed examination of mosquito anatomical features for species identification and research.
- **Automated Specimen Measurement:** Can be used to automatically measure wing dimensions, body size, and other morphological characteristics.
- **Training Data Generation:** Creates high-quality segmentation masks for training other computer vision models.
- **Embedded Systems (Edge AI):** The efficient architecture allows deployment on edge devices for real-time mosquito monitoring in field conditions.
- **Expert Systems Integration:** Provides precise segmentation capabilities that can assist entomologists in detailed specimen analysis.

**Acknowledgments:**

The development of `CulicidaeLab` is supported by a grant from the [Foundation for Assistance to Small Innovative Enterprises (FASIE)]((https://fasie.ru/)).


**Citation:**

To cite the SAM2 paper, model, or software, please use the below:

```bibtex
@article{ravi2024sam2,
  title={SAM 2: Segment Anything in Images and Videos},
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
  journal={arXiv preprint arXiv:2408.00714},
  url={https://arxiv.org/abs/2408.00714},
  year={2024}
}
```