Update README.md
Browse files
README.md
CHANGED
|
@@ -1,17 +1,44 @@
|
|
| 1 |
---
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
library_name: sam2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
---
|
| 6 |
-
Model
|
| 7 |
-
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
-
|
|
|
|
|
|
|
| 12 |
|
| 13 |
-
|
| 14 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
```python
|
| 16 |
import torch
|
| 17 |
from sam2.sam2_image_predictor import SAM2ImagePredictor
|
|
@@ -23,8 +50,7 @@ with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
|
|
| 23 |
masks, _, _ = predictor.predict(<input_prompts>)
|
| 24 |
```
|
| 25 |
|
| 26 |
-
For video prediction
|
| 27 |
-
|
| 28 |
```python
|
| 29 |
import torch
|
| 30 |
from sam2.sam2_video_predictor import SAM2VideoPredictor
|
|
@@ -35,19 +61,55 @@ with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
|
|
| 35 |
state = predictor.init_state(<your_video>)
|
| 36 |
|
| 37 |
# add new prompts and instantly get the output on the same frame
|
| 38 |
-
frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)
|
| 39 |
|
| 40 |
# propagate the prompts to get masklets throughout the video
|
| 41 |
for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
|
| 42 |
...
|
| 43 |
```
|
| 44 |
|
| 45 |
-
Refer to the [demo notebooks](https://github.com/facebookresearch/
|
| 46 |
|
| 47 |
-
|
| 48 |
|
| 49 |
-
|
| 50 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
@article{ravi2024sam2,
|
| 52 |
title={SAM 2: Segment Anything in Images and Videos},
|
| 53 |
author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
|
|
|
|
| 1 |
---
|
| 2 |
+
tags:
|
| 3 |
+
- mask-generation
|
| 4 |
+
- sam2
|
| 5 |
+
- segmentation
|
| 6 |
+
- instance-segmentation
|
| 7 |
+
- video-segmentation
|
| 8 |
library_name: sam2
|
| 9 |
+
pipeline_tag: mask-generation
|
| 10 |
+
license: apache-2.0
|
| 11 |
+
datasets:
|
| 12 |
+
- iloncka/mosquito-species-segmentation-dataset
|
| 13 |
+
base_model:
|
| 14 |
+
- facebook/sam2-hiera-tiny
|
| 15 |
---
|
| 16 |
+
Model Card for culico-net-segm-v1-nano
|
| 17 |
+
|
| 18 |
+
culico-net-segm-v1-nano - instance segmentation model focused on segmenting mosquito specimens in images and videos. This model is a result of the CulicidaeLab project and was developed by fine-tuning the SAM2 (Segment Anything Model 2) architecture.
|
| 19 |
+
|
| 20 |
+
The culico-net-segm-v1-nano is based on SAM2's efficient Hiera-Tiny backbone and was specifically adapted for mosquito specimen segmentation using a dedicated dataset. This model can handle both image segmentation and video segmentation tasks with prompt-based interaction.
|
| 21 |
|
| 22 |
+
**Model Details:**
|
| 23 |
+
- **Model Type:** Instance segmentation / Mask generation
|
| 24 |
+
- **Architecture:** SAM2 with Hiera-Tiny backbone
|
| 25 |
+
- **Input:** Images or video frames with optional prompts (points, boxes)
|
| 26 |
+
- **Output:** Binary masks for mosquito specimens
|
| 27 |
+
- **Backbone:** Hiera-Tiny (efficient transformer architecture)
|
| 28 |
|
| 29 |
+
**Papers:**
|
| 30 |
+
- SAM 2: Segment Anything in Images and Videos: https://arxiv.org/abs/2408.00714
|
| 31 |
+
- Original GitHub Repository: https://github.com/facebookresearch/sam2
|
| 32 |
|
| 33 |
+
**Dataset:** The model was trained on the iloncka/mosquito-species-segmentation-dataset. This is one of a suite of datasets which also includes iloncka/mosquito-species-detection-dataset and iloncka/mosquito-species-classification-dataset. These datasets contain images of various mosquito species with detailed segmentation annotations, crucial for training accurate segmentation models.
|
| 34 |
|
| 35 |
+
**Pretrain Dataset:** ImageNet-1k, SA-1B (Segment Anything 1B dataset)
|
| 36 |
+
|
| 37 |
+
**Model Usage:**
|
| 38 |
+
|
| 39 |
+
The model can be used for both image and video segmentation tasks. Below are code snippets demonstrating how to use the model:
|
| 40 |
+
|
| 41 |
+
**For image prediction:**
|
| 42 |
```python
|
| 43 |
import torch
|
| 44 |
from sam2.sam2_image_predictor import SAM2ImagePredictor
|
|
|
|
| 50 |
masks, _, _ = predictor.predict(<input_prompts>)
|
| 51 |
```
|
| 52 |
|
| 53 |
+
**For video prediction:**
|
|
|
|
| 54 |
```python
|
| 55 |
import torch
|
| 56 |
from sam2.sam2_video_predictor import SAM2VideoPredictor
|
|
|
|
| 61 |
state = predictor.init_state(<your_video>)
|
| 62 |
|
| 63 |
# add new prompts and instantly get the output on the same frame
|
| 64 |
+
frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)
|
| 65 |
|
| 66 |
# propagate the prompts to get masklets throughout the video
|
| 67 |
for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
|
| 68 |
...
|
| 69 |
```
|
| 70 |
|
| 71 |
+
Refer to the [official SAM2 demo notebooks](https://github.com/facebookresearch/sam2) for details.
|
| 72 |
|
| 73 |
+
**The CulicidaeLab Project:**
|
| 74 |
|
| 75 |
+
The culico-net-segm-v1-nano model is a component of the larger CulicidaeLab project. This project aims to provide a comprehensive suite of tools for mosquito monitoring and research. Other parts of the project include:
|
| 76 |
+
|
| 77 |
+
- **Related Models:**
|
| 78 |
+
- [iloncka/culico-net-cls-v1](https://huggingface.co/iloncka/culico-net-cls-v1) - Classification model
|
| 79 |
+
- [iloncka/culico-net-det-v1](https://huggingface.co/iloncka/culico-net-det-v1)
|
| 80 |
+
|
| 81 |
+
- **Datasets:**
|
| 82 |
+
- [iloncka/mosquito-species-detection-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-detection-dataset)
|
| 83 |
+
- [iloncka/mosquito-species-segmentation-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-segmentation-dataset)
|
| 84 |
+
- [iloncka/mosquito-species-classification-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-classification-dataset)
|
| 85 |
+
|
| 86 |
+
- **Python Library:** [https://github.com/iloncka-ds/culicidaelab](https://github.com/iloncka-ds/culicidaelab)
|
| 87 |
+
|
| 88 |
+
- **Mobile Applications:**
|
| 89 |
+
- [https://gitlab.com/mosquitoscan/mosquitoscan-app](https://gitlab.com/mosquitoscan/mosquitoscan-app)
|
| 90 |
+
- [https://github.com/iloncka-ds/culicidaelab-mobile](https://github.com/iloncka-ds/culicidaelab-mobile)
|
| 91 |
+
|
| 92 |
+
- **Web Application:** [https://github.com/iloncka-ds/culicidaelab-server](https://github.com/iloncka-ds/culicidaelab-server)
|
| 93 |
+
|
| 94 |
+
**Practical Applications:**
|
| 95 |
+
|
| 96 |
+
The culico-net-segm-v1-nano model and the broader CulicidaeLab project have several practical applications:
|
| 97 |
+
|
| 98 |
+
- **Precise Morphological Analysis:** Enables detailed examination of mosquito anatomical features for species identification and research.
|
| 99 |
+
- **Automated Specimen Measurement:** Can be used to automatically measure wing dimensions, body size, and other morphological characteristics.
|
| 100 |
+
- **Training Data Generation:** Creates high-quality segmentation masks for training other computer vision models.
|
| 101 |
+
- **Embedded Systems (Edge AI):** The efficient architecture allows deployment on edge devices for real-time mosquito monitoring in field conditions.
|
| 102 |
+
- **Expert Systems Integration:** Provides precise segmentation capabilities that can assist entomologists in detailed specimen analysis.
|
| 103 |
+
|
| 104 |
+
**Acknowledgments:**
|
| 105 |
+
|
| 106 |
+
The development of CulicidaeLab is supported by a grant from the Foundation for Assistance to Small Innovative Enterprises (FASIE).
|
| 107 |
+
|
| 108 |
+
**Citation:**
|
| 109 |
+
|
| 110 |
+
To cite the SAM2 paper, model, or software, please use the below:
|
| 111 |
+
|
| 112 |
+
```bibtex
|
| 113 |
@article{ravi2024sam2,
|
| 114 |
title={SAM 2: Segment Anything in Images and Videos},
|
| 115 |
author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
|