iloncka commited on
Commit
dcaeb25
·
verified ·
1 Parent(s): d4d1f10

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -14
README.md CHANGED
@@ -1,17 +1,44 @@
1
  ---
2
- license: apache-2.0
3
- pipeline_tag: mask-generation
 
 
 
 
4
  library_name: sam2
 
 
 
 
 
 
5
  ---
6
- Model cloned from this [repo](https://github.com/facebookresearch/segment-anything-2/) and will be finetuned.
7
- Repository for SAM 2: Segment Anything in Images and Videos, a foundation model towards solving promptable visual segmentation in images and videos from FAIR. See the [SAM 2 paper](https://arxiv.org/abs/2408.00714) for more information.
 
 
 
8
 
9
- The official code is publicly release in this [repo](https://github.com/facebookresearch/segment-anything-2/).
 
 
 
 
 
10
 
11
- ## Usage
 
 
12
 
13
- For image prediction:
14
 
 
 
 
 
 
 
 
15
  ```python
16
  import torch
17
  from sam2.sam2_image_predictor import SAM2ImagePredictor
@@ -23,8 +50,7 @@ with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
23
  masks, _, _ = predictor.predict(<input_prompts>)
24
  ```
25
 
26
- For video prediction:
27
-
28
  ```python
29
  import torch
30
  from sam2.sam2_video_predictor import SAM2VideoPredictor
@@ -35,19 +61,55 @@ with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
35
  state = predictor.init_state(<your_video>)
36
 
37
  # add new prompts and instantly get the output on the same frame
38
- frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>):
39
 
40
  # propagate the prompts to get masklets throughout the video
41
  for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
42
  ...
43
  ```
44
 
45
- Refer to the [demo notebooks](https://github.com/facebookresearch/segment-anything-2/tree/main/notebooks) for details.
46
 
47
- ### Citation
48
 
49
- To cite the paper, model, or software, please use the below:
50
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
  @article{ravi2024sam2,
52
  title={SAM 2: Segment Anything in Images and Videos},
53
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},
 
1
  ---
2
+ tags:
3
+ - mask-generation
4
+ - sam2
5
+ - segmentation
6
+ - instance-segmentation
7
+ - video-segmentation
8
  library_name: sam2
9
+ pipeline_tag: mask-generation
10
+ license: apache-2.0
11
+ datasets:
12
+ - iloncka/mosquito-species-segmentation-dataset
13
+ base_model:
14
+ - facebook/sam2-hiera-tiny
15
  ---
16
+ Model Card for culico-net-segm-v1-nano
17
+
18
+ culico-net-segm-v1-nano - instance segmentation model focused on segmenting mosquito specimens in images and videos. This model is a result of the CulicidaeLab project and was developed by fine-tuning the SAM2 (Segment Anything Model 2) architecture.
19
+
20
+ The culico-net-segm-v1-nano is based on SAM2's efficient Hiera-Tiny backbone and was specifically adapted for mosquito specimen segmentation using a dedicated dataset. This model can handle both image segmentation and video segmentation tasks with prompt-based interaction.
21
 
22
+ **Model Details:**
23
+ - **Model Type:** Instance segmentation / Mask generation
24
+ - **Architecture:** SAM2 with Hiera-Tiny backbone
25
+ - **Input:** Images or video frames with optional prompts (points, boxes)
26
+ - **Output:** Binary masks for mosquito specimens
27
+ - **Backbone:** Hiera-Tiny (efficient transformer architecture)
28
 
29
+ **Papers:**
30
+ - SAM 2: Segment Anything in Images and Videos: https://arxiv.org/abs/2408.00714
31
+ - Original GitHub Repository: https://github.com/facebookresearch/sam2
32
 
33
+ **Dataset:** The model was trained on the iloncka/mosquito-species-segmentation-dataset. This is one of a suite of datasets which also includes iloncka/mosquito-species-detection-dataset and iloncka/mosquito-species-classification-dataset. These datasets contain images of various mosquito species with detailed segmentation annotations, crucial for training accurate segmentation models.
34
 
35
+ **Pretrain Dataset:** ImageNet-1k, SA-1B (Segment Anything 1B dataset)
36
+
37
+ **Model Usage:**
38
+
39
+ The model can be used for both image and video segmentation tasks. Below are code snippets demonstrating how to use the model:
40
+
41
+ **For image prediction:**
42
  ```python
43
  import torch
44
  from sam2.sam2_image_predictor import SAM2ImagePredictor
 
50
  masks, _, _ = predictor.predict(<input_prompts>)
51
  ```
52
 
53
+ **For video prediction:**
 
54
  ```python
55
  import torch
56
  from sam2.sam2_video_predictor import SAM2VideoPredictor
 
61
  state = predictor.init_state(<your_video>)
62
 
63
  # add new prompts and instantly get the output on the same frame
64
+ frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, <your_prompts>)
65
 
66
  # propagate the prompts to get masklets throughout the video
67
  for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
68
  ...
69
  ```
70
 
71
+ Refer to the [official SAM2 demo notebooks](https://github.com/facebookresearch/sam2) for details.
72
 
73
+ **The CulicidaeLab Project:**
74
 
75
+ The culico-net-segm-v1-nano model is a component of the larger CulicidaeLab project. This project aims to provide a comprehensive suite of tools for mosquito monitoring and research. Other parts of the project include:
76
+
77
+ - **Related Models:**
78
+ - [iloncka/culico-net-cls-v1](https://huggingface.co/iloncka/culico-net-cls-v1) - Classification model
79
+ - [iloncka/culico-net-det-v1](https://huggingface.co/iloncka/culico-net-det-v1)
80
+
81
+ - **Datasets:**
82
+ - [iloncka/mosquito-species-detection-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-detection-dataset)
83
+ - [iloncka/mosquito-species-segmentation-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-segmentation-dataset)
84
+ - [iloncka/mosquito-species-classification-dataset](https://huggingface.co/datasets/iloncka/mosquito-species-classification-dataset)
85
+
86
+ - **Python Library:** [https://github.com/iloncka-ds/culicidaelab](https://github.com/iloncka-ds/culicidaelab)
87
+
88
+ - **Mobile Applications:**
89
+ - [https://gitlab.com/mosquitoscan/mosquitoscan-app](https://gitlab.com/mosquitoscan/mosquitoscan-app)
90
+ - [https://github.com/iloncka-ds/culicidaelab-mobile](https://github.com/iloncka-ds/culicidaelab-mobile)
91
+
92
+ - **Web Application:** [https://github.com/iloncka-ds/culicidaelab-server](https://github.com/iloncka-ds/culicidaelab-server)
93
+
94
+ **Practical Applications:**
95
+
96
+ The culico-net-segm-v1-nano model and the broader CulicidaeLab project have several practical applications:
97
+
98
+ - **Precise Morphological Analysis:** Enables detailed examination of mosquito anatomical features for species identification and research.
99
+ - **Automated Specimen Measurement:** Can be used to automatically measure wing dimensions, body size, and other morphological characteristics.
100
+ - **Training Data Generation:** Creates high-quality segmentation masks for training other computer vision models.
101
+ - **Embedded Systems (Edge AI):** The efficient architecture allows deployment on edge devices for real-time mosquito monitoring in field conditions.
102
+ - **Expert Systems Integration:** Provides precise segmentation capabilities that can assist entomologists in detailed specimen analysis.
103
+
104
+ **Acknowledgments:**
105
+
106
+ The development of CulicidaeLab is supported by a grant from the Foundation for Assistance to Small Innovative Enterprises (FASIE).
107
+
108
+ **Citation:**
109
+
110
+ To cite the SAM2 paper, model, or software, please use the below:
111
+
112
+ ```bibtex
113
  @article{ravi2024sam2,
114
  title={SAM 2: Segment Anything in Images and Videos},
115
  author={Ravi, Nikhila and Gabeur, Valentin and Hu, Yuan-Ting and Hu, Ronghang and Ryali, Chaitanya and Ma, Tengyu and Khedr, Haitham and R{\"a}dle, Roman and Rolland, Chloe and Gustafson, Laura and Mintun, Eric and Pan, Junting and Alwala, Kalyan Vasudev and Carion, Nicolas and Wu, Chao-Yuan and Girshick, Ross and Doll{\'a}r, Piotr and Feichtenhofer, Christoph},