Instructions to use ccllr/MP-SAM2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sam2
How to use ccllr/MP-SAM2 with sam2:
# Use SAM2 with images import torch from sam2.sam2_image_predictor import SAM2ImagePredictor predictor = SAM2ImagePredictor.from_pretrained("ccllr/MP-SAM2") with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image(<your_image>) masks, _, _ = predictor.predict(<input_prompts>)# Use SAM2 with videos import torch from sam2.sam2_video_predictor import SAM2VideoPredictor predictor = SAM2VideoPredictor.from_pretrained("ccllr/MP-SAM2") with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): state = predictor.init_state(<your_video>) # add new prompts and instantly get the output on the same frame frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>) # propagate the prompts to get masklets throughout the video for frame_idx, object_ids, masks in predictor.propagate_in_video(state): ... - Notebooks
- Google Colab
- Kaggle
Align public checkpoint names and model documentation
Browse files- README.md +87 -67
- SHA256SUMS +4 -4
- mp_sam2_10_seed42.torch → mp_sam2_10.torch +0 -0
- mp_sam2_5_seed42.torch → mp_sam2_5.torch +0 -0
- mp_sam2_50_seed42.torch → mp_sam2_50.torch +0 -0
README.md
CHANGED
|
@@ -1,67 +1,87 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
language:
|
| 4 |
-
- en
|
| 5 |
-
pipeline_tag: image-segmentation
|
| 6 |
-
tags:
|
| 7 |
-
- pytorch
|
| 8 |
-
- sam2
|
| 9 |
-
- microcrack-segmentation
|
| 10 |
-
- video-segmentation
|
| 11 |
-
---
|
| 12 |
-
|
| 13 |
-
# MP-SAM2
|
| 14 |
-
|
| 15 |
-
Fine-tuned
|
| 16 |
-
Segment Anything Model 2 for Microcrack Segmentation in Engineered Cementitious
|
| 17 |
-
Composite Images and Videos**.
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
|
| 30 |
-
|
|
| 31 |
-
|
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
pipeline_tag: image-segmentation
|
| 6 |
+
tags:
|
| 7 |
+
- pytorch
|
| 8 |
+
- sam2
|
| 9 |
+
- microcrack-segmentation
|
| 10 |
+
- video-segmentation
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# MP-SAM2
|
| 14 |
+
|
| 15 |
+
Fine-tuned models for **A Multi-Prompt Transfer Learning Framework Based on
|
| 16 |
+
Segment Anything Model 2 for Microcrack Segmentation in Engineered Cementitious
|
| 17 |
+
Composite Images and Videos**.
|
| 18 |
+
|
| 19 |
+
[Source and installation](https://github.com/cclyr/MP-SAM2) ·
|
| 20 |
+
[Training settings](https://github.com/cclyr/MP-SAM2/tree/main/configs) ·
|
| 21 |
+
[Model architecture](https://github.com/cclyr/MP-SAM2/blob/main/sam2/configs/sam2_hiera_l.yaml)
|
| 22 |
+
|
| 23 |
+
MP-SAM2 combines an automatic threshold mask, SP-Box, and EDT positive/negative
|
| 24 |
+
point prompts with progressive SAM2 transfer learning. The GitHub repository
|
| 25 |
+
provides training, image inference, video propagation, and evaluation scripts.
|
| 26 |
+
|
| 27 |
+
## Checkpoints
|
| 28 |
+
|
| 29 |
+
| File | Training images |
|
| 30 |
+
|---|---:|
|
| 31 |
+
| [mp_sam2_5.torch](https://huggingface.co/ccllr/MP-SAM2/resolve/main/mp_sam2_5.torch) | 5 |
|
| 32 |
+
| [mp_sam2_10.torch](https://huggingface.co/ccllr/MP-SAM2/resolve/main/mp_sam2_10.torch) | 10 |
|
| 33 |
+
| [mp_sam2_50.torch](https://huggingface.co/ccllr/MP-SAM2/resolve/main/mp_sam2_50.torch) | 50 |
|
| 34 |
+
| [mp_sam2_full.torch](https://huggingface.co/ccllr/MP-SAM2/resolve/main/mp_sam2_full.torch) | 186 (full data) |
|
| 35 |
+
|
| 36 |
+
Each file is approximately 898 MB and contains the complete model state for the
|
| 37 |
+
bundled `sam2_hiera_l.yaml` architecture. Each file is an individual trained
|
| 38 |
+
model; the paper's few-shot results summarize repeated training runs.
|
| 39 |
+
A 20-image checkpoint is not included. `SHA256SUMS` lists the file hashes.
|
| 40 |
+
|
| 41 |
+
## Usage
|
| 42 |
+
|
| 43 |
+
Follow the [source installation instructions](https://github.com/cclyr/MP-SAM2#installation).
|
| 44 |
+
From the source repository root:
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
python scripts/download_checkpoints.py --variant full
|
| 48 |
+
|
| 49 |
+
python scripts/infer_image.py \
|
| 50 |
+
--input path/to/image.png \
|
| 51 |
+
--checkpoint checkpoints/mp_sam2_full.torch \
|
| 52 |
+
--config configs/prompt.yaml \
|
| 53 |
+
--output outputs/image
|
| 54 |
+
```
|
| 55 |
+
|
| 56 |
+
Use `--variant 5`, `10`, `50`, or `all` for other checkpoints. Direct downloads
|
| 57 |
+
are also linked above. The download helper verifies MP-SAM2 file checksums.
|
| 58 |
+
|
| 59 |
+
Training and video inference also need Meta's original
|
| 60 |
+
[SAM2.1 Hiera Large checkpoint](https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt).
|
| 61 |
+
Download it into `checkpoints/` with:
|
| 62 |
+
|
| 63 |
+
```bash
|
| 64 |
+
python scripts/download_checkpoints.py --variant full --include-base
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
The transfer loader selects matching SAM2.1 tensors for the bundled MP-SAM2
|
| 68 |
+
architecture, excluding SAM2.1-specific spatial and pointer-projection heads.
|
| 69 |
+
Use this repository's runtime and configuration with these weights. The files
|
| 70 |
+
are custom PyTorch state dictionaries loaded by the MP-SAM2 scripts; a
|
| 71 |
+
Transformers `AutoModel` interface is not supplied.
|
| 72 |
+
|
| 73 |
+
`configs/prompt.yaml` controls automatic prompts for image inference and the
|
| 74 |
+
first video frame. `configs/fewshot.yaml` and `configs/full.yaml` control the
|
| 75 |
+
respective training schedules. Full instructions, data layout, and evaluation
|
| 76 |
+
commands are in the [README](https://github.com/cclyr/MP-SAM2#readme).
|
| 77 |
+
|
| 78 |
+
## Scope and license
|
| 79 |
+
|
| 80 |
+
The intended application is microcrack segmentation in engineered cementitious
|
| 81 |
+
composite images and laboratory videos. Raw images, videos, annotations,
|
| 82 |
+
experiment logs, and intermediate checkpoints are not included.
|
| 83 |
+
The original Meta checkpoint is obtained from upstream.
|
| 84 |
+
|
| 85 |
+
The four fine-tuned files are distributed under Apache License 2.0. See `LICENSE`
|
| 86 |
+
and the upstream SAM2 attribution in `NOTICE`. Citation metadata is provided in
|
| 87 |
+
the source repository's [CITATION.cff](https://github.com/cclyr/MP-SAM2/blob/main/CITATION.cff).
|
SHA256SUMS
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
e56b6e6d4392bfb9d48ff62cd92565891e77a0b9cbd8bd6b5ad2125f0e3c3be4 mp_sam2_full.torch
|
| 2 |
-
6b0cfb1bc7ac96a203b9be32da0dc6a6ddc3464c5e44405fdac6c67370921403
|
| 3 |
-
6c1f5f650e3e8d61e99ec67bcbc07d040d81c46acd198caf0b6b422e20c4e7cb
|
| 4 |
-
c0fbf7ce437527f0a2ccc00c9103b55b583abbd3add3ab6150e82a365c22252b
|
|
|
|
| 1 |
+
e56b6e6d4392bfb9d48ff62cd92565891e77a0b9cbd8bd6b5ad2125f0e3c3be4 mp_sam2_full.torch
|
| 2 |
+
6b0cfb1bc7ac96a203b9be32da0dc6a6ddc3464c5e44405fdac6c67370921403 mp_sam2_5.torch
|
| 3 |
+
6c1f5f650e3e8d61e99ec67bcbc07d040d81c46acd198caf0b6b422e20c4e7cb mp_sam2_10.torch
|
| 4 |
+
c0fbf7ce437527f0a2ccc00c9103b55b583abbd3add3ab6150e82a365c22252b mp_sam2_50.torch
|
mp_sam2_10_seed42.torch → mp_sam2_10.torch
RENAMED
|
File without changes
|
mp_sam2_5_seed42.torch → mp_sam2_5.torch
RENAMED
|
File without changes
|
mp_sam2_50_seed42.torch → mp_sam2_50.torch
RENAMED
|
File without changes
|