| | --- |
| | license: apache-2.0 |
| | --- |
| | # Segment Anything Tool |
| |
|
| | The segment anything tool is uses the [Segment Anything Model](https://github.com/facebookresearch/segment-anything) (SAM). |
| | This folder contains the ONNX files that represent the SAM and are used by the browser to compute segmentations. |
| |
|
| | ## Get model checkpoint |
| |
|
| | The SAM is [provided by Facebook](https://github.com/facebookresearch/segment-anything#model-checkpoints) in the form of a pytorch model: a `.pth` file. |
| | There are three types of model. They are, from the biggest to the smallest, `vit_h`, `vit_l` and `vit_b`. |
| | It is advised to use `vit_b` as it is smaller and faster, at the cost of a lower quality segmentation. |
| |
|
| | ## Convert to ONNX |
| |
|
| | Facebook provides a script to convert the decoder to ONNX but not the encoder. The maintainers refuse to merge PRs adding this feature. |
| | You can use the script given in one of these PRs, but the chosen solution is to use the [samexporter](https://github.com/vietanhdev/samexporter). |
| |
|
| | - Create a folder that will contain the conversion script and the original `checkpoint.pth` model downloaded in the previous step. |
| | ```sh |
| | mkdir temp |
| | cd temp |
| | ``` |
| | - Clone the samexporter repo and the segment anything repo |
| | ```sh |
| | git clone git@github.com:vietanhdev/samexporter.git |
| | git clone git@github.com:facebookresearch/segment-anything.git |
| | ``` |
| | - Install both segment anything using pip and the dependencies of the samexporter (tested with a virtual environnment with python 3.11.5) |
| | ```sh |
| | pip install -e ./segment-anything |
| | pip install torchvision==0.16.1 onnx==1.15.0 onnxruntime==1.15.1 timm==0.9.12 |
| | ``` |
| | - Go in the samexporter folder and run the commands to export the encoder and the decoder (do not use quantization) |
| | ```sh |
| | cd samexporter |
| | python -m samexporter.export_encoder --checkpoint ../checkpoint.pth --output ../encoder.onnx --model-type vit_b |
| | python -m samexporter.export_decoder --checkpoint ../checkpoint.pth --output ../decoder.onnx --model-type vit_b --return-single-mask |
| | ``` |
| | - Copy the encoder and decoder at the right location in the project (for now, the only model available is `vit_b`) |
| | ```sh |
| | cd $NIMUS_IMAGE_DIR |
| | cp encoder.onnx public/onnx-models/sam/$MODEL_NAME |
| | cp decoder.onnx public/onnx-models/sam/$MODEL_NAME |
| | ``` |
