| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - sam2act |
| - robotics |
| - robot-manipulation |
| - rlbench |
| - memorybench |
| - pytorch |
| --- |
| |
| # SAM2Act |
|
|
| SAM2Act is a multi-view robotics transformer policy for robotic manipulation. Built on RVT-2, it combines multi-resolution upsampling with visual embeddings from the SAM2 foundation model to improve 3D action prediction, multitask learning, and generalization. SAM2Act+ extends this policy with a memory bank, memory encoder, and memory attention so the agent can condition on prior observations and actions for spatial memory-dependent tasks. |
|
|
| For full project details, code, training instructions, and videos, see the [SAM2Act website](https://sam2act.github.io/) and [GitHub repository](https://github.com/sam2act/sam2act). |
|
|
| ## Models |
|
|
| This model repository stores released SAM2Act and SAM2Act+ checkpoints together with the config files needed for evaluation. |
|
|
| The repository is organized as follows: |
|
|
| ```text |
| sam2act_rlbench/ # SAM2Act checkpoint and configs for RLBench |
| sam2act_memorybench/sam2act_<task>/ # SAM2Act+ checkpoints and configs for MemoryBench |
| ``` |
|
|
| `sam2act_rlbench/` contains the RLBench checkpoint `model_89.pth` and its `exp_cfg.yaml` and `mvt_cfg.yaml` files. Each MemoryBench task folder contains the Stage 1 checkpoint `model_9.pth`, the Stage 2 memory-conditioned checkpoint `model_plus_19.pth`, and the corresponding `exp_cfg*.yaml` and `mvt_cfg*.yaml` files. |
|
|
| For evaluation commands and expected directory placement, see the [SAM2Act GitHub README](https://github.com/sam2act/sam2act) or the [SAM2Act project website](https://sam2act.github.io/). |
|
|
| ## Bibtex |
|
|
| If you use these models, please cite the SAM2Act paper: |
|
|
| ```bibtex |
| @misc{fang2025sam2act, |
| title={SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation}, |
| author={Haoquan Fang and Markus Grotz and Wilbert Pumacay and Yi Ru Wang and Dieter Fox and Ranjay Krishna and Jiafei Duan}, |
| year={2025}, |
| eprint={2501.18564}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.RO}, |
| url={https://arxiv.org/abs/2501.18564}, |
| } |
| ``` |
|
|
