| --- |
| license: apache-2.0 |
| pipeline_tag: mask-generation |
| library_name: sam2 |
| tags: |
| - visual-query-segmentation |
| - video-object-segmentation |
| - sam2 |
| - vqs |
| datasets: |
| - bf0820/VQS-4K |
| --- |
| |
| # VQ-SAM |
|
|
| Baseline checkpoint for **visual query segmentation (VQS)** from the NeurIPS 2026 |
| Datasets & Benchmarks paper [*Towards Visual Query Segmentation in the |
| Wild*](https://arxiv.org/pdf/2603.08898). |
|
|
| Given an untrimmed video and a **visual query** — one image plus a mask of the target, taken |
| from **outside** that video — VQ-SAM segments the object in every frame where it appears. |
| It extends SAM 2 with a spatio-temporal transformer block and target / distractor memory |
| evolved through a multi-stage framework with an adaptive memory generation (AMG) module. |
|
|
| - 📄 Paper: [arXiv:2603.08898](https://arxiv.org/pdf/2603.08898) |
| - 💻 Code: [github.com/VAIL-Fan/VQS](https://github.com/VAIL-Fan/VQS) |
| - 📦 Dataset: [bf0820/VQS-4K](https://huggingface.co/datasets/bf0820/VQS-4K) |
|
|
| ## Results on VQS-4K test (822 videos) |
|
|
| | mSTIoU | mSTIoU₅₀ | mSTIoU₇₅ | mTIoU | mTIoU₅₀ | mTIoU₇₅ | Rec | Succ | FPS | |
| |---|---|---|---|---|---|---|---|---| |
| | 26.0 | 23.8 | 12.7 | 29.6 | 26.8 | 14.4 | 43.6 | 42.1 | 29 | |
|
|
| Best published result on this benchmark; the strongest prior method (SAM2Long) reaches |
| 18.6 mSTIoU. These numbers are reproducible from this checkpoint with the evaluation code in |
| the repository. |
|
|
| ## Files |
|
|
| | File | Description | |
| |---|---| |
| | `vqsam_hiera_bplus.pt` | VQ-SAM, SAM 2.1 Hiera-B+ backbone, trained on the VQS-4K train split (1.0 GB) | |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/VAIL-Fan/VQS && cd VQS |
| pip install -r requirements.txt && pip install -e . |
| |
| python - <<'PY' |
| from huggingface_hub import hf_hub_download |
| hf_hub_download("bf0820/VQ-SAM", "vqsam_hiera_bplus.pt", local_dir="checkpoints") |
| PY |
| |
| python tools/test_video_segmentation.py \ |
| --sam2_cfg sam2/configs/sam2.1_training/sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml \ |
| --sam2_checkpoint checkpoints/vqsam_hiera_bplus.pt \ |
| --base_video_dir ./data/VQS-4K/images \ |
| --input_mask_dir ./data/VQS-4K/masks \ |
| --video_list_file ./splits/test_t-IoU.txt \ |
| --output_mask_dir ./outputs/predictions \ |
| --num_frames_per_segment 8 |
| ``` |
|
|
| The inference segment length must match the training clip length (8 for this checkpoint). |
|
|
| ## Training |
|
|
| SAM 2.1 Hiera-B+ initialisation, 768x768, 8 frames per clip (1 query + 7 video), batch size 1 |
| per GPU, on 4x RTX A6000. Config |
| `sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml` in the repository, with two-stage memory, |
| distractor memory, quality-weighted RAM and adaptive gating enabled. |
|
|
| ## License |
|
|
| Code and weights: Apache 2.0, inherited from [SAM 2](https://github.com/facebookresearch/sam2). |
| The VQS-4K dataset is separately licensed CC BY-NC 4.0, research use only. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{fan2026vqs, |
| title = {Towards Visual Query Segmentation in the Wild}, |
| author = {Fan, Bing and Li, Minghao and Zhang, Hanzhi and Dong, Shaohua and |
| Mareedu, Naga Prudhvi and Liu, Xiaoqiong and Shi, Weishi and |
| Feng, Yunhe and Huang, Yan and Fan, Heng}, |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS) |
| Datasets and Benchmarks Track}, |
| year = {2026}, |
| eprint = {2603.08898}, |
| archivePrefix = {arXiv} |
| } |
| ``` |
|
|
