Hugging Face's logo

bf0820
/
VQ-SAM

Mask Generation
sam2
visual-query-segmentation
video-object-segmentation
vqs
Model card Files
xet
 Community
VQ-SAM
File size: 3,332 Bytes
d0e5e6b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
---
license: apache-2.0
pipeline_tag: mask-generation
library_name: sam2
tags:
  - visual-query-segmentation
  - video-object-segmentation
  - sam2
  - vqs
datasets:
  - bf0820/VQS-4K
---

# VQ-SAM

Baseline checkpoint for **visual query segmentation (VQS)** from the NeurIPS 2026
Datasets & Benchmarks paper [*Towards Visual Query Segmentation in the
Wild*](https://arxiv.org/pdf/2603.08898).

Given an untrimmed video and a **visual query** — one image plus a mask of the target, taken
from **outside** that video — VQ-SAM segments the object in every frame where it appears.
It extends SAM 2 with a spatio-temporal transformer block and target / distractor memory
evolved through a multi-stage framework with an adaptive memory generation (AMG) module.

- 📄 Paper: [arXiv:2603.08898](https://arxiv.org/pdf/2603.08898)
- 💻 Code: [github.com/VAIL-Fan/VQS](https://github.com/VAIL-Fan/VQS)
- 📦 Dataset: [bf0820/VQS-4K](https://huggingface.co/datasets/bf0820/VQS-4K)

## Results on VQS-4K test (822 videos)

| mSTIoU | mSTIoU₅₀ | mSTIoU₇₅ | mTIoU | mTIoU₅₀ | mTIoU₇₅ | Rec | Succ | FPS |
|---|---|---|---|---|---|---|---|---|
| 26.0 | 23.8 | 12.7 | 29.6 | 26.8 | 14.4 | 43.6 | 42.1 | 29 |

Best published result on this benchmark; the strongest prior method (SAM2Long) reaches
18.6 mSTIoU. These numbers are reproducible from this checkpoint with the evaluation code in
the repository.

## Files

| File | Description |
|---|---|
| `vqsam_hiera_bplus.pt` | VQ-SAM, SAM 2.1 Hiera-B+ backbone, trained on the VQS-4K train split (1.0 GB) |

## Usage

```bash
git clone https://github.com/VAIL-Fan/VQS && cd VQS
pip install -r requirements.txt && pip install -e .

python - <<'PY'
from huggingface_hub import hf_hub_download
hf_hub_download("bf0820/VQ-SAM", "vqsam_hiera_bplus.pt", local_dir="checkpoints")
PY

python tools/test_video_segmentation.py \
  --sam2_cfg sam2/configs/sam2.1_training/sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml \
  --sam2_checkpoint checkpoints/vqsam_hiera_bplus.pt \
  --base_video_dir ./data/VQS-4K/images \
  --input_mask_dir ./data/VQS-4K/masks \
  --video_list_file ./splits/test_t-IoU.txt \
  --output_mask_dir ./outputs/predictions \
  --num_frames_per_segment 8
```

The inference segment length must match the training clip length (8 for this checkpoint).

## Training

SAM 2.1 Hiera-B+ initialisation, 768x768, 8 frames per clip (1 query + 7 video), batch size 1
per GPU, on 4x RTX A6000. Config
`sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml` in the repository, with two-stage memory,
distractor memory, quality-weighted RAM and adaptive gating enabled.

## License

Code and weights: Apache 2.0, inherited from [SAM 2](https://github.com/facebookresearch/sam2).
The VQS-4K dataset is separately licensed CC BY-NC 4.0, research use only.

## Citation

```bibtex
@inproceedings{fan2026vqs,
  title     = {Towards Visual Query Segmentation in the Wild},
  author    = {Fan, Bing and Li, Minghao and Zhang, Hanzhi and Dong, Shaohua and
               Mareedu, Naga Prudhvi and Liu, Xiaoqiong and Shi, Weishi and
               Feng, Yunhe and Huang, Yan and Fan, Heng},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)
               Datasets and Benchmarks Track},
  year      = {2026},
  eprint    = {2603.08898},
  archivePrefix = {arXiv}
}
```