VQ-SAM

Baseline checkpoint for visual query segmentation (VQS) from the NeurIPS 2026 Datasets & Benchmarks paper Towards Visual Query Segmentation in the Wild.

Given an untrimmed video and a visual query — one image plus a mask of the target, taken from outside that video — VQ-SAM segments the object in every frame where it appears. It extends SAM 2 with a spatio-temporal transformer block and target / distractor memory evolved through a multi-stage framework with an adaptive memory generation (AMG) module.

• 📄 Paper: arXiv:2603.08898
• 💻 Code: github.com/VAIL-Fan/VQS
• 📦 Dataset: bf0820/VQS-4K

Results on VQS-4K test (822 videos)

mSTIoU	mSTIoU₅₀	mSTIoU₇₅	mTIoU	mTIoU₅₀	mTIoU₇₅	Rec	Succ	FPS
26.0	23.8	12.7	29.6	26.8	14.4	43.6	42.1	29

Best published result on this benchmark; the strongest prior method (SAM2Long) reaches 18.6 mSTIoU. These numbers are reproducible from this checkpoint with the evaluation code in the repository.

Files

File	Description
vqsam_hiera_bplus.pt	VQ-SAM, SAM 2.1 Hiera-B+ backbone, trained on the VQS-4K train split (1.0 GB)

Usage

git clone https://github.com/VAIL-Fan/VQS && cd VQS
pip install -r requirements.txt && pip install -e .

python - <<'PY'
from huggingface_hub import hf_hub_download
hf_hub_download("bf0820/VQ-SAM", "vqsam_hiera_bplus.pt", local_dir="checkpoints")
PY

python tools/test_video_segmentation.py \
  --sam2_cfg sam2/configs/sam2.1_training/sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml \
  --sam2_checkpoint checkpoints/vqsam_hiera_bplus.pt \
  --base_video_dir ./data/VQS-4K/images \
  --input_mask_dir ./data/VQS-4K/masks \
  --video_list_file ./splits/test_t-IoU.txt \
  --output_mask_dir ./outputs/predictions \
  --num_frames_per_segment 8

The inference segment length must match the training clip length (8 for this checkpoint).

Training

SAM 2.1 Hiera-B+ initialisation, 768x768, 8 frames per clip (1 query + 7 video), batch size 1 per GPU, on 4x RTX A6000. Config sam2.1_hiera_b+_DAM_AdaptiveGate_Jan28.yaml in the repository, with two-stage memory, distractor memory, quality-weighted RAM and adaptive gating enabled.

License

Code and weights: Apache 2.0, inherited from SAM 2. The VQS-4K dataset is separately licensed CC BY-NC 4.0, research use only.

Citation

@inproceedings{fan2026vqs,
  title     = {Towards Visual Query Segmentation in the Wild},
  author    = {Fan, Bing and Li, Minghao and Zhang, Hanzhi and Dong, Shaohua and
               Mareedu, Naga Prudhvi and Liu, Xiaoqiong and Shi, Weishi and
               Feng, Yunhe and Huang, Yan and Fan, Heng},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)
               Datasets and Benchmarks Track},
  year      = {2026},
  eprint    = {2603.08898},
  archivePrefix = {arXiv}
}