OmniGF

Weights for OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following (NeurIPS 2026 Spotlight, paper). Code, training and evaluation instructions: github.com/cvlab-stonybrook/omnigf.

Each file holds only the weights OmniGF trains: the LoRA adapters (rank 16 on q_proj / v_proj), the head encoder, the gaze heads and the added <|gaze_pad|> token embedding. The Qwen3-VL-4B-Instruct backbone is downloaded from the Hub when the model is built.

File	Task	Paper
omnigf_gazefollow.safetensors	Gaze target estimation, GazeFollow	Table 1
omnigf_gazefollow_inout.safetensors	GazeFollow with in/out loss; warm start for VAT, ChildPlay and VSGaze
omnigf_vat.safetensors	Gaze target estimation, VideoAttentionTarget	Table 1
omnigf_childplay.safetensors	Gaze target estimation, ChildPlay	Table 2
omnigf_semantic_gazefollow.safetensors	Semantic gaze prediction, GazeFollow	Table 4
omnigf_social_vsgaze.safetensors	Social gaze prediction, VSGaze	Table 6

omnigf_social_vsgaze.safetensors is a re-run of the Table 6 recipe, because the original checkpoint was lost. On the VSGaze test set it gives Dist 0.099, AP_IO 0.992, F1_LAH 0.816, F1_LAEO 0.756 and AP_SA 0.758, against 0.098 / 0.992 / 0.814 / 0.742 / 0.787 in the paper.

Usage

From the root of the code repository:

hf download StonyBrook-CVLab/omnigf --local-dir checkpoints
hf download Idiap/resnet18-gaze360 gaze360_resnet18.pt --local-dir checkpoints
bash scripts/test_gazefollow.sh

The launchers look for these file names under ./checkpoints. Keep transformers==5.2.0; see the code repository's README.

License

CC BY-NC 4.0. The head encoder is initialised from Idiap's ResNet-18 Gaze360 model, which is licensed CC BY-NC 4.0. Qwen3-VL-4B-Instruct is licensed Apache-2.0.

Citation

@inproceedings{miao2026omnigf,
  title     = {OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following},
  author    = {Miao, Qiaomu and Wu, Haoyu and Xu, Jingyi and Hoai, Minh and Samaras, Dimitris},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}