qianyijie/avsync-evaluator

AV-Sync Evaluator — Checkpoint

Trained checkpoint for the AV-Sync Evaluator, a reference-free audio-visual synchronization metric built on Qwen2.5-Omni-3B. Given a single video clip (audio read from the track), it outputs one scalar sync score — higher means the audio and visuals are better causally aligned.

Official checkpoint for our ECCV 2026 paper "Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models".

• 💻 Code: https://github.com/chenhaoqcdyq/BeyondTimeShifts
• 📄 Paper: arXiv:2607.09091 (ECCV 2026)

Files

• avsync_eval_weights.pt — clean inference state dict (transformer.* + regression_head.*), converted from the DeepSpeed ZeRO training checkpoint.

Usage

import torch
from avsync_eval.models.evaluator import AVSyncEvaluator

model = AVSyncEvaluator(model_name="Qwen/Qwen2.5-Omni-3B", v_fps=12, v_size=140)
ckpt = torch.load("avsync_eval_weights.pt", map_location="cpu")
model.load_eval_checkpoint(ckpt["state_dict"])
model.to_eval_device()                          # -> cuda, bf16, eval()

scores = model.score_batch([video_tensor], [audio_array])   # list[float]

See the GitHub repository for the full inference, evaluation, and training pipeline.

Citation

@inproceedings{qian2026beyond,
  title     = {Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free
               Evaluator for Generative Audio-Visual Models},
  author    = {Qian, Yijie and Wang, Juncheng and Xu, Chao and Wang, Huihan and
               Feng, Yuxiang and Liu, Yang and Sun, Baigui and Liu, Yong and
               Wang, Shujun},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

License

MIT. The Qwen2.5-Omni base weights follow their provider's license.