koenv759/VanillaSFT-LoRA

Vanilla SFT — LoRA (Qwen2.5-Omni-7B)

LoRA adapter for Qwen2.5-Omni-7B, the headline §4 model from "Reasoning for Social Audio-Visual Question Answering: Where Do We Stand?" (ECCV 2026 HCMIW).

Vanilla SFT is a deliberately trivial baseline: plain supervised fine-tuning that outputs only the answer — no reasoning trace — trained on HumanOmniV2's own training data. It matches or beats HumanOmniV2's full chain-of-thought + reinforcement-learning pipeline (and other reasoning methods) across three social audio-visual QA benchmarks, at a fraction of the inference latency and training compute. We argue a baseline like this should be a mandatory point of comparison for any method proposed in this subfield.

This adapter is the LoRA, 1-epoch run (the paper's headline setting; further LoRA training degrades).

Results (our reproductions)

Model	IB Full	IB Clean	IB Hard	WorldSense	Daily-Omni
Base Qwen2.5-Omni-7B	64.58	67.83	63.22	43.69	61.07
HumanOmniV2	68.90	71.76	66.91	47.26	58.40
This model (Vanilla LoRA)	70.60	73.75	70.43	48.77	65.16

IB = IntentBench; Clean/Hard = the IntentBench-Prime variants (broken-removed / broken + text-answerable-removed). All numbers are our own reproductions, scored from the result files shipped in the code repo. Reproduce them with no GPU from that repo.

How to use

This is a LoRA adapter, not a full model — load it on top of Qwen/Qwen2.5-Omni-7B. The supported inference + evaluation path is the code repo, which handles the Qwen2.5-Omni multimodal preprocessing (frames + audio) for you:

• Code: https://github.com/koenv759/VanillaSFT — point LORA_PATH at this adapter in the eval scripts (eval/eval_intentbench.slurm, eval/eval_benchmarks.slurm).

Evaluation protocol used for the numbers above (set these to reproduce):

• Frame sampling FPS = 2, max 32 frames.
• Audio interleaved TMRoPE (USE_AUDIO_IN_VIDEO=True) — not HumanOmniV2's separate-stream protocol (that is an opt-in, SEPARATE_AUDIO=1).
• Direct-answer prompting (PLAIN_SFT=1) — the model emits only the answer, no reasoning trace.

Training

• Base model: Qwen/Qwen2.5-Omni-7B
• Method: LoRA (rank 16, alpha 32, target_modules = all-linear, ViT frozen), via ms-swift
• Data: HumanOmniV2's re-formatted training subset (Social-IQ 2.0 / EMER / Video-R1 / OmniInstruct) with reasoning traces stripped to plain answers — 30,217 examples. The data is not redistributed with this model; see the code repo for how to obtain the videos.
• Schedule: 1 epoch, effective batch 16 (bs 1 × grad-accum 16), lr 1e-4 cosine, warmup 0.05, weight decay 0.01, bf16, DeepSpeed ZeRO-2. ~4 GPUs (H100 in the paper).

Citation

Reasoning for Social Audio-Visual Question Answering: Where Do We Stand? (ECCV HCMIW 2026).

Acknowledgements

Built on HumanOmniV2 (training data + adapted eval), Qwen2.5-Omni-7B (base model), and training via ms-swift.

License

GPL-3.0-or-later, matching the code repository. The base model Qwen2.5-Omni-7B is Apache-2.0 (GPLv3-compatible); note that some of the training data is CC BY-NC-SA 4.0 (non-commercial) — the weights are not redistributed with any of that data.