| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-VL-8B-Instruct |
| pipeline_tag: video-text-to-text |
| library_name: transformers |
| tags: |
| - video-understanding |
| - reinforcement-learning |
| - adaptive-reasoning |
| - qwen3-vl |
| --- |
| |
| # EVAThink-Qwen3 |
|
|
| <!-- TODO(author): one-line expansion of the name, paper title / arXiv link, code repo link --> |
|
|
| **EVAThink-Qwen3** is a vision-language model fine-tuned from [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) with reinforcement learning to **decide per question whether to think**. When the visual evidence plainly supports an answer, it replies directly. When the evidence is sparse, discontinuous, or scattered across frames, it writes a short, evidence-grounded reasoning trace first. |
|
|
| You don't need to tell it which mode to use. Give it the plain question and options, and it picks the mode itself. |
|
|
| ## Highlights |
|
|
| - **Adaptive, not always-on, reasoning.** How often it thinks follows how hard the task is to reason about. On purely perceptual spatial questions (VSI-Bench) it thinks on 0% of questions. On knowledge-heavy lecture videos (Video-MMMU) it thinks on 86%. Within a benchmark, the rate rises with video length: on Video-MME it goes 14% → 41% → 48% from short to medium to long videos. |
| - **Short outputs.** Mean response length is 4 tokens on VSI-Bench, 14 on MVBench, 95 on LongVideoBench, 117 on Video-MME, 257 on MMVU, and 794 on Video-MMMU, where it thinks on 86% of questions. |
| - **Better than the base model or on par on all six benchmarks**, and ahead of VideoAuto-R1 (Qwen3-VL-8B) on five of six, all under the same frame budget. |
|
|
| ## Results |
|
|
| All models were evaluated by us under **one shared protocol**: |
| - 64 frames sampled uniformly (first and last frame included) |
| - each frame resized to a fixed 200,704 pixels (factor 28) |
| - temperature=0.0, greedy decoding |
|
|
| Each model uses its intended prompt: |
| - **EVAThink**: the bare question and options, with no instruction. |
| - **Qwen3-VL-8B-Instruct**: each benchmark's official prompt. |
| - **VideoAuto-R1**: its own answer-twice prompt, with confidence-based early exit. |
|
|
| | Benchmark (split, #Q) | Qwen3-VL-8B-Instruct | VideoAuto-R1-Qwen3-VL-8B | **EVAThink-Qwen3** | EVAThink think-rate | |
| |---|---|---|---|---| |
| | Video-MME w/o subs (2,700) | 64.9 | 66.0 | **66.3** | 34.4% | |
| | MVBench (4,000) | 65.8 | **69.0** | 66.5 | 5.1% | |
| | MMVU, multiple-choice (625) | 70.4 | 69.6 | **72.8** | 59.4% | |
| | Video-MMMU (900) | 65.6 | 64.3 | **67.4** | 85.7% | |
| | LongVideoBench val (1,337) | 60.4 | 54.6 | **60.7** | 46.1% | |
| | VSI-Bench (5,130) | 58.3 | 54.6 | **59.9** | 0.0% | |
|
|
| Notes: |
| - These numbers come from our own fixed-budget protocol and are not directly comparable with numbers in other papers, which use different frame counts, resolutions, and prompts. |
| - MVBench uses the original segment boundaries for the tasks that define them. |
| - VSI-Bench scores numerical questions with mean relative accuracy (MRA) and multiple-choice questions with accuracy; the table reports the average over all questions. |
| - Video-MMMU includes its 21 open-ended questions, scored with the official MMMU rules. |
| - A response counts as "thinking" (think-rate) when it doesn't start directly with the answer. |
|
|
| <!-- TODO(author): add significance / paired-test notes if desired --> |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{evathink2026, |
| title = {TODO}, |
| author = {TODO}, |
| year = {2026}, |
| note = {TODO: arXiv link} |
| } |
| ``` |
|
|
