yiyexy commited on
Commit
099bdbd
·
verified ·
1 Parent(s): e4bb786

BLINK judged score; per-model settings in the video table

Browse files
Files changed (2) hide show
  1. README.md +37 -36
  2. README_CN.md +37 -36
README.md CHANGED
@@ -21,7 +21,7 @@ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image,
21
 
22
  - **One model, two modes.** `apply_chat_template(..., streaming=True)` switches the default system prompt; eight protocol tokens carry the per-frame decision to stay silent, speak, or caption. Nothing else about the input changes between modes.
23
  - **Two video front-ends.** Uniform frame sampling, or *codec mode*, which uses the encoder's bitcost to pack informative regions into a handful of canvases. On this checkpoint codec is ahead on **17 of the 21** video benchmarks where both were run.
24
- - **Strong video and spatial results.** Against Qwen3.5-4B (thinking off), frame sampling leads on **17 of 21** comparable video benchmarks and codec on **16 of 18**, with the largest margins on temporal grounding (TimeLens +12) and spatial video (VSI-Bench +9.4).
25
  - **Honest about images.** On still images the comparison is mixed — ahead on 18 of 31 comparable benchmarks, behind on OCR and document reading. The tables below are unfiltered.
26
 
27
  ## Model Overview
@@ -45,12 +45,11 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
45
  | **General perception and reading** | | |
46
  | OCRBench v2 | 59 | 53.67 |
47
  | OCRBench | 85.2 | 79.8 |
48
- | OmniDocBench | 41.97 | -- |
49
  | PerceptionBench | 30.87 | 29.6 |
50
  | MMStar | 66.87 | 63.93 |
51
  | MMBench-EN dev | 88.84 | 88.22 |
52
  | RealWorldQA | 74.38 | 73.86 |
53
- | BLINK | 54.97 | 67.06 |
54
  | V*-Bench | 81.68 | 89.53 |
55
  | DocVQA val | 95.21 | 92.92 |
56
  | ChartQA | 84.96 | 82.76 |
@@ -77,7 +76,7 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
77
  | TraceSpatial 2D | 37.92 | 19.41 |
78
  | TraceSpatial 3D | 12.99 | 44.6 |
79
 
80
- \* BLINK: the rule-based score reads the first letter of the answer. About 15% of Qwen's answers reason first and then give the option, and are scored wrong; recomputed from the final option Qwen is around 62.
81
  \* MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
82
  \* MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
83
  \* DocVQA val: ANLS rule-based score. PerceptionBench: judged by gpt-oss-120b.
@@ -87,44 +86,46 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
87
 
88
  ### Video
89
 
90
- | Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
91
- |---|---|---|---|
92
- | **General video understanding** | | | |
93
- | MotionBench | 60.1 | 62.39 | 64.19 |
94
- | TOMATO | 34.91 | 33.63 | 38.61 |
95
- | MVBench | 64.03 | 72.03 | 75.5 |
96
- | NextQA MC test | 80.67 | 86.06 | 86.4 |
97
- | TempCompass MC | 73.35 | 75.76 | 79.68 |
98
- | MMVU val | 52.6 | 44.1 | -- |
99
- | Video-MME-v2 | 15.53 | 16.43 | 17.2 |
100
- | EgoSchema subset | 68.2 | 80 | -- |
101
- | Video-Index 512f | 19.64 | 20 | 19.76 |
102
- | MMOU test-mini | -- | 45.96 | -- |
103
- | **VideoMME** | | | |
104
- | short, no subtitles | 81.67 | 79.67 | 81.11 |
105
- | medium, no subtitles | 70.33 | 70.56 | 74.11 |
106
- | long, no subtitles | 63 | 59 | 60.56 |
107
- | short, subtitles | -- | 82.44 | 82.67 |
108
- | medium, subtitles | -- | 75.44 | 75.56 |
109
- | long, subtitles | -- | 65.78 | 65.89 |
110
- | **Video spatial** | | | |
111
- | VSI-Bench | 59.22 | 68.64 | 67.72 |
112
- | ReVSI 64f | 47.1 | 55.94 | 55.54 |
113
- | **Long video** | | | |
114
- | LVBench | 56.49 | 62.04 | 61.59 |
115
- | MLVU dev | 73.83 | 75.3 | -- |
116
- | LongVideoBench val | 62.68 | 68.96 | 71.06 |
117
- | VideoEval-Pro | 62.84 | 64.39 | 64.62 |
118
- | **Temporal grounding (mIoU)** | | | |
119
- | TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
120
- | TimeLens Charades | 48.05 | 60.58 | 61.25 |
121
- | TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
122
 
123
  \* TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
124
  \* Video-MME-v2: subtitles are fed differently on the two sides.
125
  \* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
126
  \* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
127
  \* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
 
 
128
  \* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
129
 
130
  ### Evaluation setup
 
21
 
22
  - **One model, two modes.** `apply_chat_template(..., streaming=True)` switches the default system prompt; eight protocol tokens carry the per-frame decision to stay silent, speak, or caption. Nothing else about the input changes between modes.
23
  - **Two video front-ends.** Uniform frame sampling, or *codec mode*, which uses the encoder's bitcost to pack informative regions into a handful of canvases. On this checkpoint codec is ahead on **17 of the 21** video benchmarks where both were run.
24
+ - **Strong video and spatial results.** Against Qwen3.5-4B (thinking off), frame sampling leads on **17 of 21** comparable video benchmarks and codec on **16 of 18**, with the largest matched-setting margins on temporal grounding (TimeLens Charades +12.5) and spatial video (VSI-Bench +9.4).
25
  - **Honest about images.** On still images the comparison is mixed — ahead on 18 of 31 comparable benchmarks, behind on OCR and document reading. The tables below are unfiltered.
26
 
27
  ## Model Overview
 
45
  | **General perception and reading** | | |
46
  | OCRBench v2 | 59 | 53.67 |
47
  | OCRBench | 85.2 | 79.8 |
 
48
  | PerceptionBench | 30.87 | 29.6 |
49
  | MMStar | 66.87 | 63.93 |
50
  | MMBench-EN dev | 88.84 | 88.22 |
51
  | RealWorldQA | 74.38 | 73.86 |
52
+ | BLINK | 63.56 | 67.06 |
53
  | V*-Bench | 81.68 | 89.53 |
54
  | DocVQA val | 95.21 | 92.92 |
55
  | ChartQA | 84.96 | 82.76 |
 
76
  | TraceSpatial 2D | 37.92 | 19.41 |
77
  | TraceSpatial 3D | 12.99 | 44.6 |
78
 
79
+ \* BLINK: 63.56 is Qwen's judged score. Its rule-based score was 54.97 — the extraction reads the first letter, and about 15% of its answers reason before giving the option.
80
  \* MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
81
  \* MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
82
  \* DocVQA val: ANLS rule-based score. PerceptionBench: judged by gpt-oss-120b.
 
86
 
87
  ### Video
88
 
89
+ | Benchmark | Qwen3.5-4B | setting | OV2.5-4B (frames) | setting | OV2.5-4B (codec) | setting |
90
+ |---|---|---|---|---|---|---|
91
+ | **General video understanding** | | | | | | |
92
+ | MotionBench | 60.1 | 128f × 153,664 | 62.39 | 128f × 153,664 | 64.19 | TC128 × 313,600 |
93
+ | TOMATO | 34.91 | 16f × 153,664, fps=2 | 33.63 | model default | 38.61 | TC128 × 313,600 |
94
+ | MVBench | 64.03 | 64f × 153,664 | 72.03 | 64f × 153,664 | 75.5 | TC64 × 313,600 |
95
+ | NextQA MC test | 80.67 | 64f × 153,664 | 86.06 | 64f × 153,664 | 86.4 | TC64 × 313,600 |
96
+ | TempCompass MC | 73.35 | 64f × 153,664 | 75.76 | 64f × 153,664 | 79.68 | TC64 × 313,600 |
97
+ | MMVU val | 52.6 | 64f × 153,664 | 44.1 | 64f × 153,664 | -- | -- |
98
+ | Video-MME-v2 | 15.53 | 64f × 330,000 | 16.43 | 64f × 330,000 | 17.2 | TC64 × 313,600 |
99
+ | EgoSchema subset | 68.2 | 64f × 153,664 | 80 | 64f × 153,664 | -- | -- |
100
+ | Video-Index 512f | 19.64 | task frames, ≤262,144/f | 20 | task frames, ≤262,144/f | 19.76 | TC≤256 × 313,600 |
101
+ | MMOU test-mini | -- | -- | 45.96 | model default | -- | -- |
102
+ | **VideoMME** | | | | | | |
103
+ | short, no subtitles | 81.67 | 128f × 321,489 | 79.67 | 128f × 321,489 | 81.11 | TC256 × 313,600 |
104
+ | medium, no subtitles | 70.33 | 256f × 136,900 | 70.56 | 256f × 136,900 | 74.11 | TC512 × 313,600 |
105
+ | long, no subtitles | 63 | 1450f × 120,000 | 59 | 750f × 250,000 | 60.56 | TC512 × 313,600 |
106
+ | short, subtitles | -- | -- | 82.44 | 128f × 321,489 | 82.67 | TC256 × 313,600 |
107
+ | medium, subtitles | -- | -- | 75.44 | 256f × 136,900 | 75.56 | TC256 × 313,600 |
108
+ | long, subtitles | -- | -- | 65.78 | 750f × 250,000 | 65.89 | TC256 × 313,600 |
109
+ | **Video spatial** | | | | | | |
110
+ | VSI-Bench | 59.22 | 128f × 153,664 | 68.64 | 128f × 153,664 | 67.72 | TC128 × 313,600 |
111
+ | ReVSI 64f | 47.1 | 64f × 153,664 | 55.94 | 64f × 153,664 | 55.54 | TC64 × 313,600 |
112
+ | **Long video** | | | | | | |
113
+ | LVBench | 56.49 | 2048f × 120,000 | 62.04 | 950f × 200,000 | 61.59 | TC1024 × 156,800 |
114
+ | MLVU dev | 73.83 | 512f × 72,900 | 75.3 | 512f × 72,900 | -- | -- |
115
+ | LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
116
+ | VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
117
+ | **Temporal grounding (mIoU)** | | | | | | |
118
+ | TimeLens ActivityNet | 46.83 | 128f × 153,664 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
119
+ | TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
120
+ | TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
121
 
122
  \* TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
123
  \* Video-MME-v2: subtitles are fed differently on the two sides.
124
  \* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
125
  \* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
126
  \* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
127
+ \* Settings are the ones that produced each reported score. Most rows are matched, but three are not: on **TimeLens ActivityNet** we ran 2048 frames × 409,600 px against Qwen's 128 × 153,664, so that +12.01 is not like-for-like (TimeLens Charades, at identical settings, is +12.53). On **TOMATO** Qwen ran 16 frames at fps=2 while we used the model defaults, and on **MMOU** only we have a score.
128
+ \* Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
129
  \* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
130
 
131
  ### Evaluation setup
README_CN.md CHANGED
@@ -21,7 +21,7 @@ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图
21
 
22
  - **一个模型,两种模式。** `apply_chat_template(..., streaming=True)` 切换默认 system prompt;八个协议 token 承载每一帧"沉默 / 发言 / caption"的决策。除此之外输入格式在两种模式下完全相同。
23
  - **两条视频前端。** 均匀抽帧,或 **codec 模式**——用编码器的 bitcost 把信息量大的区域打包进少量画布。在本 checkpoint 上,两边都跑出结果的 21 个视频 bench 中,codec **赢 17 个**。
24
- - **视频与空间能力突出。** 对 Qwen3.5-4B(关闭 thinking),抽帧在 21 个可比视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**;优势最大的是时间定位(TimeLens +12)和视频空间(VSI-Bench +9.4)。
25
  - **图像侧如实呈现。** 静态图像上互有胜负——31 个可比项中领先 18 个,OCR 与文档阅读落后。下面的表格未作筛选。
26
 
27
  ## 模型概览
@@ -45,12 +45,11 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
45
  | **通用感知与阅读** | | |
46
  | OCRBench v2 | 59 | 53.67 |
47
  | OCRBench | 85.2 | 79.8 |
48
- | OmniDocBench | 41.97 | -- |
49
  | PerceptionBench | 30.87 | 29.6 |
50
  | MMStar | 66.87 | 63.93 |
51
  | MMBench-EN dev | 88.84 | 88.22 |
52
  | RealWorldQA | 74.38 | 73.86 |
53
- | BLINK | 54.97 | 67.06 |
54
  | V*-Bench | 81.68 | 89.53 |
55
  | DocVQA val | 95.21 | 92.92 |
56
  | ChartQA | 84.96 | 82.76 |
@@ -77,7 +76,7 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
77
  | TraceSpatial 2D | 37.92 | 19.41 |
78
  | TraceSpatial 3D | 12.99 | 44.6 |
79
 
80
- \* BLINK:规则分取回答首字母。Qwen 约 15% 的回答先分析后给选项,因而被判错;按最终选项复算约为 62。
81
  \* MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
82
  \* MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
83
  \* DocVQA val:按 ANLS 规则分。PerceptionBench:由 gpt-oss-120b 判分。
@@ -87,44 +86,46 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
87
 
88
  ### 视频
89
 
90
- | Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
91
- |---|---|---|---|
92
- | **通用视频理解** | | | |
93
- | MotionBench | 60.1 | 62.39 | 64.19 |
94
- | TOMATO | 34.91 | 33.63 | 38.61 |
95
- | MVBench | 64.03 | 72.03 | 75.5 |
96
- | NextQA MC test | 80.67 | 86.06 | 86.4 |
97
- | TempCompass MC | 73.35 | 75.76 | 79.68 |
98
- | MMVU val | 52.6 | 44.1 | -- |
99
- | Video-MME-v2 | 15.53 | 16.43 | 17.2 |
100
- | EgoSchema subset | 68.2 | 80 | -- |
101
- | Video-Index 512f | 19.64 | 20 | 19.76 |
102
- | MMOU test-mini | -- | 45.96 | -- |
103
- | **VideoMME** | | | |
104
- | short,无字幕 | 81.67 | 79.67 | 81.11 |
105
- | medium,无字幕 | 70.33 | 70.56 | 74.11 |
106
- | long,无字幕 | 63 | 59 | 60.56 |
107
- | short,有字幕 | -- | 82.44 | 82.67 |
108
- | medium,有字幕 | -- | 75.44 | 75.56 |
109
- | long,有字幕 | -- | 65.78 | 65.89 |
110
- | **视频空间** | | | |
111
- | VSI-Bench | 59.22 | 68.64 | 67.72 |
112
- | ReVSI 64f | 47.1 | 55.94 | 55.54 |
113
- | **长视频** | | | |
114
- | LVBench | 56.49 | 62.04 | 61.59 |
115
- | MLVU dev | 73.83 | 75.3 | -- |
116
- | LongVideoBench val | 62.68 | 68.96 | 71.06 |
117
- | VideoEval-Pro | 62.84 | 64.39 | 64.62 |
118
- | **时间定位(mIoU)** | | | |
119
- | TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
120
- | TimeLens Charades | 48.05 | 60.58 | 61.25 |
121
- | TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
122
 
123
  \* TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
124
  \* Video-MME-v2:字幕送法两边不同。
125
  \* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
126
  \* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
127
  \* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
 
 
128
  \* codec 一列为规则分——回答均为选项字母,与 judge 等价。
129
 
130
  ### 评测口径
 
21
 
22
  - **一个模型,两种模式。** `apply_chat_template(..., streaming=True)` 切换默认 system prompt;八个协议 token 承载每一帧"沉默 / 发言 / caption"的决策。除此之外输入格式在两种模式下完全相同。
23
  - **两条视频前端。** 均匀抽帧,或 **codec 模式**——用编码器的 bitcost 把信息量大的区域打包进少量画布。在本 checkpoint 上,两边都跑出结果的 21 个视频 bench 中,codec **赢 17 个**。
24
+ - **视频与空间能力突出。** 对 Qwen3.5-4B(关闭 thinking),抽帧在 21 个可比视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**;在设置对齐的项上优势最大的是时间定位(TimeLens Charades +12.5)和视频空间(VSI-Bench +9.4)。
25
  - **图像侧如实呈现。** 静态图像上互有胜负——31 个可比项中领先 18 个,OCR 与文档阅读落后。下面的表格未作筛选。
26
 
27
  ## 模型概览
 
45
  | **通用感知与阅读** | | |
46
  | OCRBench v2 | 59 | 53.67 |
47
  | OCRBench | 85.2 | 79.8 |
 
48
  | PerceptionBench | 30.87 | 29.6 |
49
  | MMStar | 66.87 | 63.93 |
50
  | MMBench-EN dev | 88.84 | 88.22 |
51
  | RealWorldQA | 74.38 | 73.86 |
52
+ | BLINK | 63.56 | 67.06 |
53
  | V*-Bench | 81.68 | 89.53 |
54
  | DocVQA val | 95.21 | 92.92 |
55
  | ChartQA | 84.96 | 82.76 |
 
76
  | TraceSpatial 2D | 37.92 | 19.41 |
77
  | TraceSpatial 3D | 12.99 | 44.6 |
78
 
79
+ \* BLINK:63.56 是 Qwen 的 judge 分。其规则分为 54.97——抽取取回答首字母,而它约 15% 的回答先分析后给选项。
80
  \* MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
81
  \* MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
82
  \* DocVQA val:按 ANLS 规则分。PerceptionBench:由 gpt-oss-120b 判分。
 
86
 
87
  ### 视频
88
 
89
+ | Benchmark | Qwen3.5-4B | 设置 | OV2.5-4B(抽帧) | 设置 | OV2.5-4B(codec) | 设置 |
90
+ |---|---|---|---|---|---|---|
91
+ | **通用视频理解** | | | | | | |
92
+ | MotionBench | 60.1 | 128f × 153,664 | 62.39 | 128f × 153,664 | 64.19 | TC128 × 313,600 |
93
+ | TOMATO | 34.91 | 16f × 153,664, fps=2 | 33.63 | model default | 38.61 | TC128 × 313,600 |
94
+ | MVBench | 64.03 | 64f × 153,664 | 72.03 | 64f × 153,664 | 75.5 | TC64 × 313,600 |
95
+ | NextQA MC test | 80.67 | 64f × 153,664 | 86.06 | 64f × 153,664 | 86.4 | TC64 × 313,600 |
96
+ | TempCompass MC | 73.35 | 64f × 153,664 | 75.76 | 64f × 153,664 | 79.68 | TC64 × 313,600 |
97
+ | MMVU val | 52.6 | 64f × 153,664 | 44.1 | 64f × 153,664 | -- | -- |
98
+ | Video-MME-v2 | 15.53 | 64f × 330,000 | 16.43 | 64f × 330,000 | 17.2 | TC64 × 313,600 |
99
+ | EgoSchema subset | 68.2 | 64f × 153,664 | 80 | 64f × 153,664 | -- | -- |
100
+ | Video-Index 512f | 19.64 | task frames, ≤262,144/f | 20 | task frames, ≤262,144/f | 19.76 | TC≤256 × 313,600 |
101
+ | MMOU test-mini | -- | -- | 45.96 | model default | -- | -- |
102
+ | **VideoMME** | | | | | | |
103
+ | short,无字幕 | 81.67 | 128f × 321,489 | 79.67 | 128f × 321,489 | 81.11 | TC256 × 313,600 |
104
+ | medium,无字幕 | 70.33 | 256f × 136,900 | 70.56 | 256f × 136,900 | 74.11 | TC512 × 313,600 |
105
+ | long,无字幕 | 63 | 1450f × 120,000 | 59 | 750f × 250,000 | 60.56 | TC512 × 313,600 |
106
+ | short,有字幕 | -- | -- | 82.44 | 128f × 321,489 | 82.67 | TC256 × 313,600 |
107
+ | medium,有字幕 | -- | -- | 75.44 | 256f × 136,900 | 75.56 | TC256 × 313,600 |
108
+ | long,有字幕 | -- | -- | 65.78 | 750f × 250,000 | 65.89 | TC256 × 313,600 |
109
+ | **视频空间** | | | | | | |
110
+ | VSI-Bench | 59.22 | 128f × 153,664 | 68.64 | 128f × 153,664 | 67.72 | TC128 × 313,600 |
111
+ | ReVSI 64f | 47.1 | 64f × 153,664 | 55.94 | 64f × 153,664 | 55.54 | TC64 × 313,600 |
112
+ | **长视频** | | | | | | |
113
+ | LVBench | 56.49 | 2048f × 120,000 | 62.04 | 950f × 200,000 | 61.59 | TC1024 × 156,800 |
114
+ | MLVU dev | 73.83 | 512f × 72,900 | 75.3 | 512f × 72,900 | -- | -- |
115
+ | LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
116
+ | VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
117
+ | **时间定位(mIoU)** | | | | | | |
118
+ | TimeLens ActivityNet | 46.83 | 128f × 153,664 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
119
+ | TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
120
+ | TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
121
 
122
  \* TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
123
  \* Video-MME-v2:字幕送法两边不同。
124
  \* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
125
  \* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
126
  \* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
127
+ \* 设置列是各自取得该分数时的实际设置。多数行两边一致,但有三行不是:**TimeLens ActivityNet** 我方跑的是 2048 帧 × 409,600 px,Qwen 是 128 × 153,664,因此那 +12.01 并非同条件对比(设置完全相同的 TimeLens Charades 为 +12.53);**TOMATO** Qwen 为 16 帧 fps=2 而我方用模型默认值;**MMOU** 只有我方有分。
128
+ \* 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
129
  \* codec 一列为规则分——回答均为选项字母,与 judge 等价。
130
 
131
  ### 评测口径