Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
Add benchmark tables vs Qwen3.5-4B (iter11750)
Browse files- README.md +102 -19
- README_CN.md +102 -19
README.md
CHANGED
|
@@ -201,30 +201,113 @@ Defaults live in the `codec` field of `preprocessor_config.json` — 512 candida
|
|
| 201 |
|
| 202 |
## Benchmarks
|
| 203 |
|
| 204 |
-
|
| 205 |
-
Do NOT reuse the preview base model's table — those numbers belong to a
|
| 206 |
-
different checkpoint and would misreport this one. -->
|
| 207 |
|
| 208 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 209 |
|
| 210 |
## Evaluation protocol
|
| 211 |
|
| 212 |
-
|
| 213 |
|
| 214 |
-
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
| MVBench, NextQA, TempCompass, MMVU, ReVSI | 64 | 153664 |
|
| 226 |
-
|
| 227 |
-
Most tasks are re-checked by a judge model (Qwen3.8-27B, temperature 0) that sees the question, options, reference answer and raw output and answers YES or NO, recovering answers that are correct but miss the extraction regex; the reported score is the greater of the rule-based and judged scores. Tasks scored by geometry (IoU, point-in-mask) skip the judge and use their rule-based score.
|
| 228 |
|
| 229 |
## Architecture
|
| 230 |
|
|
|
|
| 201 |
|
| 202 |
## Benchmarks
|
| 203 |
|
| 204 |
+
OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled**. Images are scored once: the codec path applies to video only. Empty cells (`--`) mean the result is not yet available or not applicable.
|
|
|
|
|
|
|
| 205 |
|
| 206 |
+
### Image
|
| 207 |
+
|
| 208 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B |
|
| 209 |
+
|---|---|---|
|
| 210 |
+
| **General perception and reading** | | |
|
| 211 |
+
| OCRBench v2 | 59 | 53.67 |
|
| 212 |
+
| OCRBench | 85.2 | 79.8 |
|
| 213 |
+
| OmniDocBench | 41.97 | -- |
|
| 214 |
+
| PerceptionBench | 30.87 | 29.6 |
|
| 215 |
+
| MMStar | 66.87 | 63.93 |
|
| 216 |
+
| MMBench-EN dev | 88.84 | 88.22 |
|
| 217 |
+
| RealWorldQA | 74.38 | 73.86 |
|
| 218 |
+
| BLINK | 54.97 | 67.06 |
|
| 219 |
+
| V*-Bench | 81.68 | 89.53 |
|
| 220 |
+
| DocVQA val | 95.21 | 92.92 |
|
| 221 |
+
| ChartQA | 84.96 | 82.76 |
|
| 222 |
+
| AI2D | 82.35 | 82.16 |
|
| 223 |
+
| CountBench | 87.17 | 84.93 |
|
| 224 |
+
| PixMo-Count | 63.11 | 69.85 |
|
| 225 |
+
| VLMs are Blind | 80.43 | 77.23 |
|
| 226 |
+
| VLMs are Biased | 24.96 | 25.29 |
|
| 227 |
+
| MuirBench | 21.27 | 48.69 |
|
| 228 |
+
| **Grounding and spatial** | | |
|
| 229 |
+
| RefSpatial-Bench | 29.14 | 62.09 |
|
| 230 |
+
| RefCOCO val (Acc@0.5) | 89.07 | 92.64 |
|
| 231 |
+
| RefCOCOg val (Acc@0.5) | 87.65 | 88.91 |
|
| 232 |
+
| CrossPoint | 33.7 | 80.5 |
|
| 233 |
+
| SAT mcq circular | 68.67 | 74 |
|
| 234 |
+
| CV-Bench 2D | 80.74 | 83.31 |
|
| 235 |
+
| CV-Bench 3D | 92 | 94.5 |
|
| 236 |
+
| EmbSpatial | 74.7 | 83.02 |
|
| 237 |
+
| CRPE relation | 77.8 | 78.22 |
|
| 238 |
+
| ERQA | 43.5 | 46.5 |
|
| 239 |
+
| MMSI-Bench | 31.4 | 33.3 |
|
| 240 |
+
| MetaVQA | 61.75 | 61.03 |
|
| 241 |
+
| PointBench | 17.09 | 29.68 |
|
| 242 |
+
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 243 |
+
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 244 |
+
|
| 245 |
+
\* BLINK: the rule-based score reads the first letter of the answer. About 15% of Qwen's answers reason first and then give the option, and are scored wrong; recomputed from the final option Qwen is around 62.
|
| 246 |
+
\* MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
|
| 247 |
+
\* MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
|
| 248 |
+
\* DocVQA val: ANLS rule-based score.
|
| 249 |
+
\* PerceptionBench: judged by gpt-oss-120b.
|
| 250 |
+
\* PointBench: both sides emit 0–1000 coordinates while scoring is mostly in absolute pixels, so both numbers are low.
|
| 251 |
+
\* RefSpatial-Bench: Qwen re-run greedy on 2026-11-17 under the current scoring script.
|
| 252 |
+
\* SAT free-form and SAT mcq are omitted — all 150 correct options sit in first position, so answering "A" every time scores 100%. The row reported is the circular variant, which has a cyclic-consistency check.
|
| 253 |
+
|
| 254 |
+
### Video
|
| 255 |
+
|
| 256 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 257 |
+
|---|---|---|---|
|
| 258 |
+
| **General video understanding** | | | |
|
| 259 |
+
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 260 |
+
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 261 |
+
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 262 |
+
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 263 |
+
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 264 |
+
| MMVU val | 52.6 | 44.1 | -- |
|
| 265 |
+
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 266 |
+
| EgoSchema subset | 68.2 | 80 | -- |
|
| 267 |
+
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 268 |
+
| MMOU test-mini | -- | 45.96 | -- |
|
| 269 |
+
| **VideoMME** | | | |
|
| 270 |
+
| short, no subtitles | 81.67 | 79.67 | 81.11 |
|
| 271 |
+
| medium, no subtitles | 70.33 | 70.56 | 74.11 |
|
| 272 |
+
| long, no subtitles | 63 | 59 | 60.56 |
|
| 273 |
+
| short, subtitles | -- | 82.44 | 82.67 |
|
| 274 |
+
| medium, subtitles | -- | 75.44 | 75.56 |
|
| 275 |
+
| long, subtitles | -- | 65.78 | 65.89 |
|
| 276 |
+
| **Video spatial** | | | |
|
| 277 |
+
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 278 |
+
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 279 |
+
| **Long video** | | | |
|
| 280 |
+
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 281 |
+
| MLVU dev | 73.83 | 75.3 | -- |
|
| 282 |
+
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 283 |
+
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 284 |
+
| **Temporal grounding (mIoU)** | | | |
|
| 285 |
+
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 286 |
+
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 287 |
+
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 288 |
+
|
| 289 |
+
\* TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
|
| 290 |
+
\* Video-MME-v2: subtitles are fed differently on the two sides.
|
| 291 |
+
\* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 292 |
+
\* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 293 |
+
\* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
| 294 |
+
\* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 295 |
|
| 296 |
## Evaluation protocol
|
| 297 |
|
| 298 |
+
Qwen3.5-4B runs with thinking disabled. Some of its results came from temperature-0.7 sampling rather than greedy decoding; rows that have since been re-run greedy use the re-run. Every number is a single run, with no variance estimate.
|
| 299 |
|
| 300 |
+
**Images.** OV2.5 uses the processor's default pixel budget. Qwen uses 65,536–16,777,216 pixels per image.
|
| 301 |
+
|
| 302 |
+
**Video.** Frames are sampled at fps=1 and cached offline, so evaluation decodes nothing and repeated runs see identical input. For each task both sides report the best score over every setting that produced one, so the frame count and per-frame pixel budget vary by row and by model — on LVBench, for instance, Qwen's best came out of 27 settings and ours out of 16. The codec column uses the canvas budgets from the `codec` field of `preprocessor_config.json`, adjusted per task.
|
| 303 |
+
|
| 304 |
+
**Scoring.** Three modes appear above:
|
| 305 |
+
|
| 306 |
+
- *judge* — both sides re-checked by Qwen3.8-27B at temperature 0, which sees the question, options, reference answer and raw output and answers YES or NO. This recovers answers that are correct but miss the extraction regex. The reported score is the greater of the rule-based and judged scores.
|
| 307 |
+
- *rule, official metric* — computed directly by the benchmark's own metric.
|
| 308 |
+
- *rule, judge pending* — the judge should apply but has not run on at least one side, so the rule-based score stands in: MuirBench, EmbSpatial, CRPE relation, ERQA, MetaVQA, NextQA MC test, TempCompass MC, EgoSchema subset, Video-Index 512f.
|
| 309 |
+
|
| 310 |
+
All generation is greedy with `batch_size=1`.
|
|
|
|
|
|
|
|
|
|
| 311 |
|
| 312 |
## Architecture
|
| 313 |
|
README_CN.md
CHANGED
|
@@ -201,30 +201,113 @@ inputs = processor(text=text, videos=["example.mp4"],
|
|
| 201 |
|
| 202 |
## 评测结果
|
| 203 |
|
| 204 |
-
|
| 205 |
-
不要沿用 preview base 模型的那张表——那些数字属于另一个 checkpoint,
|
| 206 |
-
放在这里会误报本模型的能力。 -->
|
| 207 |
|
| 208 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 209 |
|
| 210 |
## 评测口径
|
| 211 |
|
| 212 |
-
|
| 213 |
|
| 214 |
-
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
| MVBench、NextQA、TempCompass、MMVU、ReVSI | 64 | 153664 |
|
| 226 |
-
|
| 227 |
-
多数任务会再经一个 judge 模型(Qwen3.8-27B,温度 0)复核:它看到问题、选项、参考答案和原始输出,回答 YES 或 NO,用于捞回"答对了但不匹配抽取正则"的情况;最终分取规则分与 judge 分中的较大者。纯几何判分的任务(IoU、点落在掩码内)跳过 judge,直接用规则分。
|
| 228 |
|
| 229 |
## 结构参数
|
| 230 |
|
|
|
|
| 201 |
|
| 202 |
## 评测结果
|
| 203 |
|
| 204 |
+
OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**。图像只有一列分数,因为 codec 路径只适用于视频。空格(`--`)表示尚未测出或不适用。
|
|
|
|
|
|
|
| 205 |
|
| 206 |
+
### 图像
|
| 207 |
+
|
| 208 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B |
|
| 209 |
+
|---|---|---|
|
| 210 |
+
| **通用感知与阅读** | | |
|
| 211 |
+
| OCRBench v2 | 59 | 53.67 |
|
| 212 |
+
| OCRBench | 85.2 | 79.8 |
|
| 213 |
+
| OmniDocBench | 41.97 | -- |
|
| 214 |
+
| PerceptionBench | 30.87 | 29.6 |
|
| 215 |
+
| MMStar | 66.87 | 63.93 |
|
| 216 |
+
| MMBench-EN dev | 88.84 | 88.22 |
|
| 217 |
+
| RealWorldQA | 74.38 | 73.86 |
|
| 218 |
+
| BLINK | 54.97 | 67.06 |
|
| 219 |
+
| V*-Bench | 81.68 | 89.53 |
|
| 220 |
+
| DocVQA val | 95.21 | 92.92 |
|
| 221 |
+
| ChartQA | 84.96 | 82.76 |
|
| 222 |
+
| AI2D | 82.35 | 82.16 |
|
| 223 |
+
| CountBench | 87.17 | 84.93 |
|
| 224 |
+
| PixMo-Count | 63.11 | 69.85 |
|
| 225 |
+
| VLMs are Blind | 80.43 | 77.23 |
|
| 226 |
+
| VLMs are Biased | 24.96 | 25.29 |
|
| 227 |
+
| MuirBench | 21.27 | 48.69 |
|
| 228 |
+
| **定位与空间** | | |
|
| 229 |
+
| RefSpatial-Bench | 29.14 | 62.09 |
|
| 230 |
+
| RefCOCO val (Acc@0.5) | 89.07 | 92.64 |
|
| 231 |
+
| RefCOCOg val (Acc@0.5) | 87.65 | 88.91 |
|
| 232 |
+
| CrossPoint | 33.7 | 80.5 |
|
| 233 |
+
| SAT mcq circular | 68.67 | 74 |
|
| 234 |
+
| CV-Bench 2D | 80.74 | 83.31 |
|
| 235 |
+
| CV-Bench 3D | 92 | 94.5 |
|
| 236 |
+
| EmbSpatial | 74.7 | 83.02 |
|
| 237 |
+
| CRPE relation | 77.8 | 78.22 |
|
| 238 |
+
| ERQA | 43.5 | 46.5 |
|
| 239 |
+
| MMSI-Bench | 31.4 | 33.3 |
|
| 240 |
+
| MetaVQA | 61.75 | 61.03 |
|
| 241 |
+
| PointBench | 17.09 | 29.68 |
|
| 242 |
+
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 243 |
+
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 244 |
+
|
| 245 |
+
\* BLINK:规则分取回答首字母。Qwen 约 15% 的回答先分析后给选项,因而被判错;按最终选项复算约为 62。
|
| 246 |
+
\* MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
|
| 247 |
+
\* MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
|
| 248 |
+
\* DocVQA val:按 ANLS 规则分。
|
| 249 |
+
\* PerceptionBench:由 gpt-oss-120b 判分。
|
| 250 |
+
\* PointBench:两边都输出 0~1000 坐标,而判分多按绝对像素算,因此两边绝对值都偏低。
|
| 251 |
+
\* RefSpatial-Bench:Qwen 为 2026-11-17 按现行判分脚本贪心重跑的结果。
|
| 252 |
+
\* SAT 自由作答与 SAT mcq 未列入——其 150 道题的正确选项全在第一位,一律答 A 即得 100%。表中报的是带循环一致性检查的 circular 变体。
|
| 253 |
+
|
| 254 |
+
### 视频
|
| 255 |
+
|
| 256 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 257 |
+
|---|---|---|---|
|
| 258 |
+
| **通用视频理解** | | | |
|
| 259 |
+
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 260 |
+
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 261 |
+
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 262 |
+
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 263 |
+
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 264 |
+
| MMVU val | 52.6 | 44.1 | -- |
|
| 265 |
+
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 266 |
+
| EgoSchema subset | 68.2 | 80 | -- |
|
| 267 |
+
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 268 |
+
| MMOU test-mini | -- | 45.96 | -- |
|
| 269 |
+
| **VideoMME** | | | |
|
| 270 |
+
| short,无字幕 | 81.67 | 79.67 | 81.11 |
|
| 271 |
+
| medium,无字幕 | 70.33 | 70.56 | 74.11 |
|
| 272 |
+
| long,无字幕 | 63 | 59 | 60.56 |
|
| 273 |
+
| short,有字幕 | -- | 82.44 | 82.67 |
|
| 274 |
+
| medium,有字幕 | -- | 75.44 | 75.56 |
|
| 275 |
+
| long,有字幕 | -- | 65.78 | 65.89 |
|
| 276 |
+
| **视频空间** | | | |
|
| 277 |
+
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 278 |
+
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 279 |
+
| **长视频** | | | |
|
| 280 |
+
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 281 |
+
| MLVU dev | 73.83 | 75.3 | -- |
|
| 282 |
+
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 283 |
+
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 284 |
+
| **时间定位(mIoU)** | | | |
|
| 285 |
+
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 286 |
+
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 287 |
+
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 288 |
+
|
| 289 |
+
\* TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
|
| 290 |
+
\* Video-MME-v2:字幕送法两边不同。
|
| 291 |
+
\* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 292 |
+
\* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 293 |
+
\* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
| 294 |
+
\* codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 295 |
|
| 296 |
## 评测口径
|
| 297 |
|
| 298 |
+
Qwen3.5-4B 关闭 thinking。它有一部分结果实际来自 temperature=0.7 采样而非贪心解码;已做过贪心重跑的行用重跑结果。所有数字都是单次运行,没有方差估计���
|
| 299 |
|
| 300 |
+
**图像。** OV2.5 用 processor 的默认像素预算;Qwen 为每图 65,536~16,777,216 px。
|
| 301 |
+
|
| 302 |
+
**视频。** 按 fps=1 抽帧并离线缓存,评测时不解码,重复跑输入完全一致。每个任务两边各取所有出分设置中的最高分,因此帧数和逐帧像素预算逐行、逐模型都不同——例如 LVBench,Qwen 的最高分出自 27 个设置,我们出自 16 个。codec 一列用 `preprocessor_config.json` 中 `codec` 字段的画布预算,按任务调整。
|
| 303 |
+
|
| 304 |
+
**判分。** 表中出现三种口径:
|
| 305 |
+
|
| 306 |
+
- *judge*——两边都由 Qwen3.8-27B(温度 0)复判,它看到问题、选项、参考答案和原始输出,回答 YES 或 NO,用于捞回"答对了但不匹配抽取正则"的情况。最终分取规则分与 judge 分中的较大者。
|
| 307 |
+
- *规则分(官方指标)*——按 bench 自己的官方指标直接计算。
|
| 308 |
+
- *规则分(judge 未出)*——本应用 judge,但至少一边尚未判,暂用规则分:MuirBench、EmbSpatial、CRPE relation、ERQA、MetaVQA、NextQA MC test、TempCompass MC、EgoSchema subset、Video-Index 512f。
|
| 309 |
+
|
| 310 |
+
所有生成都是贪心,`batch_size=1`。
|
|
|
|
|
|
|
|
|
|
| 311 |
|
| 312 |
## 结构参数
|
| 313 |
|