Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
BLINK judged score; per-model settings in the video table
Browse files- README.md +37 -36
- README_CN.md +37 -36
README.md
CHANGED
|
@@ -21,7 +21,7 @@ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image,
|
|
| 21 |
|
| 22 |
- **One model, two modes.** `apply_chat_template(..., streaming=True)` switches the default system prompt; eight protocol tokens carry the per-frame decision to stay silent, speak, or caption. Nothing else about the input changes between modes.
|
| 23 |
- **Two video front-ends.** Uniform frame sampling, or *codec mode*, which uses the encoder's bitcost to pack informative regions into a handful of canvases. On this checkpoint codec is ahead on **17 of the 21** video benchmarks where both were run.
|
| 24 |
-
- **Strong video and spatial results.** Against Qwen3.5-4B (thinking off), frame sampling leads on **17 of 21** comparable video benchmarks and codec on **16 of 18**, with the largest margins on temporal grounding (TimeLens +12) and spatial video (VSI-Bench +9.4).
|
| 25 |
- **Honest about images.** On still images the comparison is mixed — ahead on 18 of 31 comparable benchmarks, behind on OCR and document reading. The tables below are unfiltered.
|
| 26 |
|
| 27 |
## Model Overview
|
|
@@ -45,12 +45,11 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
|
|
| 45 |
| **General perception and reading** | | |
|
| 46 |
| OCRBench v2 | 59 | 53.67 |
|
| 47 |
| OCRBench | 85.2 | 79.8 |
|
| 48 |
-
| OmniDocBench | 41.97 | -- |
|
| 49 |
| PerceptionBench | 30.87 | 29.6 |
|
| 50 |
| MMStar | 66.87 | 63.93 |
|
| 51 |
| MMBench-EN dev | 88.84 | 88.22 |
|
| 52 |
| RealWorldQA | 74.38 | 73.86 |
|
| 53 |
-
| BLINK |
|
| 54 |
| V*-Bench | 81.68 | 89.53 |
|
| 55 |
| DocVQA val | 95.21 | 92.92 |
|
| 56 |
| ChartQA | 84.96 | 82.76 |
|
|
@@ -77,7 +76,7 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
|
|
| 77 |
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 78 |
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 79 |
|
| 80 |
-
\* BLINK:
|
| 81 |
\* MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
|
| 82 |
\* MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
|
| 83 |
\* DocVQA val: ANLS rule-based score. PerceptionBench: judged by gpt-oss-120b.
|
|
@@ -87,44 +86,46 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
|
|
| 87 |
|
| 88 |
### Video
|
| 89 |
|
| 90 |
-
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 91 |
-
|---|---|---|---|
|
| 92 |
-
| **General video understanding** | | | |
|
| 93 |
-
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 94 |
-
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 95 |
-
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 96 |
-
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 97 |
-
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 98 |
-
| MMVU val | 52.6 | 44.1 | -- |
|
| 99 |
-
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 100 |
-
| EgoSchema subset | 68.2 | 80 | -- |
|
| 101 |
-
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 102 |
-
| MMOU test-mini | -- | 45.96 | -- |
|
| 103 |
-
| **VideoMME** | | | |
|
| 104 |
-
| short, no subtitles | 81.67 | 79.67 | 81.11 |
|
| 105 |
-
| medium, no subtitles | 70.33 | 70.56 | 74.11 |
|
| 106 |
-
| long, no subtitles | 63 | 59 | 60.56 |
|
| 107 |
-
| short, subtitles | -- | 82.44 | 82.67 |
|
| 108 |
-
| medium, subtitles | -- | 75.44 | 75.56 |
|
| 109 |
-
| long, subtitles | -- | 65.78 | 65.89 |
|
| 110 |
-
| **Video spatial** | | | |
|
| 111 |
-
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 112 |
-
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 113 |
-
| **Long video** | | | |
|
| 114 |
-
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 115 |
-
| MLVU dev | 73.83 | 75.3 | -- |
|
| 116 |
-
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 117 |
-
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 118 |
-
| **Temporal grounding (mIoU)** | | | |
|
| 119 |
-
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 120 |
-
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 121 |
-
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 122 |
|
| 123 |
\* TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
|
| 124 |
\* Video-MME-v2: subtitles are fed differently on the two sides.
|
| 125 |
\* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 126 |
\* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 127 |
\* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
|
|
|
|
|
|
| 128 |
\* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 129 |
|
| 130 |
### Evaluation setup
|
|
|
|
| 21 |
|
| 22 |
- **One model, two modes.** `apply_chat_template(..., streaming=True)` switches the default system prompt; eight protocol tokens carry the per-frame decision to stay silent, speak, or caption. Nothing else about the input changes between modes.
|
| 23 |
- **Two video front-ends.** Uniform frame sampling, or *codec mode*, which uses the encoder's bitcost to pack informative regions into a handful of canvases. On this checkpoint codec is ahead on **17 of the 21** video benchmarks where both were run.
|
| 24 |
+
- **Strong video and spatial results.** Against Qwen3.5-4B (thinking off), frame sampling leads on **17 of 21** comparable video benchmarks and codec on **16 of 18**, with the largest matched-setting margins on temporal grounding (TimeLens Charades +12.5) and spatial video (VSI-Bench +9.4).
|
| 25 |
- **Honest about images.** On still images the comparison is mixed — ahead on 18 of 31 comparable benchmarks, behind on OCR and document reading. The tables below are unfiltered.
|
| 26 |
|
| 27 |
## Model Overview
|
|
|
|
| 45 |
| **General perception and reading** | | |
|
| 46 |
| OCRBench v2 | 59 | 53.67 |
|
| 47 |
| OCRBench | 85.2 | 79.8 |
|
|
|
|
| 48 |
| PerceptionBench | 30.87 | 29.6 |
|
| 49 |
| MMStar | 66.87 | 63.93 |
|
| 50 |
| MMBench-EN dev | 88.84 | 88.22 |
|
| 51 |
| RealWorldQA | 74.38 | 73.86 |
|
| 52 |
+
| BLINK | 63.56 | 67.06 |
|
| 53 |
| V*-Bench | 81.68 | 89.53 |
|
| 54 |
| DocVQA val | 95.21 | 92.92 |
|
| 55 |
| ChartQA | 84.96 | 82.76 |
|
|
|
|
| 76 |
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 77 |
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 78 |
|
| 79 |
+
\* BLINK: 63.56 is Qwen's judged score. Its rule-based score was 54.97 — the extraction reads the first letter, and about 15% of its answers reason before giving the option.
|
| 80 |
\* MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
|
| 81 |
\* MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
|
| 82 |
\* DocVQA val: ANLS rule-based score. PerceptionBench: judged by gpt-oss-120b.
|
|
|
|
| 86 |
|
| 87 |
### Video
|
| 88 |
|
| 89 |
+
| Benchmark | Qwen3.5-4B | setting | OV2.5-4B (frames) | setting | OV2.5-4B (codec) | setting |
|
| 90 |
+
|---|---|---|---|---|---|---|
|
| 91 |
+
| **General video understanding** | | | | | | |
|
| 92 |
+
| MotionBench | 60.1 | 128f × 153,664 | 62.39 | 128f × 153,664 | 64.19 | TC128 × 313,600 |
|
| 93 |
+
| TOMATO | 34.91 | 16f × 153,664, fps=2 | 33.63 | model default | 38.61 | TC128 × 313,600 |
|
| 94 |
+
| MVBench | 64.03 | 64f × 153,664 | 72.03 | 64f × 153,664 | 75.5 | TC64 × 313,600 |
|
| 95 |
+
| NextQA MC test | 80.67 | 64f × 153,664 | 86.06 | 64f × 153,664 | 86.4 | TC64 × 313,600 |
|
| 96 |
+
| TempCompass MC | 73.35 | 64f × 153,664 | 75.76 | 64f × 153,664 | 79.68 | TC64 × 313,600 |
|
| 97 |
+
| MMVU val | 52.6 | 64f × 153,664 | 44.1 | 64f × 153,664 | -- | -- |
|
| 98 |
+
| Video-MME-v2 | 15.53 | 64f × 330,000 | 16.43 | 64f × 330,000 | 17.2 | TC64 × 313,600 |
|
| 99 |
+
| EgoSchema subset | 68.2 | 64f × 153,664 | 80 | 64f × 153,664 | -- | -- |
|
| 100 |
+
| Video-Index 512f | 19.64 | task frames, ≤262,144/f | 20 | task frames, ≤262,144/f | 19.76 | TC≤256 × 313,600 |
|
| 101 |
+
| MMOU test-mini | -- | -- | 45.96 | model default | -- | -- |
|
| 102 |
+
| **VideoMME** | | | | | | |
|
| 103 |
+
| short, no subtitles | 81.67 | 128f × 321,489 | 79.67 | 128f × 321,489 | 81.11 | TC256 × 313,600 |
|
| 104 |
+
| medium, no subtitles | 70.33 | 256f × 136,900 | 70.56 | 256f × 136,900 | 74.11 | TC512 × 313,600 |
|
| 105 |
+
| long, no subtitles | 63 | 1450f × 120,000 | 59 | 750f × 250,000 | 60.56 | TC512 × 313,600 |
|
| 106 |
+
| short, subtitles | -- | -- | 82.44 | 128f × 321,489 | 82.67 | TC256 × 313,600 |
|
| 107 |
+
| medium, subtitles | -- | -- | 75.44 | 256f × 136,900 | 75.56 | TC256 × 313,600 |
|
| 108 |
+
| long, subtitles | -- | -- | 65.78 | 750f × 250,000 | 65.89 | TC256 × 313,600 |
|
| 109 |
+
| **Video spatial** | | | | | | |
|
| 110 |
+
| VSI-Bench | 59.22 | 128f × 153,664 | 68.64 | 128f × 153,664 | 67.72 | TC128 × 313,600 |
|
| 111 |
+
| ReVSI 64f | 47.1 | 64f × 153,664 | 55.94 | 64f × 153,664 | 55.54 | TC64 × 313,600 |
|
| 112 |
+
| **Long video** | | | | | | |
|
| 113 |
+
| LVBench | 56.49 | 2048f × 120,000 | 62.04 | 950f × 200,000 | 61.59 | TC1024 × 156,800 |
|
| 114 |
+
| MLVU dev | 73.83 | 512f × 72,900 | 75.3 | 512f × 72,900 | -- | -- |
|
| 115 |
+
| LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
|
| 116 |
+
| VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
|
| 117 |
+
| **Temporal grounding (mIoU)** | | | | | | |
|
| 118 |
+
| TimeLens ActivityNet | 46.83 | 128f × 153,664 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
|
| 119 |
+
| TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
|
| 120 |
+
| TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
|
| 121 |
|
| 122 |
\* TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
|
| 123 |
\* Video-MME-v2: subtitles are fed differently on the two sides.
|
| 124 |
\* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 125 |
\* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 126 |
\* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
| 127 |
+
\* Settings are the ones that produced each reported score. Most rows are matched, but three are not: on **TimeLens ActivityNet** we ran 2048 frames × 409,600 px against Qwen's 128 × 153,664, so that +12.01 is not like-for-like (TimeLens Charades, at identical settings, is +12.53). On **TOMATO** Qwen ran 16 frames at fps=2 while we used the model defaults, and on **MMOU** only we have a score.
|
| 128 |
+
\* Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
|
| 129 |
\* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 130 |
|
| 131 |
### Evaluation setup
|
README_CN.md
CHANGED
|
@@ -21,7 +21,7 @@ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图
|
|
| 21 |
|
| 22 |
- **一个模型,两种模式。** `apply_chat_template(..., streaming=True)` 切换默认 system prompt;八个协议 token 承载每一帧"沉默 / 发言 / caption"的决策。除此之外输入格式在两种模式下完全相同。
|
| 23 |
- **两条视频前端。** 均匀抽帧,或 **codec 模式**——用编码器的 bitcost 把信息量大的区域打包进少量画布。在本 checkpoint 上,两边都跑出结果的 21 个视频 bench 中,codec **赢 17 个**。
|
| 24 |
-
- **视频与空间能力突出。** 对 Qwen3.5-4B(关闭 thinking),抽帧在 21 个可比视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**;优势最大的是时间定位(TimeLens +12)和视频空间(VSI-Bench +9.4)。
|
| 25 |
- **图像侧如实呈现。** 静态图像上互有胜负——31 个可比项中领先 18 个,OCR 与文档阅读落后。下面的表格未作筛选。
|
| 26 |
|
| 27 |
## 模型概览
|
|
@@ -45,12 +45,11 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
|
|
| 45 |
| **通用感知与阅读** | | |
|
| 46 |
| OCRBench v2 | 59 | 53.67 |
|
| 47 |
| OCRBench | 85.2 | 79.8 |
|
| 48 |
-
| OmniDocBench | 41.97 | -- |
|
| 49 |
| PerceptionBench | 30.87 | 29.6 |
|
| 50 |
| MMStar | 66.87 | 63.93 |
|
| 51 |
| MMBench-EN dev | 88.84 | 88.22 |
|
| 52 |
| RealWorldQA | 74.38 | 73.86 |
|
| 53 |
-
| BLINK |
|
| 54 |
| V*-Bench | 81.68 | 89.53 |
|
| 55 |
| DocVQA val | 95.21 | 92.92 |
|
| 56 |
| ChartQA | 84.96 | 82.76 |
|
|
@@ -77,7 +76,7 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
|
|
| 77 |
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 78 |
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 79 |
|
| 80 |
-
\* BLINK:规则分取回答首字母
|
| 81 |
\* MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
|
| 82 |
\* MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
|
| 83 |
\* DocVQA val:按 ANLS 规则分。PerceptionBench:由 gpt-oss-120b 判分。
|
|
@@ -87,44 +86,46 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
|
|
| 87 |
|
| 88 |
### 视频
|
| 89 |
|
| 90 |
-
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 91 |
-
|---|---|---|---|
|
| 92 |
-
| **通用视频理解** | | | |
|
| 93 |
-
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 94 |
-
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 95 |
-
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 96 |
-
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 97 |
-
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 98 |
-
| MMVU val | 52.6 | 44.1 | -- |
|
| 99 |
-
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 100 |
-
| EgoSchema subset | 68.2 | 80 | -- |
|
| 101 |
-
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 102 |
-
| MMOU test-mini | -- | 45.96 | -- |
|
| 103 |
-
| **VideoMME** | | | |
|
| 104 |
-
| short,无字幕 | 81.67 | 79.67 | 81.11 |
|
| 105 |
-
| medium,无字幕 | 70.33 | 70.56 | 74.11 |
|
| 106 |
-
| long,无字幕 | 63 | 59 | 60.56 |
|
| 107 |
-
| short,有字幕 | -- | 82.44 | 82.67 |
|
| 108 |
-
| medium,有字幕 | -- | 75.44 | 75.56 |
|
| 109 |
-
| long,有字幕 | -- | 65.78 | 65.89 |
|
| 110 |
-
| **视频空间** | | | |
|
| 111 |
-
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 112 |
-
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 113 |
-
| **长视频** | | | |
|
| 114 |
-
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 115 |
-
| MLVU dev | 73.83 | 75.3 | -- |
|
| 116 |
-
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 117 |
-
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 118 |
-
| **时间定位(mIoU)** | | | |
|
| 119 |
-
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 120 |
-
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 121 |
-
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 122 |
|
| 123 |
\* TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
|
| 124 |
\* Video-MME-v2:字幕送法两边不同。
|
| 125 |
\* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 126 |
\* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 127 |
\* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
|
|
|
|
|
|
| 128 |
\* codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 129 |
|
| 130 |
### 评测口径
|
|
|
|
| 21 |
|
| 22 |
- **一个模型,两种模式。** `apply_chat_template(..., streaming=True)` 切换默认 system prompt;八个协议 token 承载每一帧"沉默 / 发言 / caption"的决策。除此之外输入格式在两种模式下完全相同。
|
| 23 |
- **两条视频前端。** 均匀抽帧,或 **codec 模式**——用编码器的 bitcost 把信息量大的区域打包进少量画布。在本 checkpoint 上,两边都跑出结果的 21 个视频 bench 中,codec **赢 17 个**。
|
| 24 |
+
- **视频与空间能力突出。** 对 Qwen3.5-4B(关闭 thinking),抽帧在 21 个可比视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**;在设置对齐的项上优势最大的是时间定位(TimeLens Charades +12.5)和视频空间(VSI-Bench +9.4)。
|
| 25 |
- **图像侧如实呈现。** 静态图像上互有胜负——31 个可比项中领先 18 个,OCR 与文档阅读落后。下面的表格未作筛选。
|
| 26 |
|
| 27 |
## 模型概览
|
|
|
|
| 45 |
| **通用感知与阅读** | | |
|
| 46 |
| OCRBench v2 | 59 | 53.67 |
|
| 47 |
| OCRBench | 85.2 | 79.8 |
|
|
|
|
| 48 |
| PerceptionBench | 30.87 | 29.6 |
|
| 49 |
| MMStar | 66.87 | 63.93 |
|
| 50 |
| MMBench-EN dev | 88.84 | 88.22 |
|
| 51 |
| RealWorldQA | 74.38 | 73.86 |
|
| 52 |
+
| BLINK | 63.56 | 67.06 |
|
| 53 |
| V*-Bench | 81.68 | 89.53 |
|
| 54 |
| DocVQA val | 95.21 | 92.92 |
|
| 55 |
| ChartQA | 84.96 | 82.76 |
|
|
|
|
| 76 |
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 77 |
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 78 |
|
| 79 |
+
\* BLINK:63.56 是 Qwen 的 judge 分。其规则分为 54.97——抽取取回答首字母,而它约 15% 的回答先分析后给选项。
|
| 80 |
\* MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
|
| 81 |
\* MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
|
| 82 |
\* DocVQA val:按 ANLS 规则分。PerceptionBench:由 gpt-oss-120b 判分。
|
|
|
|
| 86 |
|
| 87 |
### 视频
|
| 88 |
|
| 89 |
+
| Benchmark | Qwen3.5-4B | 设置 | OV2.5-4B(抽帧) | 设置 | OV2.5-4B(codec) | 设置 |
|
| 90 |
+
|---|---|---|---|---|---|---|
|
| 91 |
+
| **通用视频理解** | | | | | | |
|
| 92 |
+
| MotionBench | 60.1 | 128f × 153,664 | 62.39 | 128f × 153,664 | 64.19 | TC128 × 313,600 |
|
| 93 |
+
| TOMATO | 34.91 | 16f × 153,664, fps=2 | 33.63 | model default | 38.61 | TC128 × 313,600 |
|
| 94 |
+
| MVBench | 64.03 | 64f × 153,664 | 72.03 | 64f × 153,664 | 75.5 | TC64 × 313,600 |
|
| 95 |
+
| NextQA MC test | 80.67 | 64f × 153,664 | 86.06 | 64f × 153,664 | 86.4 | TC64 × 313,600 |
|
| 96 |
+
| TempCompass MC | 73.35 | 64f × 153,664 | 75.76 | 64f × 153,664 | 79.68 | TC64 × 313,600 |
|
| 97 |
+
| MMVU val | 52.6 | 64f × 153,664 | 44.1 | 64f × 153,664 | -- | -- |
|
| 98 |
+
| Video-MME-v2 | 15.53 | 64f × 330,000 | 16.43 | 64f × 330,000 | 17.2 | TC64 × 313,600 |
|
| 99 |
+
| EgoSchema subset | 68.2 | 64f × 153,664 | 80 | 64f × 153,664 | -- | -- |
|
| 100 |
+
| Video-Index 512f | 19.64 | task frames, ≤262,144/f | 20 | task frames, ≤262,144/f | 19.76 | TC≤256 × 313,600 |
|
| 101 |
+
| MMOU test-mini | -- | -- | 45.96 | model default | -- | -- |
|
| 102 |
+
| **VideoMME** | | | | | | |
|
| 103 |
+
| short,无字幕 | 81.67 | 128f × 321,489 | 79.67 | 128f × 321,489 | 81.11 | TC256 × 313,600 |
|
| 104 |
+
| medium,无字幕 | 70.33 | 256f × 136,900 | 70.56 | 256f × 136,900 | 74.11 | TC512 × 313,600 |
|
| 105 |
+
| long,无字幕 | 63 | 1450f × 120,000 | 59 | 750f × 250,000 | 60.56 | TC512 × 313,600 |
|
| 106 |
+
| short,有字幕 | -- | -- | 82.44 | 128f × 321,489 | 82.67 | TC256 × 313,600 |
|
| 107 |
+
| medium,有字幕 | -- | -- | 75.44 | 256f × 136,900 | 75.56 | TC256 × 313,600 |
|
| 108 |
+
| long,有字幕 | -- | -- | 65.78 | 750f × 250,000 | 65.89 | TC256 × 313,600 |
|
| 109 |
+
| **视频空间** | | | | | | |
|
| 110 |
+
| VSI-Bench | 59.22 | 128f × 153,664 | 68.64 | 128f × 153,664 | 67.72 | TC128 × 313,600 |
|
| 111 |
+
| ReVSI 64f | 47.1 | 64f × 153,664 | 55.94 | 64f × 153,664 | 55.54 | TC64 × 313,600 |
|
| 112 |
+
| **长视频** | | | | | | |
|
| 113 |
+
| LVBench | 56.49 | 2048f × 120,000 | 62.04 | 950f × 200,000 | 61.59 | TC1024 × 156,800 |
|
| 114 |
+
| MLVU dev | 73.83 | 512f × 72,900 | 75.3 | 512f × 72,900 | -- | -- |
|
| 115 |
+
| LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
|
| 116 |
+
| VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
|
| 117 |
+
| **时间定位(mIoU)** | | | | | | |
|
| 118 |
+
| TimeLens ActivityNet | 46.83 | 128f × 153,664 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
|
| 119 |
+
| TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
|
| 120 |
+
| TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
|
| 121 |
|
| 122 |
\* TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
|
| 123 |
\* Video-MME-v2:字幕送法两边不同。
|
| 124 |
\* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 125 |
\* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 126 |
\* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
| 127 |
+
\* 设置列是各自取得该分数时的实际设置。多数行两边一致,但有三行不是:**TimeLens ActivityNet** 我方跑的是 2048 帧 × 409,600 px,Qwen 是 128 × 153,664,因此那 +12.01 并非同条件对比(设置完全相同的 TimeLens Charades 为 +12.53);**TOMATO** Qwen 为 16 帧 fps=2 而我方用模型默认值;**MMOU** 只有我方有分。
|
| 128 |
+
\* 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
|
| 129 |
\* codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 130 |
|
| 131 |
### 评测口径
|