Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
Restructure model cards along the Qwen3.5 card layout
Browse files- README.md +148 -182
- README_CN.md +146 -180
README.md
CHANGED
|
@@ -13,11 +13,133 @@ tags:
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
-
|
| 17 |
|
| 18 |
-
|
| 19 |
|
| 20 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
```bash
|
| 23 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
@@ -25,7 +147,7 @@ pip install flash-attn --no-build-isolation # optional, recommended
|
|
| 25 |
pip install decord # frame-sampling video path only
|
| 26 |
```
|
| 27 |
|
| 28 |
-
## Images
|
| 29 |
|
| 30 |
```python
|
| 31 |
import torch
|
|
@@ -54,11 +176,9 @@ with torch.inference_mode():
|
|
| 54 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 55 |
```
|
| 56 |
|
| 57 |
-
Images go through `smart_resize`, keeping aspect ratio and aligning to the patch grid
|
| 58 |
-
|
| 59 |
-
For several images, put one `{"type": "image"}` per image in `content` and pass `images=` in the same order. Each placeholder expands to `<|vision_start|><|image_pad|><|vision_end|>`, matched by position.
|
| 60 |
|
| 61 |
-
## Video — frame sampling
|
| 62 |
|
| 63 |
```python
|
| 64 |
messages = [{"role": "user", "content": [
|
|
@@ -77,58 +197,34 @@ inputs = processor(
|
|
| 77 |
|
| 78 |
`target_fps` and `num_frames` are also accepted; precedence is in `choose_target_frames` in `video_processing_llava_onevision2_5.py`. `max_frames` caps at 768.
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
| Case | Frames | Pixels per frame |
|
| 83 |
-
|---|---|---|
|
| 84 |
-
| Short video, detail questions | 64 – 128 | 153664 – 321489 |
|
| 85 |
-
| Medium | 256 | 136900 |
|
| 86 |
-
| Long video, global understanding | 640 – 768 | 72900 – 108900 |
|
| 87 |
-
|
| 88 |
-
All frames are encoded in one vision-tower forward pass, which can exhaust memory on very long inputs. `OV_VISION_CHUNK_PATCHES=200000` encodes them in groups instead, split along frame boundaries — attention never crosses a frame, so the result is unchanged.
|
| 89 |
-
|
| 90 |
-
**Timestamps.** Video input always carries them. The `<|video_pad|>` placeholder expands into one block per frame, each prefixed with its time, taken from the decoder's real sample positions:
|
| 91 |
|
| 92 |
```
|
| 93 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 94 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 95 |
```
|
| 96 |
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
```python
|
| 100 |
-
videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
|
| 101 |
-
```
|
| 102 |
|
| 103 |
-
Frames
|
| 104 |
-
|
| 105 |
-
## Streaming
|
| 106 |
-
|
| 107 |
-
Frames arrive one per turn, and each turn the model emits a state token: stay silent, or speak. One keyword switches the default system prompt, and changes nothing else:
|
| 108 |
|
| 109 |
| Mode | Call | Default system |
|
| 110 |
|---|---|---|
|
| 111 |
| Offline (default) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 112 |
| Streaming | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 113 |
|
| 114 |
-
An explicit `system` message wins in both modes. Offline is the default so existing evaluation harnesses work unchanged.
|
| 115 |
-
|
| 116 |
-
### Protocol tokens
|
| 117 |
-
|
| 118 |
| Token | Id | Meaning |
|
| 119 |
|---|---|---|
|
| 120 |
| `<silence>` | 248077 | this turn: say nothing |
|
| 121 |
| `<response>` | 248078 | this turn: speak |
|
| 122 |
-
| `<question_memory>` / `</question_memory>` | 248079 / 248080 | a new question, written once
|
| 123 |
| `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | an earlier answer, re-injected once it is due |
|
| 124 |
| `<captioning>` | 248083 | this turn: speak, as event-level captioning |
|
| 125 |
| `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
|
| 126 |
|
| 127 |
-
All eight are non-special added tokens, so `skip_special_tokens=True` still returns them — the state token is the output you read.
|
| 128 |
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
Strict user/assistant alternation, at most one frame per user turn; a turn with no new frame uses a bare text time prefix instead. Give each frame its `timestamp`, and the trailing newline as its own text item:
|
| 132 |
|
| 133 |
```python
|
| 134 |
messages = [{"role": "user", "content": [
|
|
@@ -140,8 +236,6 @@ text = processor.apply_chat_template(
|
|
| 140 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 141 |
```
|
| 142 |
|
| 143 |
-
which renders as:
|
| 144 |
-
|
| 145 |
```
|
| 146 |
<|im_start|>user
|
| 147 |
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
|
@@ -153,22 +247,11 @@ which renders as:
|
|
| 153 |
|
| 154 |
A plain string `content` is emitted verbatim, for harnesses that render the markers themselves.
|
| 155 |
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
The template fixes the format; two numbers come from the video and cannot be recovered from a bare frame:
|
| 159 |
|
| 160 |
-
|
| 161 |
-
2. **The `t` column of `patch_positions`** — handed independent images, the processor returns the right block layout but `t=0` for every frame. Rewrite each frame's rows to `floor(seconds × fps)`, or every frame reads as the start of a new visual segment.
|
| 162 |
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
The automatic timestamp expansion described above belongs to the offline `<|video_pad|>` branch. Streaming feeds frames one at a time and does not use it.
|
| 166 |
-
|
| 167 |
-
Not included here: the real-time loop around the model — frame scheduling, deciding when to stop generating, re-injecting question and answer memory as it falls due, and carrying context across decision points.
|
| 168 |
-
|
| 169 |
-
## Video — codec mode
|
| 170 |
-
|
| 171 |
-
Instead of sampling whole frames, codec mode uses encoder bitcost to find informative regions and packs them into a few *canvases*. Patches on one canvas may come from different source frames, each carrying its own `(source frame, row, column)` for 3-D RoPE — so canvases must not be fed through `images=`.
|
| 172 |
|
| 173 |
Preprocessing comes from `codec-video-prep` (bundles patched FFmpeg and dav1d; Linux x86_64, CPython 3.10/3.11, `numpy<2`, system `ffprobe`; H.264/HEVC/VP9/AV1 only; offline files only, since budgeting needs the whole video):
|
| 174 |
|
|
@@ -183,141 +266,24 @@ inputs = processor(text=text, videos=["example.mp4"],
|
|
| 183 |
video_backend="codec", return_tensors="pt").to(model.device)
|
| 184 |
```
|
| 185 |
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
Each canvas gets one time marker: a point `<6.5 seconds>` when all its patches come from one source frame, a range `<0.0 to 1.8 seconds>` when it spans several (schema `canvas_patch_span_point_range_v1`). Temporal questions work as on the frame path.
|
| 189 |
-
|
| 190 |
-
Defaults live in the `codec` field of `preprocessor_config.json` — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — and are overridden per call with `codec_config={"target_canvas": 32, ...}`. Other keys: `strategy` (`adaptive_pwindow`, `adaptive_rdo`, `readiness`), `num_sampled_frames`, `max_pixels`, `canvas_format`. `patch=16` and `block_size=3` must match the model's `patch_size` and `spatial_merge_size`; preprocessing raises rather than failing silently. Canvases cache in `~/.cache/codec_video_prep/canvases` (`CVP_CODEC_CACHE_ROOT` to relocate).
|
| 191 |
-
|
| 192 |
-
## Notes
|
| 193 |
-
|
| 194 |
-
**Batching.** Pass a list of texts and the matching images with `padding=True`. Padding is left-sided, so slice each output by its own `input_ids` length.
|
| 195 |
-
|
| 196 |
-
**Offline.** Set `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`. Load the full model once in a single process before starting more: remote-code modules compile into `$HF_HOME/modules/transformers_modules/<name>/`, and concurrent first loads race there. That cache is keyed by leaf directory name only, so give local copies distinct names.
|
| 197 |
-
|
| 198 |
-
**Generation.** `generation_config.json` sets only `eos_token_id=248044` and `use_cache=true`. Use `do_sample=False` for deterministic output, and a generous `max_new_tokens` on tasks where the model reasons before answering — 128 cuts it off mid-reasoning.
|
| 199 |
-
|
| 200 |
-
**Answer format.** On multiple-choice tasks the model may write a sentence rather than a bare letter. If a regex will parse the output, ask for a letter: `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range in the prompt, e.g. integers 0–1000. The eight streaming tokens survive `skip_special_tokens`; strip them explicitly if you do not want them offline.
|
| 201 |
-
|
| 202 |
-
## Benchmarks
|
| 203 |
-
|
| 204 |
-
OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled**. Images are scored once: the codec path applies to video only. Empty cells (`--`) mean the result is not yet available or not applicable.
|
| 205 |
-
|
| 206 |
-
### Image
|
| 207 |
-
|
| 208 |
-
| Benchmark | Qwen3.5-4B | OV2.5-4B |
|
| 209 |
-
|---|---|---|
|
| 210 |
-
| **General perception and reading** | | |
|
| 211 |
-
| OCRBench v2 | 59 | 53.67 |
|
| 212 |
-
| OCRBench | 85.2 | 79.8 |
|
| 213 |
-
| OmniDocBench | 41.97 | -- |
|
| 214 |
-
| PerceptionBench | 30.87 | 29.6 |
|
| 215 |
-
| MMStar | 66.87 | 63.93 |
|
| 216 |
-
| MMBench-EN dev | 88.84 | 88.22 |
|
| 217 |
-
| RealWorldQA | 74.38 | 73.86 |
|
| 218 |
-
| BLINK | 54.97 | 67.06 |
|
| 219 |
-
| V*-Bench | 81.68 | 89.53 |
|
| 220 |
-
| DocVQA val | 95.21 | 92.92 |
|
| 221 |
-
| ChartQA | 84.96 | 82.76 |
|
| 222 |
-
| AI2D | 82.35 | 82.16 |
|
| 223 |
-
| CountBench | 87.17 | 84.93 |
|
| 224 |
-
| PixMo-Count | 63.11 | 69.85 |
|
| 225 |
-
| VLMs are Blind | 80.43 | 77.23 |
|
| 226 |
-
| VLMs are Biased | 24.96 | 25.29 |
|
| 227 |
-
| MuirBench | 21.27 | 48.69 |
|
| 228 |
-
| **Grounding and spatial** | | |
|
| 229 |
-
| RefSpatial-Bench | 29.14 | 62.09 |
|
| 230 |
-
| RefCOCO val (Acc@0.5) | 89.07 | 92.64 |
|
| 231 |
-
| RefCOCOg val (Acc@0.5) | 87.65 | 88.91 |
|
| 232 |
-
| CrossPoint | 33.7 | 80.5 |
|
| 233 |
-
| SAT mcq circular | 68.67 | 74 |
|
| 234 |
-
| CV-Bench 2D | 80.74 | 83.31 |
|
| 235 |
-
| CV-Bench 3D | 92 | 94.5 |
|
| 236 |
-
| EmbSpatial | 74.7 | 83.02 |
|
| 237 |
-
| CRPE relation | 77.8 | 78.22 |
|
| 238 |
-
| ERQA | 43.5 | 46.5 |
|
| 239 |
-
| MMSI-Bench | 31.4 | 33.3 |
|
| 240 |
-
| MetaVQA | 61.75 | 61.03 |
|
| 241 |
-
| PointBench | 17.09 | 29.68 |
|
| 242 |
-
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 243 |
-
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 244 |
-
|
| 245 |
-
\* BLINK: the rule-based score reads the first letter of the answer. About 15% of Qwen's answers reason first and then give the option, and are scored wrong; recomputed from the final option Qwen is around 62.
|
| 246 |
-
\* MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
|
| 247 |
-
\* MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
|
| 248 |
-
\* DocVQA val: ANLS rule-based score.
|
| 249 |
-
\* PerceptionBench: judged by gpt-oss-120b.
|
| 250 |
-
\* PointBench: both sides emit 0–1000 coordinates while scoring is mostly in absolute pixels, so both numbers are low.
|
| 251 |
-
\* RefSpatial-Bench: Qwen re-run greedy on 2026-11-17 under the current scoring script.
|
| 252 |
-
\* SAT free-form and SAT mcq are omitted — all 150 correct options sit in first position, so answering "A" every time scores 100%. The row reported is the circular variant, which has a cyclic-consistency check.
|
| 253 |
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 257 |
-
|---|---|---|---|
|
| 258 |
-
| **General video understanding** | | | |
|
| 259 |
-
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 260 |
-
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 261 |
-
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 262 |
-
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 263 |
-
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 264 |
-
| MMVU val | 52.6 | 44.1 | -- |
|
| 265 |
-
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 266 |
-
| EgoSchema subset | 68.2 | 80 | -- |
|
| 267 |
-
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 268 |
-
| MMOU test-mini | -- | 45.96 | -- |
|
| 269 |
-
| **VideoMME** | | | |
|
| 270 |
-
| short, no subtitles | 81.67 | 79.67 | 81.11 |
|
| 271 |
-
| medium, no subtitles | 70.33 | 70.56 | 74.11 |
|
| 272 |
-
| long, no subtitles | 63 | 59 | 60.56 |
|
| 273 |
-
| short, subtitles | -- | 82.44 | 82.67 |
|
| 274 |
-
| medium, subtitles | -- | 75.44 | 75.56 |
|
| 275 |
-
| long, subtitles | -- | 65.78 | 65.89 |
|
| 276 |
-
| **Video spatial** | | | |
|
| 277 |
-
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 278 |
-
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 279 |
-
| **Long video** | | | |
|
| 280 |
-
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 281 |
-
| MLVU dev | 73.83 | 75.3 | -- |
|
| 282 |
-
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 283 |
-
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 284 |
-
| **Temporal grounding (mIoU)** | | | |
|
| 285 |
-
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 286 |
-
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 287 |
-
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 288 |
|
| 289 |
-
|
| 290 |
-
\* Video-MME-v2: subtitles are fed differently on the two sides.
|
| 291 |
-
\* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 292 |
-
\* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 293 |
-
\* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
| 294 |
-
\* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 295 |
|
| 296 |
-
|
| 297 |
|
| 298 |
-
|
| 299 |
|
| 300 |
-
**
|
| 301 |
|
| 302 |
-
|
| 303 |
|
| 304 |
-
**
|
| 305 |
|
| 306 |
-
|
| 307 |
-
- *rule, official metric* — computed directly by the benchmark's own metric.
|
| 308 |
-
- *rule, judge pending* — the judge should apply but has not run on at least one side, so the rule-based score stands in: MuirBench, EmbSpatial, CRPE relation, ERQA, MetaVQA, NextQA MC test, TempCompass MC, EgoSchema subset, Video-Index 512f.
|
| 309 |
|
| 310 |
-
|
| 311 |
|
| 312 |
-
|
| 313 |
-
|
| 314 |
-
| Item | Value |
|
| 315 |
-
|---|---|
|
| 316 |
-
| Language layers | 32 (linear attention, full attention every 4) |
|
| 317 |
-
| hidden_size | 2560 |
|
| 318 |
-
| Attention heads / KV heads | 16 / 4 |
|
| 319 |
-
| Vocabulary | 248320 |
|
| 320 |
-
| Vision layers / hidden_size | 24 / 1024 |
|
| 321 |
-
| patch_size / spatial_merge_size | 16 / 3 |
|
| 322 |
-
| max_position_embeddings | 262144 |
|
| 323 |
-
| Weights | bfloat16, 3 shards, about 9.9 GB |
|
|
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
+
> This repository contains the instruction-tuned weights, the remote modeling code and both chat-template modes.
|
| 17 |
|
| 18 |
+
LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image, multi-image and video input. Alongside the usual offline tasks it is trained for **streaming** video, where frames arrive one at a time and the model decides at every step whether to stay silent or speak — the same weights and the same chat template serve both, switched by one keyword.
|
| 19 |
|
| 20 |
+
## Highlights
|
| 21 |
+
|
| 22 |
+
- **One model, two modes.** `apply_chat_template(..., streaming=True)` switches the default system prompt; eight protocol tokens carry the per-frame decision to stay silent, speak, or caption. Nothing else about the input changes between modes.
|
| 23 |
+
- **Two video front-ends.** Uniform frame sampling, or *codec mode*, which uses the encoder's bitcost to pack informative regions into a handful of canvases. On this checkpoint codec is ahead on **17 of the 21** video benchmarks where both were run.
|
| 24 |
+
- **Strong video and spatial results.** Against Qwen3.5-4B (thinking off), frame sampling leads on **17 of 21** comparable video benchmarks and codec on **16 of 18**, with the largest margins on temporal grounding (TimeLens +12) and spatial video (VSI-Bench +9.4).
|
| 25 |
+
- **Honest about images.** On still images the comparison is mixed — ahead on 18 of 31 comparable benchmarks, behind on OCR and document reading. The tables below are unfiltered.
|
| 26 |
+
|
| 27 |
+
## Model Overview
|
| 28 |
+
|
| 29 |
+
- **Type:** instruction-tuned vision-language model, LLaVA-OneVision-2 architecture
|
| 30 |
+
- **Parameters:** 5.25B total (4B language model plus vision tower and adapter)
|
| 31 |
+
- **Language side:** 32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4
|
| 32 |
+
- **Vision side:** 24 layers, hidden size 1024, `patch_size=16`, `spatial_merge_size=3`, attention within a frame
|
| 33 |
+
- **Vocabulary:** 248320, including the 8 streaming protocol tokens
|
| 34 |
+
- **Context length:** 262144
|
| 35 |
+
- **Weights:** bfloat16, 3 safetensors shards, about 9.9 GB
|
| 36 |
+
|
| 37 |
+
## Benchmark Results
|
| 38 |
+
|
| 39 |
+
OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled**. Images are scored once: the codec path applies to video only. Empty cells (`--`) mean the result is not yet available or not applicable.
|
| 40 |
+
|
| 41 |
+
### Image
|
| 42 |
+
|
| 43 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B |
|
| 44 |
+
|---|---|---|
|
| 45 |
+
| **General perception and reading** | | |
|
| 46 |
+
| OCRBench v2 | 59 | 53.67 |
|
| 47 |
+
| OCRBench | 85.2 | 79.8 |
|
| 48 |
+
| OmniDocBench | 41.97 | -- |
|
| 49 |
+
| PerceptionBench | 30.87 | 29.6 |
|
| 50 |
+
| MMStar | 66.87 | 63.93 |
|
| 51 |
+
| MMBench-EN dev | 88.84 | 88.22 |
|
| 52 |
+
| RealWorldQA | 74.38 | 73.86 |
|
| 53 |
+
| BLINK | 54.97 | 67.06 |
|
| 54 |
+
| V*-Bench | 81.68 | 89.53 |
|
| 55 |
+
| DocVQA val | 95.21 | 92.92 |
|
| 56 |
+
| ChartQA | 84.96 | 82.76 |
|
| 57 |
+
| AI2D | 82.35 | 82.16 |
|
| 58 |
+
| CountBench | 87.17 | 84.93 |
|
| 59 |
+
| PixMo-Count | 63.11 | 69.85 |
|
| 60 |
+
| VLMs are Blind | 80.43 | 77.23 |
|
| 61 |
+
| VLMs are Biased | 24.96 | 25.29 |
|
| 62 |
+
| MuirBench | 21.27 | 48.69 |
|
| 63 |
+
| **Grounding and spatial** | | |
|
| 64 |
+
| RefSpatial-Bench | 29.14 | 62.09 |
|
| 65 |
+
| RefCOCO val (Acc@0.5) | 89.07 | 92.64 |
|
| 66 |
+
| RefCOCOg val (Acc@0.5) | 87.65 | 88.91 |
|
| 67 |
+
| CrossPoint | 33.7 | 80.5 |
|
| 68 |
+
| SAT mcq circular | 68.67 | 74 |
|
| 69 |
+
| CV-Bench 2D | 80.74 | 83.31 |
|
| 70 |
+
| CV-Bench 3D | 92 | 94.5 |
|
| 71 |
+
| EmbSpatial | 74.7 | 83.02 |
|
| 72 |
+
| CRPE relation | 77.8 | 78.22 |
|
| 73 |
+
| ERQA | 43.5 | 46.5 |
|
| 74 |
+
| MMSI-Bench | 31.4 | 33.3 |
|
| 75 |
+
| MetaVQA | 61.75 | 61.03 |
|
| 76 |
+
| PointBench | 17.09 | 29.68 |
|
| 77 |
+
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 78 |
+
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 79 |
+
|
| 80 |
+
\* BLINK: the rule-based score reads the first letter of the answer. About 15% of Qwen's answers reason first and then give the option, and are scored wrong; recomputed from the final option Qwen is around 62.
|
| 81 |
+
\* MuirBench: Qwen has only a rule-based score and is likely affected by the same first-letter parsing. The judge has not been run, so this row is not yet a fair comparison.
|
| 82 |
+
\* MMBench-EN dev: per-row accuracy over 4329 rows; the official protocol is circular evaluation.
|
| 83 |
+
\* DocVQA val: ANLS rule-based score. PerceptionBench: judged by gpt-oss-120b.
|
| 84 |
+
\* PointBench: both sides emit 0–1000 coordinates while scoring is mostly in absolute pixels, so both numbers are low.
|
| 85 |
+
\* RefSpatial-Bench: Qwen re-run greedy on 2026-11-17 under the current scoring script.
|
| 86 |
+
\* SAT free-form and SAT mcq are omitted — all 150 correct options sit in first position, so answering "A" every time scores 100%. The row reported is the circular variant, which has a cyclic-consistency check.
|
| 87 |
+
|
| 88 |
+
### Video
|
| 89 |
+
|
| 90 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 91 |
+
|---|---|---|---|
|
| 92 |
+
| **General video understanding** | | | |
|
| 93 |
+
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 94 |
+
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 95 |
+
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 96 |
+
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 97 |
+
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 98 |
+
| MMVU val | 52.6 | 44.1 | -- |
|
| 99 |
+
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 100 |
+
| EgoSchema subset | 68.2 | 80 | -- |
|
| 101 |
+
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 102 |
+
| MMOU test-mini | -- | 45.96 | -- |
|
| 103 |
+
| **VideoMME** | | | |
|
| 104 |
+
| short, no subtitles | 81.67 | 79.67 | 81.11 |
|
| 105 |
+
| medium, no subtitles | 70.33 | 70.56 | 74.11 |
|
| 106 |
+
| long, no subtitles | 63 | 59 | 60.56 |
|
| 107 |
+
| short, subtitles | -- | 82.44 | 82.67 |
|
| 108 |
+
| medium, subtitles | -- | 75.44 | 75.56 |
|
| 109 |
+
| long, subtitles | -- | 65.78 | 65.89 |
|
| 110 |
+
| **Video spatial** | | | |
|
| 111 |
+
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 112 |
+
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 113 |
+
| **Long video** | | | |
|
| 114 |
+
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 115 |
+
| MLVU dev | 73.83 | 75.3 | -- |
|
| 116 |
+
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 117 |
+
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 118 |
+
| **Temporal grounding (mIoU)** | | | |
|
| 119 |
+
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 120 |
+
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 121 |
+
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 122 |
+
|
| 123 |
+
\* TOMATO: Qwen ran at fps=2 / 16 frames; we passed no frame or pixel settings and used the model defaults.
|
| 124 |
+
\* Video-MME-v2: subtitles are fed differently on the two sides.
|
| 125 |
+
\* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
|
| 126 |
+
\* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
|
| 127 |
+
\* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
|
| 128 |
+
\* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
|
| 129 |
+
|
| 130 |
+
### Evaluation setup
|
| 131 |
+
|
| 132 |
+
Qwen3.5-4B runs with thinking disabled. Some of its results came from temperature-0.7 sampling rather than greedy decoding; rows since re-run greedy use the re-run. Every number is a single run, with no variance estimate.
|
| 133 |
+
|
| 134 |
+
- **Images.** OV2.5 uses the processor's default pixel budget; Qwen uses 65,536–16,777,216 pixels per image.
|
| 135 |
+
- **Video.** Frames are sampled at fps=1 and cached offline, so evaluation decodes nothing and repeated runs see identical input. For each task both sides report the best score over every setting that produced one, so frame count and per-frame pixel budget vary by row and by model — on LVBench, Qwen's best came out of 27 settings and ours out of 16. The codec column uses the canvas budgets from the `codec` field of `preprocessor_config.json`, adjusted per task.
|
| 136 |
+
- **Scoring.** *judge* — both sides re-checked by Qwen3.8-27B at temperature 0, which sees the question, options, reference answer and raw output and answers YES or NO; the reported score is the greater of the rule-based and judged scores. *rule, official metric* — the benchmark's own metric. *rule, judge pending* — the judge should apply but has not run on at least one side: MuirBench, EmbSpatial, CRPE relation, ERQA, MetaVQA, NextQA MC test, TempCompass MC, EgoSchema subset, Video-Index 512f.
|
| 137 |
+
|
| 138 |
+
All generation is greedy with `batch_size=1`.
|
| 139 |
+
|
| 140 |
+
## Quickstart
|
| 141 |
+
|
| 142 |
+
> The modeling code ships with the weights, so `trust_remote_code=True` is required everywhere.
|
| 143 |
|
| 144 |
```bash
|
| 145 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
|
|
| 147 |
pip install decord # frame-sampling video path only
|
| 148 |
```
|
| 149 |
|
| 150 |
+
### Images
|
| 151 |
|
| 152 |
```python
|
| 153 |
import torch
|
|
|
|
| 176 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 177 |
```
|
| 178 |
|
| 179 |
+
Images go through `smart_resize`, keeping aspect ratio and aligning to the patch grid; the budget defaults to `min_pixels=3136` / `max_pixels=4000000`. For several images, put one `{"type": "image"}` per image in `content` and pass `images=` in the same order.
|
|
|
|
|
|
|
| 180 |
|
| 181 |
+
### Video — frame sampling
|
| 182 |
|
| 183 |
```python
|
| 184 |
messages = [{"role": "user", "content": [
|
|
|
|
| 197 |
|
| 198 |
`target_fps` and `num_frames` are also accepted; precedence is in `choose_target_frames` in `video_processing_llava_onevision2_5.py`. `max_frames` caps at 768.
|
| 199 |
|
| 200 |
+
Timestamps are always injected: the `<|video_pad|>` placeholder expands into one block per frame, prefixed with that frame's real sample time, so temporal questions work directly.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
|
| 202 |
```
|
| 203 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 204 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 205 |
```
|
| 206 |
|
| 207 |
+
### Streaming
|
|
|
|
|
|
|
|
|
|
|
|
|
| 208 |
|
| 209 |
+
Frames arrive one per turn and each turn the model emits a state token. One keyword switches the default system prompt and changes nothing else; an explicit `system` message wins in both modes.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
|
| 211 |
| Mode | Call | Default system |
|
| 212 |
|---|---|---|
|
| 213 |
| Offline (default) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 214 |
| Streaming | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 215 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 216 |
| Token | Id | Meaning |
|
| 217 |
|---|---|---|
|
| 218 |
| `<silence>` | 248077 | this turn: say nothing |
|
| 219 |
| `<response>` | 248078 | this turn: speak |
|
| 220 |
+
| `<question_memory>` / `</question_memory>` | 248079 / 248080 | a new question, written once when it is asked |
|
| 221 |
| `<answer_memory>` / `</answer_memory>` | 248081 / 248082 | an earlier answer, re-injected once it is due |
|
| 222 |
| `<captioning>` | 248083 | this turn: speak, as event-level captioning |
|
| 223 |
| `<caption_begin>` | 248084 | user-side marker that this stream is a captioning task |
|
| 224 |
|
| 225 |
+
All eight are non-special added tokens, so `skip_special_tokens=True` still returns them — the state token is the output you read. The memory tokens wrap a JSON object, e.g. `{"timestep": "00:00:06", "content": "..."}`.
|
| 226 |
|
| 227 |
+
Strict user/assistant alternation, at most one frame per user turn; a turn with no new frame uses a bare text time prefix. Give each frame its `timestamp`, and the trailing newline as its own text item:
|
|
|
|
|
|
|
| 228 |
|
| 229 |
```python
|
| 230 |
messages = [{"role": "user", "content": [
|
|
|
|
| 236 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 237 |
```
|
| 238 |
|
|
|
|
|
|
|
| 239 |
```
|
| 240 |
<|im_start|>user
|
| 241 |
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
|
|
|
| 247 |
|
| 248 |
A plain string `content` is emitted verbatim, for harnesses that render the markers themselves.
|
| 249 |
|
| 250 |
+
> **Not in this repository:** the real-time loop around the model — frame scheduling, deciding when to stop generating, re-injecting question and answer memory as it falls due, and carrying context across decision points.
|
|
|
|
|
|
|
| 251 |
|
| 252 |
+
### Video — codec mode
|
|
|
|
| 253 |
|
| 254 |
+
Codec mode uses encoder bitcost to pack informative regions into a few *canvases*. Patches on one canvas may come from different source frames, each carrying its own `(source frame, row, column)` for 3-D RoPE.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 255 |
|
| 256 |
Preprocessing comes from `codec-video-prep` (bundles patched FFmpeg and dav1d; Linux x86_64, CPython 3.10/3.11, `numpy<2`, system `ffprobe`; H.264/HEVC/VP9/AV1 only; offline files only, since budgeting needs the whole video):
|
| 257 |
|
|
|
|
| 266 |
video_backend="codec", return_tensors="pt").to(model.device)
|
| 267 |
```
|
| 268 |
|
| 269 |
+
Each canvas gets one time marker: a point `<6.5 seconds>` when all its patches come from one source frame, a range `<0.0 to 1.8 seconds>` when it spans several (schema `canvas_patch_span_point_range_v1`).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 270 |
|
| 271 |
+
Defaults live in the `codec` field of `preprocessor_config.json` — 512 candidate frames, 64 canvases, patch 16, 3×3 blocks, 313600 pixels per canvas — overridden per call with `codec_config={"target_canvas": 32, ...}`. Other keys: `strategy` (`adaptive_pwindow`, `adaptive_rdo`, `readiness`), `num_sampled_frames`, `max_pixels`, `canvas_format`. Canvases cache in `~/.cache/codec_video_prep/canvases` (`CVP_CODEC_CACHE_ROOT` to relocate).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 272 |
|
| 273 |
+
## Best Practices
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 274 |
|
| 275 |
+
1. **Pass video paths, not pre-decoded frames.** Only a path lets the processor recover real sample times and source frame numbers. Given a bare frame list it falls back to frame ordinals as seconds, so a 100-second video looks N seconds long and temporal answers are wrong (it raises a `UserWarning`). If you must decode yourself, hand over the timing too: `videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`. Frames passed through `images=` get no timestamps at all.
|
| 276 |
|
| 277 |
+
2. **Keep `patch_positions` truthful.** In codec mode it comes back in the processor output and must be forwarded to `model.forward(...)` — do not drop it. In streaming, where each frame is handed over as an independent image, the layout is right but `t` is `0` for every frame; rewrite each frame's rows to `floor(seconds × fps)`, or every frame reads as the start of a new visual segment. Only the *rank* of `t` matters to the model, but real frame numbers keep the input auditable.
|
| 278 |
|
| 279 |
+
3. **Tune frames and per-frame pixels together.** Cost is roughly their product: 64–128 frames at 153664–321489 px for short video, 256 at 136900 for medium, 640–768 at 72900–108900 for long. For very long inputs set `OV_VISION_CHUNK_PATCHES=200000`, which encodes frames in groups split along frame boundaries — attention never crosses a frame, so the result is unchanged.
|
| 280 |
|
| 281 |
+
4. **Match codec geometry to the model.** `patch=16` and `block_size=3` must equal `patch_size` and `spatial_merge_size`; preprocessing raises rather than failing silently.
|
| 282 |
|
| 283 |
+
5. **Decode greedily, and give reasoning room.** `generation_config.json` sets only `eos_token_id=248044` and `use_cache=true`. Use `do_sample=False` for deterministic output and a generous `max_new_tokens` where the model reasons before answering — 128 cuts it off mid-reasoning. For batching, pass a list of texts with `padding=True`; padding is left-sided, so slice each output by its own `input_ids` length.
|
| 284 |
|
| 285 |
+
6. **Ask for the format you will parse.** On multiple-choice tasks the model may write a sentence rather than a bare letter; if a regex will read the output, request one: `Answer with the option's letter from the given choices directly.` For grounding, state the coordinate range in the prompt, e.g. integers 0–1000.
|
|
|
|
|
|
|
| 286 |
|
| 287 |
+
7. **Warm up remote code before going multi-process.** Modules compile into `$HF_HOME/modules/transformers_modules/<name>/` and concurrent first loads race there, so load the full model once in a single process first. That cache is keyed by leaf directory name only — give local copies distinct names.
|
| 288 |
|
| 289 |
+
8. **Strip the streaming tokens if you do not want them.** All eight survive `skip_special_tokens=True` by design.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README_CN.md
CHANGED
|
@@ -13,11 +13,133 @@ tags:
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
-
|
| 17 |
|
| 18 |
-
|
| 19 |
|
| 20 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
```bash
|
| 23 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
@@ -25,7 +147,7 @@ pip install flash-attn --no-build-isolation # 可选,建议装
|
|
| 25 |
pip install decord # 仅抽帧视频路径需要
|
| 26 |
```
|
| 27 |
|
| 28 |
-
## 图
|
| 29 |
|
| 30 |
```python
|
| 31 |
import torch
|
|
@@ -54,11 +176,9 @@ with torch.inference_mode():
|
|
| 54 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 55 |
```
|
| 56 |
|
| 57 |
-
图片走 `smart_resize`,保持长宽比并对齐 patch 网格
|
| 58 |
-
|
| 59 |
-
多图时每张图放一个 `{"type": "image"}`,`images=` 按同样顺序传入。每个占位符展开成 `<|vision_start|><|image_pad|><|vision_end|>`,按位置对应。
|
| 60 |
|
| 61 |
-
## 视频 — 抽帧
|
| 62 |
|
| 63 |
```python
|
| 64 |
messages = [{"role": "user", "content": [
|
|
@@ -77,44 +197,22 @@ inputs = processor(
|
|
| 77 |
|
| 78 |
也接受 `target_fps` 和 `num_frames`,优先级见 `video_processing_llava_onevision2_5.py` 的 `choose_target_frames`。`max_frames` 封顶 768。
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
| 场景 | 帧数 | 逐帧像素 |
|
| 83 |
-
|---|---|---|
|
| 84 |
-
| 短视频、细节提问 | 64 – 128 | 153664 – 321489 |
|
| 85 |
-
| 中等 | 256 | 136900 |
|
| 86 |
-
| 长视频、整体理解 | 640 – 768 | 72900 – 108900 |
|
| 87 |
-
|
| 88 |
-
所有帧是在视觉塔里一次前向编码的,超长输入可能打爆显存。设 `OV_VISION_CHUNK_PATCHES=200000` 可改为分组编码,按帧边界切分,注意力不跨帧,结果不变。
|
| 89 |
-
|
| 90 |
-
**时间戳**一定会注入。`<|video_pad|>` 占位符展开成逐帧的块,每块前面是该帧的时间,取自解码器的真实采样位置:
|
| 91 |
|
| 92 |
```
|
| 93 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 94 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 95 |
```
|
| 96 |
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
```python
|
| 100 |
-
videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_list}]
|
| 101 |
-
```
|
| 102 |
-
|
| 103 |
-
通过 `images=` 传帧则完全不注入时间戳。
|
| 104 |
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
画面每轮进来一帧,模型每轮输出一个状态 token:沉默,或发言。一个关键字切换默认 system,其余什么都不改:
|
| 108 |
|
| 109 |
| 模式 | 调用 | 默认 system |
|
| 110 |
|---|---|---|
|
| 111 |
| 离线(默认) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 112 |
| 流式 | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 113 |
|
| 114 |
-
显式传入的 `system` 在两种模式下都优先。默认走离线,是为了让现有离线评测不改代码也能用。
|
| 115 |
-
|
| 116 |
-
### 协议 token
|
| 117 |
-
|
| 118 |
| Token | Id | 含义 |
|
| 119 |
|---|---|---|
|
| 120 |
| `<silence>` | 248077 | 当前轮不发言 |
|
|
@@ -124,9 +222,7 @@ videos=[{"frames": frames, "timestamps": seconds_list, "frame_indices": index_li
|
|
| 124 |
| `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
|
| 125 |
| `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
|
| 126 |
|
| 127 |
-
八个都是非 special 的 added token,所以 `skip_special_tokens=True` 仍会解出来——状态 token 正是你要读的输出。
|
| 128 |
-
|
| 129 |
-
### 怎么构造一轮
|
| 130 |
|
| 131 |
严格 user / assistant 交替,每个 user 轮最多一帧;没有新画面的轮次改用纯文本时间前缀。每帧给上 `timestamp`,帧后的换行作为单独的 text 项:
|
| 132 |
|
|
@@ -140,8 +236,6 @@ text = processor.apply_chat_template(
|
|
| 140 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 141 |
```
|
| 142 |
|
| 143 |
-
渲染结果:
|
| 144 |
-
|
| 145 |
```
|
| 146 |
<|im_start|>user
|
| 147 |
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
|
@@ -153,22 +247,11 @@ text = processor.apply_chat_template(
|
|
| 153 |
|
| 154 |
content 传字符串则原样输出,适合自己已经拼好标记的 harness。
|
| 155 |
|
| 156 |
-
|
| 157 |
|
| 158 |
-
|
| 159 |
|
| 160 |
-
|
| 161 |
-
2. **`patch_positions` 的 `t` 列**——把每帧当独立 image 交给 processor 时,返回的块布局是对的,但 `t` 每帧都是 0。需要改成 `floor(秒数 × fps)`,否则每帧都会被当成一段新视觉的开头。
|
| 162 |
-
|
| 163 |
-
模型只用 `t` 在本段内的**名次**,视觉塔也只做帧内注意力;写真实帧号只是为了输入可审计。
|
| 164 |
-
|
| 165 |
-
上面说的逐帧时间戳自动展开属于离线的 `<|video_pad|>` 分支,流式逐帧输入用不到。
|
| 166 |
-
|
| 167 |
-
本仓库不包含围绕模型的实时循环:按时钟取帧、决定何时停止生成、到期回插 question / answer memory、在决策点之间维持上下文。
|
| 168 |
-
|
| 169 |
-
## 视频 — codec 模式
|
| 170 |
-
|
| 171 |
-
codec 模式不按整帧采样,而是用编码器 bitcost 找出信息量大的区域,打包进少量**画布**。同一张画布上的 patch 可以来自不同源帧,各自带 `(源帧, 行, 列)` 供 3-D RoPE 使用——所以画布不能通过 `images=` 传。
|
| 172 |
|
| 173 |
预处理来自 `codec-video-prep`(自带打过补丁的 FFmpeg 和 dav1d;限 Linux x86_64、CPython 3.10/3.11、`numpy<2`、系统 `ffprobe`;只支持 H.264/HEVC/VP9/AV1;只适用于离线文件,因为预算计算需要整段视频):
|
| 174 |
|
|
@@ -183,141 +266,24 @@ inputs = processor(text=text, videos=["example.mp4"],
|
|
| 183 |
video_backend="codec", return_tensors="pt").to(model.device)
|
| 184 |
```
|
| 185 |
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
每张画布只有一个时间标记:所有 patch 来自同一源帧时是点 `<6.5 seconds>`,跨多帧时是区间 `<0.0 to 1.8 seconds>`(schema `canvas_patch_span_point_range_v1`)。时间类问题与抽帧路径表现一致。
|
| 189 |
-
|
| 190 |
-
默认配方写在 `preprocessor_config.json` 的 `codec` 字段——512 候选帧、64 画布、patch 16、3×3 块、每画布 313600 像素——单次调用用 `codec_config={"target_canvas": 32, ...}` 覆盖。其余键:`strategy`(`adaptive_pwindow`、`adaptive_rdo`、`readiness`)、`num_sampled_frames`、`max_pixels`、`canvas_format`。`patch=16` 和 `block_size=3` 必须与模型的 `patch_size`、`spatial_merge_size` 一致,不一致时预处理直接报错而非静默出错。画布缓存在 `~/.cache/codec_video_prep/canvases`(用 `CVP_CODEC_CACHE_ROOT` 换位置)。
|
| 191 |
|
| 192 |
-
|
| 193 |
|
| 194 |
-
|
| 195 |
|
| 196 |
-
**
|
| 197 |
|
| 198 |
-
**
|
| 199 |
|
| 200 |
-
**
|
| 201 |
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**。图像只有一列分数,因为 codec 路径只适用于视频。空格(`--`)表示尚未测出或不适用。
|
| 205 |
|
| 206 |
-
|
| 207 |
|
| 208 |
-
|
| 209 |
-
|---|---|---|
|
| 210 |
-
| **通用感知与阅读** | | |
|
| 211 |
-
| OCRBench v2 | 59 | 53.67 |
|
| 212 |
-
| OCRBench | 85.2 | 79.8 |
|
| 213 |
-
| OmniDocBench | 41.97 | -- |
|
| 214 |
-
| PerceptionBench | 30.87 | 29.6 |
|
| 215 |
-
| MMStar | 66.87 | 63.93 |
|
| 216 |
-
| MMBench-EN dev | 88.84 | 88.22 |
|
| 217 |
-
| RealWorldQA | 74.38 | 73.86 |
|
| 218 |
-
| BLINK | 54.97 | 67.06 |
|
| 219 |
-
| V*-Bench | 81.68 | 89.53 |
|
| 220 |
-
| DocVQA val | 95.21 | 92.92 |
|
| 221 |
-
| ChartQA | 84.96 | 82.76 |
|
| 222 |
-
| AI2D | 82.35 | 82.16 |
|
| 223 |
-
| CountBench | 87.17 | 84.93 |
|
| 224 |
-
| PixMo-Count | 63.11 | 69.85 |
|
| 225 |
-
| VLMs are Blind | 80.43 | 77.23 |
|
| 226 |
-
| VLMs are Biased | 24.96 | 25.29 |
|
| 227 |
-
| MuirBench | 21.27 | 48.69 |
|
| 228 |
-
| **定位与空间** | | |
|
| 229 |
-
| RefSpatial-Bench | 29.14 | 62.09 |
|
| 230 |
-
| RefCOCO val (Acc@0.5) | 89.07 | 92.64 |
|
| 231 |
-
| RefCOCOg val (Acc@0.5) | 87.65 | 88.91 |
|
| 232 |
-
| CrossPoint | 33.7 | 80.5 |
|
| 233 |
-
| SAT mcq circular | 68.67 | 74 |
|
| 234 |
-
| CV-Bench 2D | 80.74 | 83.31 |
|
| 235 |
-
| CV-Bench 3D | 92 | 94.5 |
|
| 236 |
-
| EmbSpatial | 74.7 | 83.02 |
|
| 237 |
-
| CRPE relation | 77.8 | 78.22 |
|
| 238 |
-
| ERQA | 43.5 | 46.5 |
|
| 239 |
-
| MMSI-Bench | 31.4 | 33.3 |
|
| 240 |
-
| MetaVQA | 61.75 | 61.03 |
|
| 241 |
-
| PointBench | 17.09 | 29.68 |
|
| 242 |
-
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 243 |
-
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 244 |
|
| 245 |
-
|
| 246 |
-
\* MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
|
| 247 |
-
\* MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
|
| 248 |
-
\* DocVQA val:按 ANLS 规则分。
|
| 249 |
-
\* PerceptionBench:由 gpt-oss-120b 判分。
|
| 250 |
-
\* PointBench:两边都输出 0~1000 坐标,而判分多按绝对像素算,因此两边绝对值都偏低。
|
| 251 |
-
\* RefSpatial-Bench:Qwen 为 2026-11-17 按现行判分脚本贪心重跑的结果。
|
| 252 |
-
\* SAT 自由作答与 SAT mcq 未列入——其 150 道题的正确选项全在第一位,一律答 A 即得 100%。表中报的是带循环一致性检查的 circular 变体。
|
| 253 |
-
|
| 254 |
-
### 视频
|
| 255 |
-
|
| 256 |
-
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 257 |
-
|---|---|---|---|
|
| 258 |
-
| **通用视频理解** | | | |
|
| 259 |
-
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 260 |
-
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 261 |
-
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 262 |
-
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 263 |
-
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 264 |
-
| MMVU val | 52.6 | 44.1 | -- |
|
| 265 |
-
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 266 |
-
| EgoSchema subset | 68.2 | 80 | -- |
|
| 267 |
-
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 268 |
-
| MMOU test-mini | -- | 45.96 | -- |
|
| 269 |
-
| **VideoMME** | | | |
|
| 270 |
-
| short,无字幕 | 81.67 | 79.67 | 81.11 |
|
| 271 |
-
| medium,无字幕 | 70.33 | 70.56 | 74.11 |
|
| 272 |
-
| long,无字幕 | 63 | 59 | 60.56 |
|
| 273 |
-
| short,有字幕 | -- | 82.44 | 82.67 |
|
| 274 |
-
| medium,有字幕 | -- | 75.44 | 75.56 |
|
| 275 |
-
| long,有字幕 | -- | 65.78 | 65.89 |
|
| 276 |
-
| **视频空间** | | | |
|
| 277 |
-
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 278 |
-
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 279 |
-
| **长视频** | | | |
|
| 280 |
-
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 281 |
-
| MLVU dev | 73.83 | 75.3 | -- |
|
| 282 |
-
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 283 |
-
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 284 |
-
| **时间定位(mIoU)** | | | |
|
| 285 |
-
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 286 |
-
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 287 |
-
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 288 |
-
|
| 289 |
-
\* TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
|
| 290 |
-
\* Video-MME-v2:字幕送法两边不同。
|
| 291 |
-
\* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 292 |
-
\* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 293 |
-
\* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
| 294 |
-
\* codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 295 |
-
|
| 296 |
-
## 评测口径
|
| 297 |
-
|
| 298 |
-
Qwen3.5-4B 关闭 thinking。它有一部分结果实际来自 temperature=0.7 采样而非贪心解码;已做过贪心重跑的行用重跑结果。所有数字都是单次运行,没有方差估计。
|
| 299 |
-
|
| 300 |
-
**图像。** OV2.5 用 processor 的默认像素预算;Qwen 为每图 65,536~16,777,216 px。
|
| 301 |
-
|
| 302 |
-
**视频。** 按 fps=1 抽帧并离线缓存,评测时不解码,重复跑输入完全一致。每个任务两边各取所有出分设置中的最高分,因此帧数和逐帧像素预算逐行、逐模型都不同——例如 LVBench,Qwen 的最高分出自 27 个设置,我们出自 16 个。codec 一列用 `preprocessor_config.json` 中 `codec` 字段的画布预算,按任务调整。
|
| 303 |
-
|
| 304 |
-
**判分。** 表中出现三种口径:
|
| 305 |
-
|
| 306 |
-
- *judge*——两边都由 Qwen3.8-27B(温度 0)复判,它看到问题、选项、参考答案和原始输出,回答 YES 或 NO,用于捞回"答对了但不匹配抽取正则"的情况。最终分取规则分与 judge 分中的较大者。
|
| 307 |
-
- *规则分(官方指标)*——按 bench 自己的官方指标直接计算。
|
| 308 |
-
- *规则分(judge 未出)*——本应用 judge,但至少一边尚未判,暂用规则分:MuirBench、EmbSpatial、CRPE relation、ERQA、MetaVQA、NextQA MC test、TempCompass MC、EgoSchema subset、Video-Index 512f。
|
| 309 |
-
|
| 310 |
-
所有生成都是贪心,`batch_size=1`。
|
| 311 |
|
| 312 |
-
|
| 313 |
-
|
| 314 |
-
| 项 | 值 |
|
| 315 |
-
|---|---|
|
| 316 |
-
| 语言层数 | 32(线性注意力,每 4 层一个全注意力) |
|
| 317 |
-
| hidden_size | 2560 |
|
| 318 |
-
| 注意力头 / KV 头 | 16 / 4 |
|
| 319 |
-
| 词表 | 248320 |
|
| 320 |
-
| 视觉层数 / hidden_size | 24 / 1024 |
|
| 321 |
-
| patch_size / spatial_merge_size | 16 / 3 |
|
| 322 |
-
| max_position_embeddings | 262144 |
|
| 323 |
-
| 权重 | bfloat16,3 分片,约 9.9 GB |
|
|
|
|
| 13 |
|
| 14 |
# LLaVA-OneVision-2.5-4B-Instruct
|
| 15 |
|
| 16 |
+
> 本仓库包含指令微调后的权重、随权重分发的建模代码,以及离线 / 流式两套对话模版。
|
| 17 |
|
| 18 |
+
LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图、多图和视频输入。除常规离线任务外,它还针对**流式**视频做了训练:画面逐帧到达,模型在每一步决定是保持沉默还是发言——两者共用同一份权重和同一个对话模版,用一个关键字切换。
|
| 19 |
|
| 20 |
+
## 亮点
|
| 21 |
+
|
| 22 |
+
- **一个模型,两种模式。** `apply_chat_template(..., streaming=True)` 切换默认 system prompt;八个协议 token 承载每一帧"沉默 / 发言 / caption"的决策。除此之外输入格式在两种模式下完全相同。
|
| 23 |
+
- **两条视频前端。** 均匀抽帧,或 **codec 模式**——用编码器的 bitcost 把信息量大的区域打包进少量画布。在本 checkpoint 上,两边都跑出结果的 21 个视频 bench 中,codec **赢 17 个**。
|
| 24 |
+
- **视频与空间能力突出。** 对 Qwen3.5-4B(关闭 thinking),抽帧在 21 个可比视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**;优势最大的是时间定位(TimeLens +12)和视频空间(VSI-Bench +9.4)。
|
| 25 |
+
- **图像侧如实呈现。** 静态图像上互有胜负——31 个可比项中领先 18 个,OCR 与文档阅读落后。下面的表格未作筛选。
|
| 26 |
+
|
| 27 |
+
## 模型概览
|
| 28 |
+
|
| 29 |
+
- **类型:** 指令微调视觉语言模型,LLaVA-OneVision-2 架构
|
| 30 |
+
- **参数量:** 合计 5.25B(4B 语言模型 + 视觉塔与 adapter)
|
| 31 |
+
- **语言侧:** 32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层
|
| 32 |
+
- **视觉侧:** 24 层,hidden size 1024,`patch_size=16`,`spatial_merge_size=3`,帧内注意力
|
| 33 |
+
- **词表:** 248320,含 8 个流式协议 token
|
| 34 |
+
- **上下文长度:** 262144
|
| 35 |
+
- **权重:** bfloat16,3 个 safetensors 分片,约 9.9 GB
|
| 36 |
+
|
| 37 |
+
## 评测结果
|
| 38 |
+
|
| 39 |
+
OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**。图像只有一列分数,因为 codec 路径只适用于视频。空格(`--`)表示尚未测出或不适用。
|
| 40 |
+
|
| 41 |
+
### 图像
|
| 42 |
+
|
| 43 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B |
|
| 44 |
+
|---|---|---|
|
| 45 |
+
| **通用感知与阅读** | | |
|
| 46 |
+
| OCRBench v2 | 59 | 53.67 |
|
| 47 |
+
| OCRBench | 85.2 | 79.8 |
|
| 48 |
+
| OmniDocBench | 41.97 | -- |
|
| 49 |
+
| PerceptionBench | 30.87 | 29.6 |
|
| 50 |
+
| MMStar | 66.87 | 63.93 |
|
| 51 |
+
| MMBench-EN dev | 88.84 | 88.22 |
|
| 52 |
+
| RealWorldQA | 74.38 | 73.86 |
|
| 53 |
+
| BLINK | 54.97 | 67.06 |
|
| 54 |
+
| V*-Bench | 81.68 | 89.53 |
|
| 55 |
+
| DocVQA val | 95.21 | 92.92 |
|
| 56 |
+
| ChartQA | 84.96 | 82.76 |
|
| 57 |
+
| AI2D | 82.35 | 82.16 |
|
| 58 |
+
| CountBench | 87.17 | 84.93 |
|
| 59 |
+
| PixMo-Count | 63.11 | 69.85 |
|
| 60 |
+
| VLMs are Blind | 80.43 | 77.23 |
|
| 61 |
+
| VLMs are Biased | 24.96 | 25.29 |
|
| 62 |
+
| MuirBench | 21.27 | 48.69 |
|
| 63 |
+
| **定位与空间** | | |
|
| 64 |
+
| RefSpatial-Bench | 29.14 | 62.09 |
|
| 65 |
+
| RefCOCO val (Acc@0.5) | 89.07 | 92.64 |
|
| 66 |
+
| RefCOCOg val (Acc@0.5) | 87.65 | 88.91 |
|
| 67 |
+
| CrossPoint | 33.7 | 80.5 |
|
| 68 |
+
| SAT mcq circular | 68.67 | 74 |
|
| 69 |
+
| CV-Bench 2D | 80.74 | 83.31 |
|
| 70 |
+
| CV-Bench 3D | 92 | 94.5 |
|
| 71 |
+
| EmbSpatial | 74.7 | 83.02 |
|
| 72 |
+
| CRPE relation | 77.8 | 78.22 |
|
| 73 |
+
| ERQA | 43.5 | 46.5 |
|
| 74 |
+
| MMSI-Bench | 31.4 | 33.3 |
|
| 75 |
+
| MetaVQA | 61.75 | 61.03 |
|
| 76 |
+
| PointBench | 17.09 | 29.68 |
|
| 77 |
+
| TraceSpatial 2D | 37.92 | 19.41 |
|
| 78 |
+
| TraceSpatial 3D | 12.99 | 44.6 |
|
| 79 |
+
|
| 80 |
+
\* BLINK:规则分取回答首字母。Qwen 约 15% 的回答先分析后给选项,因而被判错;按最终选项复算约为 62。
|
| 81 |
+
\* MuirBench:Qwen 只有规则分,很可能受同样的首字母解析影响,judge 未出,这一行暂不宜作为公平比较。
|
| 82 |
+
\* MMBench-EN dev:judge 为逐行准确率(4329 行);官方协议是循环评测。
|
| 83 |
+
\* DocVQA val:按 ANLS 规则分。PerceptionBench:由 gpt-oss-120b 判分。
|
| 84 |
+
\* PointBench:两边都输出 0~1000 坐标,而判分多按绝对像素算,因此两边绝对值都偏低。
|
| 85 |
+
\* RefSpatial-Bench:Qwen 为 2026-11-17 按现行判分脚本贪心重跑的结果。
|
| 86 |
+
\* SAT 自由作答与 SAT mcq 未列入——其 150 道题的正确选项全在第一位,一律答 A 即得 100%。表中报的是带循环一致性检查的 circular 变体。
|
| 87 |
+
|
| 88 |
+
### 视频
|
| 89 |
+
|
| 90 |
+
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 91 |
+
|---|---|---|---|
|
| 92 |
+
| **通用视频理解** | | | |
|
| 93 |
+
| MotionBench | 60.1 | 62.39 | 64.19 |
|
| 94 |
+
| TOMATO | 34.91 | 33.63 | 38.61 |
|
| 95 |
+
| MVBench | 64.03 | 72.03 | 75.5 |
|
| 96 |
+
| NextQA MC test | 80.67 | 86.06 | 86.4 |
|
| 97 |
+
| TempCompass MC | 73.35 | 75.76 | 79.68 |
|
| 98 |
+
| MMVU val | 52.6 | 44.1 | -- |
|
| 99 |
+
| Video-MME-v2 | 15.53 | 16.43 | 17.2 |
|
| 100 |
+
| EgoSchema subset | 68.2 | 80 | -- |
|
| 101 |
+
| Video-Index 512f | 19.64 | 20 | 19.76 |
|
| 102 |
+
| MMOU test-mini | -- | 45.96 | -- |
|
| 103 |
+
| **VideoMME** | | | |
|
| 104 |
+
| short,无字幕 | 81.67 | 79.67 | 81.11 |
|
| 105 |
+
| medium,无字幕 | 70.33 | 70.56 | 74.11 |
|
| 106 |
+
| long,无字幕 | 63 | 59 | 60.56 |
|
| 107 |
+
| short,有字幕 | -- | 82.44 | 82.67 |
|
| 108 |
+
| medium,有字幕 | -- | 75.44 | 75.56 |
|
| 109 |
+
| long,有字幕 | -- | 65.78 | 65.89 |
|
| 110 |
+
| **视频空间** | | | |
|
| 111 |
+
| VSI-Bench | 59.22 | 68.64 | 67.72 |
|
| 112 |
+
| ReVSI 64f | 47.1 | 55.94 | 55.54 |
|
| 113 |
+
| **长视频** | | | |
|
| 114 |
+
| LVBench | 56.49 | 62.04 | 61.59 |
|
| 115 |
+
| MLVU dev | 73.83 | 75.3 | -- |
|
| 116 |
+
| LongVideoBench val | 62.68 | 68.96 | 71.06 |
|
| 117 |
+
| VideoEval-Pro | 62.84 | 64.39 | 64.62 |
|
| 118 |
+
| **时间定位(mIoU)** | | | |
|
| 119 |
+
| TimeLens ActivityNet | 46.83 | 58.84 | 59.11 |
|
| 120 |
+
| TimeLens Charades | 48.05 | 60.58 | 61.25 |
|
| 121 |
+
| TimeLens QVHighlights | 67.51 | 68.39 | 70.31 |
|
| 122 |
+
|
| 123 |
+
\* TOMATO:Qwen 为 fps=2 / 16 帧;我方未传帧数和像素参数,用的是模型默认值。
|
| 124 |
+
\* Video-MME-v2:字幕送法两边不同。
|
| 125 |
+
\* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
|
| 126 |
+
\* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
|
| 127 |
+
\* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
|
| 128 |
+
\* codec 一列为规则分——回答均为选项字母,与 judge 等价。
|
| 129 |
+
|
| 130 |
+
### 评测口径
|
| 131 |
+
|
| 132 |
+
Qwen3.5-4B 关闭 thinking。它有一部分结果实际来自 temperature=0.7 采样而非贪心解码;已做过贪心重跑的行用重跑结果。所有数字都是单次运行,没有方差估计。
|
| 133 |
+
|
| 134 |
+
- **图像。** OV2.5 用 processor 的默认像素预算;Qwen 为每图 65,536~16,777,216 px。
|
| 135 |
+
- **视频。** 按 fps=1 抽帧并离线缓存,评测时不解码,重复跑输入完全一致。每个任务两边各取所有出分设置中的最高分,因此帧数和逐帧像素预算逐行、逐模型都不同——例如 LVBench,Qwen 的最高分出自 27 个设置,我们出自 16 个。codec 一列用 `preprocessor_config.json` 中 `codec` 字段的画布预算,按任务调整。
|
| 136 |
+
- **判分。** *judge*——两边都由 Qwen3.8-27B(温度 0)复判,它看到问题、选项、参考答案和原始输出,回答 YES 或 NO;最终分取规则分与 judge 分中的较大者。*规则分(官方指标)*——按 bench 自己的指标计算。*规则分(judge 未出)*——本应用 judge 但至少一边尚未判:MuirBench、EmbSpatial、CRPE relation、ERQA、MetaVQA、NextQA MC test、TempCompass MC、EgoSchema subset、Video-Index 512f。
|
| 137 |
+
|
| 138 |
+
所有生成都是贪心,`batch_size=1`。
|
| 139 |
+
|
| 140 |
+
## 快速开始
|
| 141 |
+
|
| 142 |
+
> 建模代码随权重分发,所有入口都必须 `trust_remote_code=True`。
|
| 143 |
|
| 144 |
```bash
|
| 145 |
pip install "transformers>=5.7.0.dev0" accelerate torch torchvision
|
|
|
|
| 147 |
pip install decord # 仅抽帧视频路径需要
|
| 148 |
```
|
| 149 |
|
| 150 |
+
### 图像
|
| 151 |
|
| 152 |
```python
|
| 153 |
import torch
|
|
|
|
| 176 |
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
|
| 177 |
```
|
| 178 |
|
| 179 |
+
图片走 `smart_resize`,保持长宽比并对齐 patch 网格,预算默认 `min_pixels=3136` / `max_pixels=4000000`。多图时每张图放一个 `{"type": "image"}`,`images=` 按同样顺序传入。
|
|
|
|
|
|
|
| 180 |
|
| 181 |
+
### 视频 — 抽帧
|
| 182 |
|
| 183 |
```python
|
| 184 |
messages = [{"role": "user", "content": [
|
|
|
|
| 197 |
|
| 198 |
也接受 `target_fps` 和 `num_frames`,优先级见 `video_processing_llava_onevision2_5.py` 的 `choose_target_frames`。`max_frames` 封顶 768。
|
| 199 |
|
| 200 |
+
时间戳一定会注入:`<|video_pad|>` 占位符展开成逐帧的块,每块前面是该帧的真实采样时刻,因此可以直接问时间定位类问题。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
|
| 202 |
```
|
| 203 |
<0.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 204 |
<1.0 seconds><|vision_start|><|image_pad|>×N<|vision_end|>
|
| 205 |
```
|
| 206 |
|
| 207 |
+
### 流式
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 208 |
|
| 209 |
+
画面每轮进来一帧,模型每轮输出一个状态 token。一个关键字切换默认 system,其余什么都不改;显式传入的 `system` 在两种模式下都优先。
|
|
|
|
|
|
|
| 210 |
|
| 211 |
| 模式 | 调用 | 默认 system |
|
| 212 |
|---|---|---|
|
| 213 |
| 离线(默认) | `apply_chat_template(messages, ...)` | `You are a helpful assistant.` |
|
| 214 |
| 流式 | `apply_chat_template(messages, ..., streaming=True)` | `You are a real-time streaming video assistant.` |
|
| 215 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 216 |
| Token | Id | 含义 |
|
| 217 |
|---|---|---|
|
| 218 |
| `<silence>` | 248077 | 当前轮不发言 |
|
|
|
|
| 222 |
| `<captioning>` | 248083 | 当前轮发言,且是事件级 caption |
|
| 223 |
| `<caption_begin>` | 248084 | user 侧标记,说明这条流是 caption 任务 |
|
| 224 |
|
| 225 |
+
八个都是非 special 的 added token,所以 `skip_special_tokens=True` 仍会解出来——状态 token 正是你要读的输出。两个 memory token 内部包 JSON,例如 `{"timestep": "00:00:06", "content": "..."}`。
|
|
|
|
|
|
|
| 226 |
|
| 227 |
严格 user / assistant 交替,每个 user 轮最多一帧;没有新画面的轮次改用纯文本时间前缀。每帧给上 `timestamp`,帧后的换行作为单独的 text 项:
|
| 228 |
|
|
|
|
| 236 |
messages, tokenize=False, add_generation_prompt=True, streaming=True)
|
| 237 |
```
|
| 238 |
|
|
|
|
|
|
|
| 239 |
```
|
| 240 |
<|im_start|>user
|
| 241 |
<question_memory>{"timestep": "00:00:00", "content": "..."}</question_memory>
|
|
|
|
| 247 |
|
| 248 |
content 传字符串则原样输出,适合自己已经拼好标记的 harness。
|
| 249 |
|
| 250 |
+
> **本仓库不包含:** 围绕模型的实时循环——按时钟取帧、决定何时停止生成、到期回插 question / answer memory、在决策点之间维持上下文。
|
| 251 |
|
| 252 |
+
### 视频 — codec 模式
|
| 253 |
|
| 254 |
+
codec 模式用编码器 bitcost 把信息量大的区域打包进少量**画布**。同一张画布上的 patch 可以来自不同源帧,各自带 `(源帧, 行, 列)` 供 3-D RoPE 使用。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 255 |
|
| 256 |
预处理来自 `codec-video-prep`(自带打过补丁的 FFmpeg 和 dav1d;限 Linux x86_64、CPython 3.10/3.11、`numpy<2`、系统 `ffprobe`;只支持 H.264/HEVC/VP9/AV1;只适用于离线文件,因为预算计算需要整段视频):
|
| 257 |
|
|
|
|
| 266 |
video_backend="codec", return_tensors="pt").to(model.device)
|
| 267 |
```
|
| 268 |
|
| 269 |
+
每张画布只有一个时间标记:所有 patch 来自同一源帧时是点 `<6.5 seconds>`,跨多帧时是区间 `<0.0 to 1.8 seconds>`(schema `canvas_patch_span_point_range_v1`)。
|
|
|
|
|
|
|
|
|
|
|
|
|
| 270 |
|
| 271 |
+
默认配方写在 `preprocessor_config.json` 的 `codec` 字段——512 候选帧、64 画布、patch 16、3×3 块、每画布 313600 像素——单次调用用 `codec_config={"target_canvas": 32, ...}` 覆盖。其余键:`strategy`(`adaptive_pwindow`、`adaptive_rdo`、`readiness`)、`num_sampled_frames`、`max_pixels`、`canvas_format`。画布缓存在 `~/.cache/codec_video_prep/canvases`(用 `CVP_CODEC_CACHE_ROOT` 换位置)。
|
| 272 |
|
| 273 |
+
## 最佳实践
|
| 274 |
|
| 275 |
+
1. **传视频路径,不要传自己解码好的帧。** 只有路径才能让 processor 拿到真实采样时刻和源帧号。只给一串帧时它会退化成用帧序号当秒数,于是 100 秒的视频看起来只有 N 秒,时间类回答全错(此时会抛 `UserWarning`)。必须自己解码就把时间一并交出:`videos=[{"frames": ..., "timestamps": ..., "frame_indices": ...}]`。通过 `images=` 传帧则完全不注入时间戳。
|
| 276 |
|
| 277 |
+
2. **保持 `patch_positions` 为真。** codec 模式下它在 processor 输出里,必须转发给 `model.forward(...)`,不要丢掉。流式下每帧作为独立 image 交付,块布局是对的但 `t` 每帧都是 0,需要改成 `floor(秒数 × fps)`,否则每帧都会被当成一段新视觉的开头。模型只用 `t` 的**名次**,但写真实帧号能让输入可审计。
|
| 278 |
|
| 279 |
+
3. **帧数与逐帧像素一起调。** 开销约等于两者之积:短视频 64~128 帧 × 153664~321489 px,中等 256 帧 × 136900,长视频 640~768 帧 × 72900~108900。超长输入设 `OV_VISION_CHUNK_PATCHES=200000`,它按帧边界分组编码,注意力不跨帧,结果不变。
|
| 280 |
|
| 281 |
+
4. **codec 几何必须对齐模型。** `patch=16` 和 `block_size=3` 必须等于 `patch_size` 和 `spatial_merge_size`,不一致时预处理直接报错而非静默出错。
|
|
|
|
|
|
|
| 282 |
|
| 283 |
+
5. **贪心解码,并留足推理长度。** `generation_config.json` 只设了 `eos_token_id=248044` 和 `use_cache=true`。确定性输出用 `do_sample=False`;需要先推理再作答的任务要给足 `max_new_tokens`,128 会在中途截断。批量时传一组 text 并设 `padding=True`,padding 在左侧,按各自的 `input_ids` 长度切输出。
|
| 284 |
|
| 285 |
+
6. **要什么格式就在 prompt 里说。** 选择题上模型可能写整句而不是一个字母;如果后续用正则解析,就要求给字母:`Answer with the option's letter from the given choices directly.`。grounding 任务请写明坐标范围,例如 0~1000 的整数。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 286 |
|
| 287 |
+
7. **多进程前先单进程预热 remote code。** 模块会编译到 `$HF_HOME/modules/transformers_modules/<名字>/`,多进程首次加载会在那里竞争,所以先单进程完整加载一次。该缓存只按叶子目录名索引——本地副本要取不同的目录名。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 288 |
|
| 289 |
+
8. **不需要流式 token 就显式剥除。** 八个都按设计不会被 `skip_special_tokens=True` 过滤。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|