| --- |
| language: |
| - en |
| - zh |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: video-text-to-text |
| tags: |
| - multimodal |
| - vision-language |
| - image-text-to-text |
| - video-text-to-text |
| - llava |
| - llava-onevision-2 |
| - qwen3 |
| --- |
| |
| # LLaVA-OneVision-2-8B-Instruct |
|
|
| [Paper](https://huggingface.co/papers/2605.25979) | [Project Page](https://evolvinglmms-lab.github.io/LLaVA-OneVision-2/) | [GitHub](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-2) |
|
|
| LLaVA-OneVision-2 (LLaVA-OV-2) is a multimodal vision-language model that handles **single images, multi-image, and video** inputs, built on a Qwen3-8B language backbone with a OneVision-style vision encoder. Its key advance is codec-stream tokenization, which treats compressed video as a continuous bit-cost stream for efficient long-video understanding. |
|
|
| The model is distributed as a HuggingFace `transformers` checkpoint with custom code (`trust_remote_code=True`). |
|
|
| ## Requirements |
|
|
| ### Base (image + frame-sampling video) |
|
|
| ```bash |
| pip install "transformers>=5.7.0" "torch>=2.4" pillow requests decord |
| ``` |
|
|
| ### Optional: codec video backend |
|
|
| The model ships a second video backend (`video_backend="codec"`) that replaces |
| uniform frame sampling with codec-aware **canvas packing** driven by motion |
| vectors and bit-cost — typically yielding stronger long-video accuracy at the |
| same token budget. To enable it you need two extra pieces: |
|
|
| ```bash |
| # 1. The cv-preinfer CLI (PyPI: codec-video-prep) drives canvas extraction. |
| pip install codec-video-prep opencv-python |
| |
| # 2. A working `ffmpeg` binary must be on PATH. |
| # Verify with: ffmpeg -version |
| ``` |
|
|
| **ffmpeg version:** ffmpeg **4.4.x – 7.x** is recommended. |
|
|
| The codec backend additionally needs **POSIX `flock`** (already present on |
| Linux/macOS) for the on-disk result cache, and roughly **2 GB free disk** under |
| `$ONLINE_CODEC_CACHE_DIR` (defaults to `$HF_HOME/online_codec`) per |
| processed video. |
|
|
| ## Quick start |
|
|
| The repository ships a ready-to-run `demo_inference.py` that covers both image and video paths. |
|
|
| ```bash |
| # Image (default sample image; no auth required) |
| python demo_inference.py |
| |
| # Image, custom file + prompt |
| python demo_inference.py --mode image --media /path/to/cat.jpg \ |
| --prompt "What is the cat doing?" |
| |
| # Video (16 uniformly-sampled frames; max-pixels caps per-frame resolution for memory) |
| python demo_inference.py --mode video --media /path/to/clip.mp4 \ |
| --num-frames 16 --max-pixels 200704 \ |
| --prompt "Describe what happens in this video." |
| ``` |
|
|
| ## Programmatic use |
|
|
| ```python |
| import torch |
| from transformers import AutoProcessor, AutoModelForImageTextToText |
| from PIL import Image |
| |
| MODEL_ID = "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" |
| |
| processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True) |
| model = AutoModelForImageTextToText.from_pretrained( |
| MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda", |
| ).eval() |
| |
| # ----- Image ----- |
| image = Image.open("cat.jpg").convert("RGB") |
| messages = [{"role": "user", "content": [ |
| {"type": "image"}, |
| {"type": "text", "text": "Describe this image in detail."}, |
| ]}] |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True) |
| inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} |
| |
| out = model.generate(**inputs, max_new_tokens=256, do_sample=False) |
| print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| |
| # ----- Video ----- |
| # Lower max_pixels if you hit OOM on long videos. |
| processor.video_processor.max_pixels = 200704 |
| |
| messages = [{"role": "user", "content": [ |
| {"type": "video"}, |
| {"type": "text", "text": "Describe what happens in this video."}, |
| ]}] |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = processor( |
| text=[text], videos=["clip.mp4"], return_tensors="pt", padding=True, |
| num_frames=16, # exact frame count; or use target_fps / max_frames |
| ) |
| inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} |
| out = model.generate(**inputs, max_new_tokens=256, do_sample=False) |
| print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### Video — codec backend (recommended for long videos) |
|
|
| The codec backend is exposed as a single processor kwarg |
| (`video_backend="codec"`). Everything else — canvas extraction via |
| `cv-preinfer`, on-disk caching, patch-position bookkeeping, chat-template |
| rewriting — happens inside `processor(...)`: |
|
|
| ```python |
| # Make sure: `pip install codec-video-prep opencv-python` and ffmpeg on PATH. |
| messages = [{"role": "user", "content": [ |
| {"type": "video"}, |
| {"type": "text", "text": "Describe what happens in this long video."}, |
| ]}] |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| |
| inputs = processor( |
| text=[text], |
| videos=["long_clip.mp4"], |
| video_backend="codec", |
| max_pixels=150000, # per-canvas pixel budget; lower if OOM |
| return_tensors="pt", |
| padding=True, |
| # Optional: override codec defaults from preprocessor_config.json |
| # codec_config={"target_canvas": 32, "group_size": 32, "images_per_group": 4}, |
| ) |
| inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} |
| |
| out = model.generate(**inputs, max_new_tokens=256, do_sample=False) |
| print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Notes |
|
|
| - The vision tower is a OneVision-style encoder; the language backbone is **Qwen3-8B**. |
| - `chat_template.jinja` follows the Qwen3 chat format and emits `<|vision_start|>...<|vision_end|>` placeholders; the processor expands them per-frame (frames backend) or per-canvas-patch-run (codec backend). |
| - Two video backends are available via `processor(..., video_backend=...)`: **`"frames"`** (default, uniform sampling) and **`"codec"`** (canvas packing via `cv-preinfer`, requires `codec-video-prep` + `ffmpeg`). |
| - Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends. |
|
|
| ## License |
|
|
| Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{LLaVA-OneVision-2, |
| title={LLaVA-OneVision-2}, |
| author={llava-onevision contributors}, |
| booktitle={arXiv}, |
| year={2026} |
| } |
| |
| @inproceedings{LLaVA-OneVision-1.5, |
| title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training}, |
| author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang}, |
| booktitle={arXiv}, |
| year={2025} |
| } |
| ``` |
