README.md

metadata

language:
  - en
  - zh
library_name: transformers
license: apache-2.0
pipeline_tag: video-text-to-text
tags:
  - multimodal
  - vision-language
  - image-text-to-text
  - video-text-to-text
  - llava
  - llava-onevision-2
  - qwen3

LLaVA-OneVision-2-8B-Instruct

Paper | Project Page | GitHub

LLaVA-OneVision-2 (LLaVA-OV-2) is a multimodal vision-language model that handles single images, multi-image, and video inputs, built on a Qwen3-8B language backbone with a OneVision-style vision encoder. Its key advance is codec-stream tokenization, which treats compressed video as a continuous bit-cost stream for efficient long-video understanding.

The model is distributed as a HuggingFace transformers checkpoint with custom code (trust_remote_code=True).

Requirements

Base (image + frame-sampling video)

pip install "transformers>=5.7.0" "torch>=2.4" pillow requests decord

Optional: codec video backend

The model ships a second video backend (video_backend="codec") that replaces uniform frame sampling with codec-aware canvas packing driven by motion vectors and bit-cost — typically yielding stronger long-video accuracy at the same token budget. To enable it you need two extra pieces:

# 1. The cv-preinfer CLI (PyPI: codec-video-prep) drives canvas extraction.
pip install codec-video-prep opencv-python

# 2. A working `ffmpeg` binary must be on PATH.
#    Verify with: ffmpeg -version

ffmpeg version: ffmpeg 4.4.x – 7.x is recommended.

The codec backend additionally needs POSIX flock (already present on Linux/macOS) for the on-disk result cache, and roughly 2 GB free disk under $ONLINE_CODEC_CACHE_DIR (defaults to $HF_HOME/online_codec) per processed video.

Quick start

The repository ships a ready-to-run demo_inference.py that covers both image and video paths.

# Image (default sample image; no auth required)
python demo_inference.py

# Image, custom file + prompt
python demo_inference.py --mode image --media /path/to/cat.jpg \
    --prompt "What is the cat doing?"

# Video (16 uniformly-sampled frames; max-pixels caps per-frame resolution for memory)
python demo_inference.py --mode video --media /path/to/clip.mp4 \
    --num-frames 16 --max-pixels 200704 \
    --prompt "Describe what happens in this video."

Programmatic use

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

MODEL_ID = "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda",
).eval()

# ----- Image -----
image = Image.open("cat.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image in detail."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

# ----- Video -----
# Lower max_pixels if you hit OOM on long videos.
processor.video_processor.max_pixels = 200704

messages = [{"role": "user", "content": [
    {"type": "video"},
    {"type": "text", "text": "Describe what happens in this video."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(
    text=[text], videos=["clip.mp4"], return_tensors="pt", padding=True,
    num_frames=16,  # exact frame count; or use target_fps / max_frames
)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Video — codec backend (recommended for long videos)

The codec backend is exposed as a single processor kwarg (video_backend="codec"). Everything else — canvas extraction via cv-preinfer, on-disk caching, patch-position bookkeeping, chat-template rewriting — happens inside processor(...):

# Make sure: `pip install codec-video-prep opencv-python` and ffmpeg on PATH.
messages = [{"role": "user", "content": [
    {"type": "video"},
    {"type": "text", "text": "Describe what happens in this long video."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = processor(
    text=[text],
    videos=["long_clip.mp4"],
    video_backend="codec",
    max_pixels=150000,          # per-canvas pixel budget; lower if OOM
    return_tensors="pt",
    padding=True,
    # Optional: override codec defaults from preprocessor_config.json
    # codec_config={"target_canvas": 32, "group_size": 32, "images_per_group": 4},
)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Notes

• The vision tower is a OneVision-style encoder; the language backbone is Qwen3-8B.
• chat_template.jinja follows the Qwen3 chat format and emits <|vision_start|>...<|vision_end|> placeholders; the processor expands them per-frame (frames backend) or per-canvas-patch-run (codec backend).
• Two video backends are available via processor(..., video_backend=...): "frames" (default, uniform sampling) and "codec" (canvas packing via cv-preinfer, requires codec-video-prep + ffmpeg).
• Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends.

License

Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see Qwen/Qwen3-8B.

Citation

@inproceedings{LLaVA-OneVision-2,
  title={LLaVA-OneVision-2},
  author={llava-onevision contributors},
  booktitle={arXiv},
  year={2026}
}

@inproceedings{LLaVA-OneVision-1.5,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
  booktitle={arXiv},
  year={2025}
 }