LLaVA-OneVision2-8B-Instruct

A multimodal vision-language model that handles single images, multi-image, and video inputs, built on a Qwen3-8B language backbone with a OneVision-style vision encoder.

The model is distributed as a HuggingFace transformers checkpoint with custom code (trust_remote_code=True).

Requirements

pip install "transformers>=5.7.0" "torch>=2.4" pillow requests decord

Quick start

The repository ships a ready-to-run demo_inference.py that covers both image and video paths.

# Image (default sample image; no auth required)
python demo_inference.py

# Image, custom file + prompt
python demo_inference.py --mode image --media /path/to/cat.jpg \
    --prompt "What is the cat doing?"

# Video (16 uniformly-sampled frames; max-pixels caps per-frame resolution for memory)
python demo_inference.py --mode video --media /path/to/clip.mp4 \
    --num-frames 16 --max-pixels 200704 \
    --prompt "Describe what happens in this video."

Programmatic use

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

MODEL_ID = "lmms-lab-encoder/LLaVA-OneVision2-8B-Instruct"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda",
).eval()

# ----- Image -----
image = Image.open("cat.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image in detail."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}

out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

# ----- Video -----
# Lower max_pixels if you hit OOM on long videos.
processor.video_processor.max_pixels = 200704

messages = [{"role": "user", "content": [
    {"type": "video"},
    {"type": "text", "text": "Describe what happens in this video."},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(
    text=[text], videos=["clip.mp4"], return_tensors="pt", padding=True,
    num_frames=16,  # exact frame count; or use target_fps / max_frames
)
inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Notes

• The vision tower is a OneVision-style encoder; the language backbone is Qwen3-8B.
• chat_template.jinja follows the Qwen3 chat format and emits <|vision_start|>...<|vision_end|> placeholders; the processor expands them per-frame for video.
• Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation.

License

Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see Qwen/Qwen3-8B.