Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
Clean stale comments and docstrings (no code change)
Browse files- README.md +1 -1
- README_CN.md +1 -1
- chat_template.jinja +2 -3
- configuration_llava_onevision2_5.py +1 -3
- modeling_llava_onevision2_5.py +10 -12
README.md
CHANGED
|
@@ -124,7 +124,7 @@ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image,
|
|
| 124 |
|
| 125 |
<table style="width:100%; display:table;">
|
| 126 |
<tbody>
|
| 127 |
-
<tr><td align="left" style="width:28%"><b>Type</b></td><td align="left">instruction-tuned vision-language model, LLaVA-OneVision-2 architecture</td></tr>
|
| 128 |
<tr><td align="left" style="width:28%"><b>Parameters</b></td><td align="left">5.25B total (4B language model plus vision tower and adapter)</td></tr>
|
| 129 |
<tr><td align="left" style="width:28%"><b>Language side</b></td><td align="left">32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4</td></tr>
|
| 130 |
<tr><td align="left" style="width:28%"><b>Vision side</b></td><td align="left">24 layers, hidden size 1024, <code>patch_size=16</code>, <code>spatial_merge_size=3</code>, attention within a frame</td></tr>
|
|
|
|
| 124 |
|
| 125 |
<table style="width:100%; display:table;">
|
| 126 |
<tbody>
|
| 127 |
+
<tr><td align="left" style="width:28%"><b>Type</b></td><td align="left">instruction-tuned vision-language model, LLaVA-OneVision-2.5 architecture</td></tr>
|
| 128 |
<tr><td align="left" style="width:28%"><b>Parameters</b></td><td align="left">5.25B total (4B language model plus vision tower and adapter)</td></tr>
|
| 129 |
<tr><td align="left" style="width:28%"><b>Language side</b></td><td align="left">32 layers, hidden size 2560, FFN 9216, 16 attention heads / 4 KV heads, linear attention with one full-attention layer every 4</td></tr>
|
| 130 |
<tr><td align="left" style="width:28%"><b>Vision side</b></td><td align="left">24 layers, hidden size 1024, <code>patch_size=16</code>, <code>spatial_merge_size=3</code>, attention within a frame</td></tr>
|
README_CN.md
CHANGED
|
@@ -124,7 +124,7 @@ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图
|
|
| 124 |
|
| 125 |
<table style="width:100%; display:table;">
|
| 126 |
<tbody>
|
| 127 |
-
<tr><td align="left" style="width:28%"><b>类型</b></td><td align="left">指令微调视觉语言模型,LLaVA-OneVision-2 架构</td></tr>
|
| 128 |
<tr><td align="left" style="width:28%"><b>参数量</b></td><td align="left">合计 5.25B(4B 语言模型 + 视觉塔与 adapter)</td></tr>
|
| 129 |
<tr><td align="left" style="width:28%"><b>语言侧</b></td><td align="left">32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层</td></tr>
|
| 130 |
<tr><td align="left" style="width:28%"><b>视觉侧</b></td><td align="left">24 层,hidden size 1024,<code>patch_size=16</code>,<code>spatial_merge_size=3</code>,帧内注意力</td></tr>
|
|
|
|
| 124 |
|
| 125 |
<table style="width:100%; display:table;">
|
| 126 |
<tbody>
|
| 127 |
+
<tr><td align="left" style="width:28%"><b>类型</b></td><td align="left">指令微调视觉语言模型,LLaVA-OneVision-2.5 架构</td></tr>
|
| 128 |
<tr><td align="left" style="width:28%"><b>参数量</b></td><td align="left">合计 5.25B(4B 语言模型 + 视觉塔与 adapter)</td></tr>
|
| 129 |
<tr><td align="left" style="width:28%"><b>语言侧</b></td><td align="left">32 层,hidden size 2560,FFN 9216,16 个注意力头 / 4 个 KV 头,线性注意力每 4 层夹一个全注意力层</td></tr>
|
| 130 |
<tr><td align="left" style="width:28%"><b>视觉侧</b></td><td align="left">24 层,hidden size 1024,<code>patch_size=16</code>,<code>spatial_merge_size=3</code>,帧内注意力</td></tr>
|
chat_template.jinja
CHANGED
|
@@ -1,11 +1,10 @@
|
|
| 1 |
-
{#- OV2.5 Qwen3.5 chat template with two modes, aligned with the training renderer (
|
| 2 |
template + canvas timestamps). Pick the mode per call:
|
| 3 |
processor.apply_chat_template(messages, ...) -> offline (default)
|
| 4 |
processor.apply_chat_template(messages, ..., streaming=True) -> streaming
|
| 5 |
The modes differ only in the default system turn, exactly as in training: offline samples get
|
| 6 |
the qwen2-vl default system, data_type=streaming samples get the streaming prompt. An explicit
|
| 7 |
-
system message always wins.
|
| 8 |
-
time by finalize_streaming_export.py, so they cannot drift.)
|
| 9 |
Layout: <|im_start|>system\n{system}<|im_end|>\n then per turn <|im_start|>{role}\n{content}<|im_end|>\n.
|
| 10 |
- String content is emitted verbatim (callers that pre-render "<T seconds><|vision_start|>...").
|
| 11 |
- An image item may carry "timestamp" (seconds = frame_index / fps); it is rendered as the trainer
|
|
|
|
| 1 |
+
{#- OV2.5 Qwen3.5 chat template with two modes, aligned with the training renderer (`qwen2-vl`
|
| 2 |
template + canvas timestamps). Pick the mode per call:
|
| 3 |
processor.apply_chat_template(messages, ...) -> offline (default)
|
| 4 |
processor.apply_chat_template(messages, ..., streaming=True) -> streaming
|
| 5 |
The modes differ only in the default system turn, exactly as in training: offline samples get
|
| 6 |
the qwen2-vl default system, data_type=streaming samples get the streaming prompt. An explicit
|
| 7 |
+
system message always wins.
|
|
|
|
| 8 |
Layout: <|im_start|>system\n{system}<|im_end|>\n then per turn <|im_start|>{role}\n{content}<|im_end|>\n.
|
| 9 |
- String content is emitted verbatim (callers that pre-render "<T seconds><|vision_start|>...").
|
| 10 |
- An image item may carry "timestamp" (seconds = frame_index / fps); it is rendered as the trainer
|
configuration_llava_onevision2_5.py
CHANGED
|
@@ -34,9 +34,7 @@ class LlavaOnevision2VisionConfig(PreTrainedConfig):
|
|
| 34 |
class LlavaOnevision2Config(PreTrainedConfig):
|
| 35 |
r"""
|
| 36 |
This is the configuration class to store the configuration of a [`LlavaOnevision2Model`]. It is used to instantiate a
|
| 37 |
-
LlavaOnevision2Model model according to the specified arguments, defining the model architecture.
|
| 38 |
-
with the defaults will yield a similar configuration to that of
|
| 39 |
-
Llava-Onevision 1.5 [lmms-lab/LLaVA-OneVision-1.5-8B-Instruct](https://huggingface.co/lmms-lab/LLaVA-OneVision-1.5-8B-Instruct).
|
| 40 |
|
| 41 |
Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
|
| 42 |
documentation from [`PreTrainedConfig`] for more information.
|
|
|
|
| 34 |
class LlavaOnevision2Config(PreTrainedConfig):
|
| 35 |
r"""
|
| 36 |
This is the configuration class to store the configuration of a [`LlavaOnevision2Model`]. It is used to instantiate a
|
| 37 |
+
LlavaOnevision2Model model according to the specified arguments, defining the model architecture.
|
|
|
|
|
|
|
| 38 |
|
| 39 |
Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
|
| 40 |
documentation from [`PreTrainedConfig`] for more information.
|
modeling_llava_onevision2_5.py
CHANGED
|
@@ -61,8 +61,7 @@ def _mrope_positions_from_patch_positions(
|
|
| 61 |
):
|
| 62 |
"""Train-parity 3D M-RoPE positions.
|
| 63 |
|
| 64 |
-
Verbatim port of the TRAINING-side ``_rope_from_patch_positions`` (
|
| 65 |
-
llava_onevision2_model.py) so eval positions match what the model saw in
|
| 66 |
training exactly — including its convention that the text position cursor
|
| 67 |
does NOT skip past a visual's span (text after an image resumes at the
|
| 68 |
pre-image counter). Do NOT "fix" that here; train/infer parity wins.
|
|
@@ -773,7 +772,7 @@ class Siglip2MultiheadAttentionPoolingHead(nn.Module):
|
|
| 773 |
|
| 774 |
class LlavaOnevision2VisionPretrainedModel(LlavaOnevision2PreTrainedModel):
|
| 775 |
"""
|
| 776 |
-
LLaVA-OneVision
|
| 777 |
|
| 778 |
This vision model is designed to work with Qwen2VL-style image processing:
|
| 779 |
- Receives pre-processed patches in 3x3 block spatial order
|
|
@@ -1085,18 +1084,18 @@ class LlavaOnevision2VisionPretrainedModel(LlavaOnevision2PreTrainedModel):
|
|
| 1085 |
grid_thw: Optional[torch.Tensor] = None,
|
| 1086 |
patch_positions: Optional[torch.Tensor] = None,
|
| 1087 |
) -> dict:
|
| 1088 |
-
"""Stage-by-stage forward used by
|
| 1089 |
|
| 1090 |
Mirrors the production ``forward`` exactly EXCEPT that ``patch_positions``
|
| 1091 |
passed in are assumed to be in row-major (t, h, w) order — matching the
|
| 1092 |
-
|
| 1093 |
which also receives row-major positions and applies a block-layout
|
| 1094 |
reorder. Production ``forward`` skips that reorder because production
|
| 1095 |
``patch_positions`` come pre-arranged in block layout from the
|
| 1096 |
Qwen2VL processor (`build_patch_positions`).
|
| 1097 |
|
| 1098 |
-
Layer-wise outputs match the schema produced by
|
| 1099 |
-
``
|
| 1100 |
"""
|
| 1101 |
output: dict[str, torch.Tensor] = {}
|
| 1102 |
|
|
@@ -1581,7 +1580,7 @@ class LlavaOnevision2ForConditionalGeneration(LlavaOnevision2PreTrainedModel, Ge
|
|
| 1581 |
`position_ids`, this tensor is not affected by padding.
|
| 1582 |
|
| 1583 |
Note (P0 native-video alias):
|
| 1584 |
-
The companion ``
|
| 1585 |
pass ``pixel_values_videos`` / ``video_grid_thw`` / ``second_per_grid_ts``
|
| 1586 |
to this forward. Instead it aliases the video patch tensor as
|
| 1587 |
``pixel_values=`` and ``image_grid_thw=``, so video inputs share the
|
|
@@ -1590,8 +1589,7 @@ class LlavaOnevision2ForConditionalGeneration(LlavaOnevision2PreTrainedModel, Ge
|
|
| 1590 |
text tags emitted by the processor). The ``*_videos`` and
|
| 1591 |
``second_per_grid_ts`` kwargs are kept declared here only for API
|
| 1592 |
completeness and future use (e.g. 3D mRoPE / ``get_rope_index``); they
|
| 1593 |
-
are NOT consumed by the current OneVision encoder.
|
| 1594 |
-
NATIVE_VIDEO_PLAN.md (stage 3.1) for the full rationale.
|
| 1595 |
|
| 1596 |
Example:
|
| 1597 |
|
|
@@ -1600,8 +1598,8 @@ class LlavaOnevision2ForConditionalGeneration(LlavaOnevision2PreTrainedModel, Ge
|
|
| 1600 |
>>> import requests
|
| 1601 |
>>> from transformers import AutoProcessor, LlavaOnevision2ForConditionalGeneration
|
| 1602 |
|
| 1603 |
-
>>> model = LlavaOnevision2ForConditionalGeneration.from_pretrained("
|
| 1604 |
-
>>> processor = AutoProcessor.from_pretrained("
|
| 1605 |
|
| 1606 |
>>> messages = [
|
| 1607 |
{
|
|
|
|
| 61 |
):
|
| 62 |
"""Train-parity 3D M-RoPE positions.
|
| 63 |
|
| 64 |
+
Verbatim port of the TRAINING-side ``_rope_from_patch_positions`` (training code) so eval positions match what the model saw in
|
|
|
|
| 65 |
training exactly — including its convention that the text position cursor
|
| 66 |
does NOT skip past a visual's span (text after an image resumes at the
|
| 67 |
pre-image counter). Do NOT "fix" that here; train/infer parity wins.
|
|
|
|
| 772 |
|
| 773 |
class LlavaOnevision2VisionPretrainedModel(LlavaOnevision2PreTrainedModel):
|
| 774 |
"""
|
| 775 |
+
LLaVA-OneVision-2.5 Vision Model.
|
| 776 |
|
| 777 |
This vision model is designed to work with Qwen2VL-style image processing:
|
| 778 |
- Receives pre-processed patches in 3x3 block spatial order
|
|
|
|
| 1084 |
grid_thw: Optional[torch.Tensor] = None,
|
| 1085 |
patch_positions: Optional[torch.Tensor] = None,
|
| 1086 |
) -> dict:
|
| 1087 |
+
"""Stage-by-stage forward used by consistency tests against the training implementation.
|
| 1088 |
|
| 1089 |
Mirrors the production ``forward`` exactly EXCEPT that ``patch_positions``
|
| 1090 |
passed in are assumed to be in row-major (t, h, w) order — matching the
|
| 1091 |
+
training-side ``forward_debug`` test contract,
|
| 1092 |
which also receives row-major positions and applies a block-layout
|
| 1093 |
reorder. Production ``forward`` skips that reorder because production
|
| 1094 |
``patch_positions`` come pre-arranged in block layout from the
|
| 1095 |
Qwen2VL processor (`build_patch_positions`).
|
| 1096 |
|
| 1097 |
+
Layer-wise outputs match the schema produced by the training-side
|
| 1098 |
+
``forward_debug``.
|
| 1099 |
"""
|
| 1100 |
output: dict[str, torch.Tensor] = {}
|
| 1101 |
|
|
|
|
| 1580 |
`position_ids`, this tensor is not affected by padding.
|
| 1581 |
|
| 1582 |
Note (P0 native-video alias):
|
| 1583 |
+
The companion ``LlavaOnevision2Processor.__call__(videos=...)`` does NOT
|
| 1584 |
pass ``pixel_values_videos`` / ``video_grid_thw`` / ``second_per_grid_ts``
|
| 1585 |
to this forward. Instead it aliases the video patch tensor as
|
| 1586 |
``pixel_values=`` and ``image_grid_thw=``, so video inputs share the
|
|
|
|
| 1589 |
text tags emitted by the processor). The ``*_videos`` and
|
| 1590 |
``second_per_grid_ts`` kwargs are kept declared here only for API
|
| 1591 |
completeness and future use (e.g. 3D mRoPE / ``get_rope_index``); they
|
| 1592 |
+
are NOT consumed by the current OneVision encoder.
|
|
|
|
| 1593 |
|
| 1594 |
Example:
|
| 1595 |
|
|
|
|
| 1598 |
>>> import requests
|
| 1599 |
>>> from transformers import AutoProcessor, LlavaOnevision2ForConditionalGeneration
|
| 1600 |
|
| 1601 |
+
>>> model = LlavaOnevision2ForConditionalGeneration.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True)
|
| 1602 |
+
>>> processor = AutoProcessor.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True)
|
| 1603 |
|
| 1604 |
>>> messages = [
|
| 1605 |
{
|