---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- ocr
- document-parsing
- document-understanding
- multimodal
- markdown
- tables
- formulas
- vllm
---
# OvisOCR
## Introduction
We introduce **OvisOCR**, a lightweight end-to-end multimodal large language model (MLLM) tailored for high-fidelity document parsing. Unlike conventional **Crop-OCR-Merge** systems that rely on layout detection, localized cropping, specialized recognizers, and heuristic merging, OvisOCR directly maps full-page document images into structured Markdown outputs.
OvisOCR is designed for information-dense documents containing natural language text, tables, mathematical formulas, figures, and complex layouts. It preserves fine-grained textual fidelity while maintaining global document structure and human reading order. With only **1.3B parameters**, OvisOCR achieves outstanding overall performance on OmniDocBench v1.5.
## Key Features
- **Strictly End-to-End Document Parsing**
OvisOCR directly maps full-page visual signals to structured Markdown without localized slicing, layout-dependent recognition, or post-hoc merging. This streamlined paradigm reduces error propagation and improves global serialization consistency.
- **Synergistic Data Construction**
Our data construction pipeline builds high-quality supervision by combining the strengths of a specialized OCR engine and a general-purpose MLLM. The specialized perceiver supplies dense local evidence, while the general reasoner checks for hallucinations, content completeness, table validity, formula syntax, and logical reading order.
- **Multi-Granularity Alignment**
OvisOCR uses element-aware optimization for heterogeneous document constituents. Text, tables, and formulas are optimized with tailored reward signals, including edit-distance-based text fidelity, TEDS-based table similarity, and CDM-based formula visual correctness.
- **Strong Document Parsing Capability with Compact Scale**
With only 1.3B parameters, OvisOCR achieves outstanding performance on OmniDocBench v1.5, surpassing strong specialized parsers, large general MLLMs, and traditional pipeline tools.
## Inference
```bash
pip install "vllm==0.18.1" pillow
```
```python
from PIL import Image
from vllm import LLM, SamplingParams
class OvisOCRParser:
def __init__(self, model_name_or_path: str):
self.model = LLM(
model=model_name_or_path,
tensor_parallel_size=1,
trust_remote_code=True,
gpu_memory_utilization=0.8,
)
prompt = 'Extract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <' + 'img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: . Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing.'
self.prompt = self.model.get_tokenizer().apply_chat_template(
[{"role": "user", "content": f"\n{prompt}"}],
tokenize=False,
add_generation_prompt=True,
enable_thinking=False
)
self.sampling_params = SamplingParams(
max_tokens=16384,
temperature=0.0,
)
def _clean_truncated_repeats(
self,
text: str,
min_text_len: int = 8000,
max_period: int = 200,
min_period: int = 1,
min_repeat_chars: int = 100,
min_repeat_times: int = 5
) -> str:
n = len(text)
if n < min_text_len:
return text
max_period = min(max_period, n - 1)
for unit_len in range(min_period, max_period + 1):
if text[n - 1] != text[n - 1 - unit_len]:
continue
match_len = 1
idx = n - 2
while idx >= unit_len and text[idx] == text[idx - unit_len]:
match_len += 1
idx -= 1
total_len = match_len + unit_len
repeat_times = total_len // unit_len
tail_len = total_len % unit_len
if repeat_times >= min_repeat_times and total_len >= min_repeat_chars:
return text[: n - total_len + unit_len] + text[n - tail_len:]
return text
def parse(self, images: list[Image.Image], filter_imgtags: bool = True) -> list[str]:
vllm_inputs = [
{
"prompt": self.prompt,
"multi_modal_data": {"image": image},
"mm_processor_kwargs": {
"images_kwargs": {
"min_pixels": 448 * 448,
"max_pixels": 2880 * 2880,
}
}
}
for image in images
]
outputs = self.model.generate(vllm_inputs, self.sampling_params)
markdowns = []
for output in outputs:
text = output.outputs[0].text.strip()
if filter_imgtags:
text = "\n\n".join(
block
for block in text.split("\n\n")
if not block.strip().startswith('