metadata
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models
OCRVerse is a holistic OCR method developed in an end-to-end manner that enables unified text-centric OCR (e.g., documents, books) and vision-centric OCR (e.g., charts, web pages, scientific plots). It is proposed to bridge the gap between recognizing text elements and identifying visual elements from information-dense images.
- Paper: OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models
- Repository: DocTron-hub/OCRVerse
Usage Example
The following is a simple example of how to use OCRVerse for document parsing tasks.
Installation
pip install "transformers>=4.57.0"
Inference
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
import torch
# Load model
model_path = 'DocTron/OCRVerse'
model = Qwen3VLForConditionalGeneration.from_pretrained(
model_path,
dtype="auto",
device_map="cuda",
trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
# Prepare input with image and text
image_path = "YOUR_IMAGE_PATH"
# We recommend using the following prompt to better performance, since it is used throughout the training process.
prompt = "Extract the main content from the document in the image, keeping the original structure. Convert all formulas to LaTeX and all tables to HTML."
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image_path},
{"type": "text", "text": prompt},
]
}
]
# Preparation for inference
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
)
inputs = inputs.to(model.device)
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.tokenizer.batch_decode(
generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])
Citation
@misc{zhong2026ocrverse,
title={OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models},
author={Yufeng Zhong and Lei Chen and Xuanle Zhao and Wenkang Han and Liming Zheng and Jing Huang and Deyang Jiang and Yilin Cao and Lin Ma and Zhixiong Zeng},
year={2026},
eprint={2601.21639},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2601.21639},
}