File size: 2,842 Bytes
9c2d7ad |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 |
---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- ocr
- vision-language-model
- document-understanding
---
# OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models
OCRVerse is a holistic OCR method that enables unified text-centric OCR (extracting text from documents like books and magazines) and vision-centric OCR (identifying visual elements from information-dense sources like charts, web pages, and scientific plots) in an end-to-end manner.
- **Paper:** [OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models](https://huggingface.co/papers/2601.21639)
- **GitHub Repository:** [DocTron-hub/OCRVerse](https://github.com/DocTron-hub/OCRVerse)
## Usage Example
To use OCRVerse, please ensure you have the `transformers` library installed:
```shell
pip install "transformers>=4.57.0"
```
### Text-Centric Document Parsing
Below is a simple example of how to use OCRVerse for document parsing tasks.
```python
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
import torch
# Load model
model_path = 'DocTron/OCRVerse'
model = Qwen3VLForConditionalGeneration.from_pretrained(
model_path,
dtype="auto",
device_map="cuda",
trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
# Prepare input with image and text
image_path = "path/to/your/image.jpg"
# We recommend using the following prompt for better performance
prompt = "Extract the main content from the document in the image, keeping the original structure. Convert all formulas to LaTeX and all tables to HTML."
messages = [
{
"role": "user",
"content": [
{"type": "image", "image": image_path},
{"type": "text", "text": prompt},
]
}
]
# Preparation for inference
inputs = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors="pt"
)
inputs = inputs.to(model.device)
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
generated_ids = [
output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.tokenizer.batch_decode(
generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])
```
## Citation
If you find this project useful, please cite our paper:
```bibtex
@article{zhong2026ocrverse,
title={OCRVerse: Towards Holistic OCR in End-to-End Vision-Language Models},
author={Yufeng Zhong and Lei Chen and Xuanle Zhao and Wenkang Han and Liming Zheng and Jing Huang and Deyang Jiang and Yilin Cao and Lin Ma and Zhixiong Zeng},
journal={arXiv preprint arXiv:2601.21639},
year={2026}
}
``` |