README.md

---
license: other
license_name: dots-ocr-license
license_link: https://huggingface.co/davanstrien/dots.ocr-1.5/blob/main/dots.ocr-1.5%20LICENSE%20AGREEMENT
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- image-to-text
- ocr
- document-parse
- layout
- table
- formula
- custom_code
language:
- en
- zh
- multilingual
---

> **Unofficial mirror.** This is a copy of [dots.ocr-1.5 from ModelScope](https://modelscope.cn/models/rednote-hilab/dots.ocr-1.5), uploaded to Hugging Face for easier access. All credit goes to the original authors at **rednote-hilab (Xiaohongshu)**. The original v1 model is at [rednote-hilab/dots.ocr](https://huggingface.co/rednote-hilab/dots.ocr) on HF. If the authors publish an official HF release of v1.5, please use that instead.
>
> Source: [ModelScope](https://modelscope.cn/models/rednote-hilab/dots.ocr-1.5) | [GitHub](https://github.com/rednote-hilab/dots.ocr)

# dots.ocr-1.5: Recognize Any Human Scripts and Symbols

A **3B-parameter** multimodal OCR model (1.2B vision encoder + 1.7B language model) from rednote-hilab. Designed for universal accessibility, it can recognize virtually any human script and achieves SOTA performance in multilingual document parsing among models of comparable size.

## Key Capabilities

1. **Multilingual Document Parsing** — SOTA on standard benchmarks among specialized OCR models, particularly strong on multilingual documents
2. **Structured Graphics to SVG** — Converts charts, diagrams, chemical formulas, and logos directly into SVG code
3. **Web Screen Parsing & Scene Text Spotting** — Handles web screenshots and scene text
4. **Object Grounding & Counting** — General vision tasks beyond pure OCR
5. **General OCR & Visual QA** — DocVQA 91.85, ChartQA 83.2, OCRBench 86.0

## Quick Start with UV Scripts

Process any HF dataset with a single command using [uv-scripts/ocr](https://huggingface.co/datasets/uv-scripts/ocr):

```bash
# Basic OCR
hf jobs uv run --flavor l4x1 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-ocr-1.5.py \
    your-input-dataset your-output-dataset \
    --model davanstrien/dots.ocr-1.5

# Layout analysis with bounding boxes
hf jobs uv run --flavor l4x1 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/ocr/raw/main/dots-ocr-1.5.py \
    your-input-dataset your-output-dataset \
    --model davanstrien/dots.ocr-1.5 \
    --prompt-mode layout-all
```

## Benchmarks

### Document Parsing (Elo Score)

| Model | olmOCR-Bench | OmniDocBench v1.5 | XDocParse |
|-------|:---:|:---:|:---:|
| GLM-OCR | 859.9 | 937.5 | 742.1 |
| PaddleOCR-VL-1.5 | 873.6 | 965.6 | 797.6 |
| HuanyuanOCR | 978.9 | 974.4 | 895.9 |
| dots.ocr | 1027.4 | 994.7 | 1133.4 |
| **dots.ocr-1.5** | **1089.0** | **1025.8** | **1157.1** |
| Gemini 3 Pro | 1171.2 | 1102.1 | 1273.9 |

### olmOCR-bench (detailed)

| Model | ArXiv | Old scans math | Tables | Overall |
|-------|:---:|:---:|:---:|:---:|
| olmOCR v0.4.0 | 83.0 | 82.3 | 84.9 | 82.4±1.1 |
| Chandra OCR 0.1.0 | 82.2 | 80.3 | 88.0 | 83.1±0.9 |
| **dots.ocr-1.5** | **85.9** | **85.5** | **90.7** | **83.9±0.9** |

### General Vision Tasks

| DocVQA | ChartQA | OCRBench | AI2D | CharXiv Descriptive | RefCOCO |
|:---:|:---:|:---:|:---:|:---:|:---:|
| 91.85 | 83.2 | 86.0 | 82.16 | 77.4 | 80.03 |

## Usage

### vLLM (recommended)

**Important:** When using `llm.chat()`, you must pass `chat_template_content_format="string"`. The model's tokenizer chat template expects string content, not OpenAI-format lists. Without this, the model produces empty output.

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="davanstrien/dots.ocr-1.5",
    trust_remote_code=True,
    max_model_len=24000,
    gpu_memory_utilization=0.9,
)

sampling_params = SamplingParams(temperature=0.1, top_p=0.9, max_tokens=24000)

messages = [{
    "role": "user",
    "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
        {"type": "text", "text": "Extract the text content from this image."},
    ],
}]

outputs = llm.chat(
    [messages],
    sampling_params,
    chat_template_content_format="string",  # Required!
)
print(outputs[0].outputs[0].text)
```

### vLLM Server

```bash
vllm serve davanstrien/dots.ocr-1.5 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9 \
    --chat-template-content-format string \
    --trust-remote-code
```

### Transformers

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
from qwen_vl_utils import process_vision_info

model = AutoModelForCausalLM.from_pretrained(
    "davanstrien/dots.ocr-1.5",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("davanstrien/dots.ocr-1.5", trust_remote_code=True)

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "document.jpg"},
        {"type": "text", "text": "Extract the text content from this image."},
    ],
}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt").to("cuda")

generated_ids = model.generate(**inputs, max_new_tokens=24000)
output = processor.batch_decode(
    [out[len(inp):] for inp, out in zip(inputs.input_ids, generated_ids)],
    skip_special_tokens=True,
)[0]
print(output)
```

## Prompt Modes

| Mode | Description | Output |
|------|-------------|--------|
| `ocr` | Text extraction (default) | Markdown |
| `layout-all` | Layout + bboxes + categories + text | JSON |
| `layout-only` | Layout + bboxes + categories (no text) | JSON |
| `web-parsing` | Webpage layout analysis | JSON |
| `scene-spotting` | Scene text detection | Text |
| `grounding-ocr` | Text from bounding box region | Text |
| `general` | Free-form (custom prompt) | Varies |

### Bbox Coordinate System (layout modes)

Bounding boxes are in the **resized image coordinate space**, not original image coordinates. The model uses `Qwen2VLImageProcessor` which resizes images so that `width × height ≤ 11,289,600` pixels, with dimensions rounded to multiples of 28.

To map bboxes back to original coordinates:

```python
import math

def smart_resize(height, width, factor=28, min_pixels=3136, max_pixels=11289600):
    h_bar = max(factor, round(height / factor) * factor)
    w_bar = max(factor, round(width / factor) * factor)
    if h_bar * w_bar > max_pixels:
        beta = math.sqrt((height * width) / max_pixels)
        h_bar = math.floor(height / beta / factor) * factor
        w_bar = math.floor(width / beta / factor) * factor
    elif h_bar * w_bar < min_pixels:
        beta = math.sqrt(min_pixels / (height * width))
        h_bar = math.ceil(height * beta / factor) * factor
        w_bar = math.ceil(width * beta / factor) * factor
    return h_bar, w_bar

resized_h, resized_w = smart_resize(orig_h, orig_w)
scale_x, scale_y = orig_w / resized_w, orig_h / resized_h
# orig_x = bbox_x * scale_x, orig_y = bbox_y * scale_y
```

## Model Details

- **Architecture:** DotsOCRForCausalLM (custom code, `trust_remote_code=True` required)
- **Parameters:** 3B total (1.2B vision encoder, 1.7B language model)
- **Precision:** BF16
- **Max context:** 131,072 tokens
- **Vision:** Patch size 14, spatial merge size 2, flash_attention_2
- **Languages:** English, Chinese (simplified + traditional), multilingual (Tibetan, Kannada, Russian, Dutch, and more)

## Limitations

- Complex table and formula extraction remains challenging for the compact 3B architecture
- SVG parsing for pictures needs further robustness improvements
- Occasional parsing failures on edge cases

## License

This model is released under the [dots.ocr License Agreement](dots.ocr-1.5%20LICENSE%20AGREEMENT), which is based on the MIT License with supplementary terms covering responsible use, attribution, and data governance. Per the license: *"If Licensee distributes modified weights or fine-tuned models based on the Model Materials, Licensee must prominently display the following statement: 'Built with dots.ocr.'"*

## Citation

```bibtex
@misc{dots_ocr_1_5,
  title={dots.ocr-1.5: Recognize Any Human Scripts and Symbols},
  author={rednote-hilab},
  year={2025},
  url={https://github.com/rednote-hilab/dots.ocr}
}
```

Built with dots.ocr.