File size: 15,207 Bytes

---
license: apache-2.0
language:
- en
- zh
pipeline_tag: image-text-to-text
tags:
- ocr
- document-parse
---

## Flame-OCR

<p align="center">
  <img src="./flameocr_logo.png" style="width: 500px;" alt="Logo">
</p>

<p align="center">
    <a href="https://www.modelscope.cn/models/anyforge/Flame-OCR" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/%E9%AD%94%E6%90%AD-ModelScope-blue"></a>
    <a href="https://huggingface.co/anyforge/Flame-OCR" target="_blank"><img src="https://img.shields.io/badge/%F0%9F%A4%97-HuggingFace-blue"></a>
    <br>
        <a href="https://www.modelscope.cn/models/anyforge/Flame-OCR-gguf" target="_blank"><img alt="Static Badge" src="https://img.shields.io/badge/%E9%AD%94%E6%90%AD-ModelScope(GGUF)-blue"></a>
        <a href="https://huggingface.co/anyforge/Flame-OCR-gguf" target="_blank"><img src="https://img.shields.io/badge/%F0%9F%A4%97-HuggingFace(GGUF)-blue"></a>
    <br>
        👉 <a href="https://github.com/anyforge/flame-ocr" target="_blank">Flame-OCR Github</a>
</p>


<p align="center">
  <a href="https://huggingface.co/anyforge/Flame-OCR/blob/main/README.md">English</a> | <a href="https://huggingface.co/anyforge/Flame-OCR/blob/main/README_zh.md">简体中文</a>
</p>

**Flame-OCR: A high-speed, high-precision OCR model inspired by the purifying power of flame.**

**Drawing inspiration from the purifying essence of the 'Yi Huo' (Heavenly Flames) in Battle Through the Heavens.**

We have released **Flame-OCR**, a lightweight (0.8B parameters) end-to-end document page parsing model. Given a document page image, it generates a Markdown representation in natural reading order, encompassing text, formulas, tables, and visual regions.

**Flame-OCR** is built upon the OvisOCR2 (Qwen3.5-0.8B) model and has undergone high-quality datasets post-training, specifically through expansion and enhancement training. This significantly improves its performance on invoices, medical records, physical examination reports, financial documents, tables, formulas, handwriting, and ancient texts. While maintaining a compact size and preserving its original capabilities, the model achieves robust document parsing performance.

**Flame-OCR** achieved a comprehensive score of 94.46 on OmniDocBench v1.6.

<p align="center">
  <img src="./performance.png" alt="Performance" width="100%" />
</p>

<p align="center">
  <img src="./output.gif" width="100%" alt="Long-horizon OCR demo" />
</p>

## Inference

- **Based on Transformers**

```bash
pip install "transformers>=5.0.0" pillow
```

```python
import time
import threading
import torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from PIL import Image

MIN_PIXELS = 448 * 448
MAX_PIXELS = 2880 * 2880
MAX_NEW_TOKENS = 16384
STREAM_MIN_CHARS = 64
STREAM_MAX_INTERVAL = 0.25


model_path = '/path/to/model or model_id'

processor = AutoProcessor.from_pretrained(
    model_path,
    min_pixels=MIN_PIXELS,
    max_pixels=MAX_PIXELS,
)
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    model_path,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation=None  # or Flash Attention
).eval()

prompt = (
    "\nExtract all readable content from the image in natural human reading order "
    "and output the result as a single Markdown document. For charts or images, "
    'represent them using an HTML image tag: <img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, '
    "where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). "
    "Format formulas as LaTeX. Format tables as HTML: <table>...</table>. "
    "Transcribe all other text as standard Markdown. Preserve the original text "
    "without translation or paraphrasing."
)

def clean_truncated_repeats(
    text: str,
    min_text_len: int = 8000,
    max_period: int = 200,
    min_period: int = 1,
    min_repeat_chars: int = 100,
    min_repeat_times: int = 5,
) -> str:
    """Remove a repeated suffix created when generation reaches its token ceiling."""
    n = len(text)
    if n < min_text_len:
        return text

    max_period = min(max_period, n - 1)
    for unit_len in range(min_period, max_period + 1):
        if text[n - 1] != text[n - 1 - unit_len]:
            continue
        match_len = 1
        idx = n - 2
        while idx >= unit_len and text[idx] == text[idx - unit_len]:
            match_len += 1
            idx -= 1
        total_len = match_len + unit_len
        repeat_times = total_len // unit_len
        tail_len = total_len % unit_len
        if repeat_times >= min_repeat_times and total_len >= min_repeat_chars:
            return text[: n - total_len + unit_len] + text[n - tail_len :]
    return text

def _model_inputs(page_image: Image.Image):
    if processor is None or model is None:
        raise RuntimeError("OvisOCR2 is not loaded.")

    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": page_image},
                {"type": "text", "text": prompt},
            ],
        }
    ]
    return processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_dict=True,
        return_tensors="pt",
        enable_thinking=False,
    ).to(model.device)


def generation_token_ids(active_processor) -> dict[str, int]:
    """Use tokenizer stop IDs; this checkpoint's config and tokenizer differ."""
    tokenizer = active_processor.tokenizer
    return {
        "eos_token_id": int(tokenizer.eos_token_id),
        "pad_token_id": int(tokenizer.pad_token_id),
    }

def infer_stream(page_image: Image.Image):
    from transformers import TextIteratorStreamer

    inputs = _model_inputs(page_image)
    streamer = TextIteratorStreamer(
        processor.tokenizer,
        skip_prompt=True,
        skip_special_tokens=True,
        clean_up_tokenization_spaces=False,
    )
    errors: list[BaseException] = []

    def generate() -> None:
        try:
            with torch.inference_mode():
                model.generate(
                    **inputs,
                    streamer=streamer,
                    max_new_tokens=MAX_NEW_TOKENS,
                    do_sample=False,
                    temperature=None,
                    top_p=None,
                    top_k=None,
                    **generation_token_ids(processor),
                )
        except BaseException as error:
            errors.append(error)
            streamer.on_finalized_text("", stream_end=True)

    worker = threading.Thread(target=generate, name="ovisocr2-generate", daemon=True)
    worker.start()
    text = ""
    last_yielded = ""
    last_yield_time = time.monotonic()
    for fragment in streamer:
        text += fragment
        now = time.monotonic()
        if (
            len(text) - len(last_yielded) >= STREAM_MIN_CHARS
            or now - last_yield_time >= STREAM_MAX_INTERVAL
        ):
            yield text
            last_yielded = text
            last_yield_time = now

    worker.join()
    if errors:
        raise RuntimeError("Model generation failed.") from errors[0]
    final_text = clean_truncated_repeats(text.strip())
    if final_text and final_text != last_yielded:
        yield final_text

def infer(page_image: Image.Image):
    inputs = _model_inputs(page_image)
    outputs = model.generate(**inputs, max_new_tokens=MAX_NEW_TOKENS)
    result = processor.decode(outputs[0][inputs["input_ids"].shape[-1]:-1])
    return result
  

image = Image.open("/path/to/image")

## Inference
result = infer(image)
print(result)

## Stream Inference
for line in infer_stream(image):
    if line:
        print(line)
```

- **Based on vLLM**

```bash
pip install "vllm==0.22.1" pillow
```

```python
from PIL import Image
from vllm import LLM, SamplingParams


class CustomOCRParser:
    def __init__(self, model_name_or_path: str):
        self.model = LLM(
            model=model_name_or_path,
            tensor_parallel_size=1,
            gpu_memory_utilization=0.8,
            gdn_prefill_backend="triton"
        )

        prompt = '\nExtract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <' + 'img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: <table>...</table>. Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing.'
        self.prompt = self.model.get_tokenizer().apply_chat_template(
            [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}],
            tokenize=False,
            add_generation_prompt=True,
            enable_thinking=False
        )

        self.sampling_params = SamplingParams(
            max_tokens=16384,
            temperature=0.0
        )

    def _clean_truncated_repeats(
        self,
        text: str,
        min_text_len: int = 8000,
        max_period: int = 200,
        min_period: int = 1,
        min_repeat_chars: int = 100,
        min_repeat_times: int = 5
    ) -> str:
        n = len(text)
        if n < min_text_len:
            return text

        max_period = min(max_period, n - 1)
        for unit_len in range(min_period, max_period + 1):
            if text[n - 1] != text[n - 1 - unit_len]:
                continue

            match_len = 1
            idx = n - 2
            while idx >= unit_len and text[idx] == text[idx - unit_len]:
                match_len += 1
                idx -= 1

            total_len = match_len + unit_len
            repeat_times = total_len // unit_len
            tail_len = total_len % unit_len

            if repeat_times >= min_repeat_times and total_len >= min_repeat_chars:
                return text[: n - total_len + unit_len] + text[n - tail_len:]

        return text

    def parse(self, images: list[Image.Image], filter_imgtags: bool = False) -> list[str]:
        vllm_inputs = [
            {
                "prompt": self.prompt,
                "multi_modal_data": {"image": image},
                "mm_processor_kwargs": {
                    "images_kwargs": {
                        "min_pixels": 448 * 448,
                        "max_pixels": 2880 * 2880
                    }
                }
            }
            for image in images
        ]

        outputs = self.model.generate(vllm_inputs, self.sampling_params)

        markdowns = []
        for output in outputs:
            text = output.outputs[0].text.strip()
            if filter_imgtags:
                text = "\n\n".join(
                    block
                    for block in text.split("\n\n")
                    if not block.strip().startswith('<img src="images/bbox_')
                )
            markdowns.append(self._clean_truncated_repeats(text))

        return markdowns


if __name__ == "__main__":
    parser = CustomOCRParser("/path/to/model")
    images = [Image.open("/path/to/image"), Image.open("/path/to/image")]
    markdowns = parser.parse(images)
    print(markdowns[0])
```

- **vllm cli**

```bash

# use modelscope
export VLLM_USE_MODELSCOPE=true

# or model local path
vllm serve anyforge/Flame-OCR \
--served-model-name "flame-ocr" \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.8 \
--gdn-prefill-backend triton \
--max-model-len 16384 \
--max-num-seqs 1 \
--enable-chunked-prefill \
--max-num-batched-tokens 16384



import base64
from openai import OpenAI

def image_to_base64(image_path: str) -> str:
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

image_path = "/path/to/test.png"
base64_image = image_to_base64(image_path)

prompt = '\nExtract all readable content from the image in natural human reading order and output the result as a single Markdown document. For charts or images, represent them using an HTML image tag: <' + 'img src="images/bbox_{left}_{top}_{right}_{bottom}.jpg" />, where left, top, right, bottom are bounding box coordinates scaled to [0, 1000). Format formulas as LaTeX. Format tables as HTML: <table>...</table>. Transcribe all other text as standard Markdown. Preserve the original text without translation or paraphrasing.'

response = client.chat.completions.create(
    model="flame-ocr",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{base64_image}"
                    }
                },
                {
                    "type": "text",
                    "text": prompt
                }
            ]
        }
    ],
    max_tokens=8192,
    temperature=0.0,
    extra_body={
        "mm_processor_kwargs": {
            "images_kwargs": {
                "min_pixels": 448 * 448,
                "max_pixels": 2880 * 2880
            }
        }
    }
)

print(response.choices[0].message.content)

```

- **Extract HTML Image Tags for Visual Regions**

```python
import re
from pathlib import Path

from PIL import Image


BBOX_IMAGE_PATTERN = re.compile(
    r'<img src=' + r'"images/bbox_(\d+)_(\d+)_(\d+)_(\d+)\.jpg" />'
)


def save_renderable_markdown_with_visual_regions(
    markdown: str,
    page_image: Image.Image,
    output_dir: str,
) -> None:
    output_dir = Path(output_dir)
    images_dir = output_dir / "images"
    images_dir.mkdir(parents=True, exist_ok=True)

    width, height = page_image.size
    for left, top, right, bottom in BBOX_IMAGE_PATTERN.findall(markdown):
        x1 = max(0, min(width, round(int(left) * width / 1000)))
        y1 = max(0, min(height, round(int(top) * height / 1000)))
        x2 = max(0, min(width, round(int(right) * width / 1000)))
        y2 = max(0, min(height, round(int(bottom) * height / 1000)))
        if x2 <= x1 or y2 <= y1:
            continue

        crop_path = images_dir / f"bbox_{left}_{top}_{right}_{bottom}.jpg"
        page_image.crop((x1, y1, x2, y2)).convert("RGB").save(crop_path)

    (output_dir / "output.md").write_text(markdown, encoding="utf-8")


parser = CustomOCRParser("/path/to/model")
page_image = Image.open("test1.jpg")
markdown = parser.parse([page_image], filter_imgtags=False)[0]
save_renderable_markdown_with_visual_regions(markdown, page_image, "output")
```

## Buy me a coffee

<p align="left">
    <img src="./alipay.png" width="200">
    <img src="./wechatpay.png" width="200">
</p>

## Acknowledgements

We sincerely thank OvisOCR2, Qwen, and OmniDocBench for providing valuable code, model weights, and benchmark datasets. We also extend our gratitude to everyone who has contributed to this open-source project!

## Disclaimer

Due to the diversity and complexity of real-world documents, **Flame-OCR** may still produce erroneous or incomplete outputs. Please manually verify the results in critical application scenarios.