File size: 6,465 Bytes

---
license: apache-2.0
tags:
- ocr
- text-detection
- scene-text-detection
- mmocr
- monkeyocr v2
---

# MonkeyOCRv2 Detection

This repository provides the text detection experiments from the
[MonkeyOCRv2 paper](https://arxiv.org/abs/2607.11562). The visual encoder from
[MonkeyOCRv2-AS](https://huggingface.co/zenosai/MonkeyOCRv2-AS) (ViTAEv2-S,
21M parameters) is integrated into **DBNet** and **PSENet** scene text
detectors via [MMOCR](https://github.com/open-mmlab/mmocr). The four-stage
ViTAEv2 features (strides 4/8/16/32) are exposed as `res2`–`res5` and fed to
the standard FPNC / FPNF necks, so no change to the detection heads is
required.

Training and evaluation follow the official MMOCR protocols on Total-Text,
CTW1500, and ICDAR2015.

## Models and Results

For each detector, three visual backbones are compared under identical
settings: the original ImageNet-pretrained encoder, the text-specific
[oCLIP](https://github.com/bytedance/oclip) encoder, and MonkeyOCRv2.
MonkeyOCRv2 consistently improves F-score across all datasets and detector
architectures.

### Total-Text

| Method                  |        P |    R |        F |
| ----------------------- | -------: | ---: | -------: |
| DBNet (ResNet-50)       |     82.6 | 78.4 |     80.4 |
| DBNet + oCLIP           |     85.1 | 81.7 |     83.4 |
| **DBNet + MonkeyOCRv2** | **87.7** | 80.1 | **83.7** |

### CTW1500

| Method                   |        P |    R |        F |
| ------------------------ | -------: | ---: | -------: |
| PSENet (ResNet-50)       |     80.1 | 82.7 |     81.4 |
| PSENet + oCLIP           |     82.1 | 85.5 |     83.8 |
| **PSENet + MonkeyOCRv2** | **88.3** | 82.0 | **85.1** |

### ICDAR2015

| Method                   |        P |        R |        F |
| ------------------------ | -------: | -------: | -------: |
| PSENet (ResNet-50)       |     84.0 |     76.2 |     79.9 |
| PSENet + oCLIP           |     87.3 |     82.6 |     84.9 |
| **PSENet + MonkeyOCRv2** | **90.4** |     80.3 | **85.0** |
| DBNet (ResNet-50)        |     88.8 |     81.5 |     85.0 |
| DBNet + oCLIP            |     90.9 |     84.1 |     87.4 |
| **DBNet + MonkeyOCRv2**  | **91.2** | **86.0** | **88.5** |

### Checkpoints

Download the checkpoints from
[HB16888/MonkeyOCRv2\_det](https://huggingface.co/HB16888/MonkeyOCRv2_det)
(HuggingFace) or
[WangXinhan/MonkeyOCRv2\_det](https://modelscope.cn/models/WangXinhan/MonkeyOCRv2_det)
(ModelScope):

```bash
# HuggingFace
hf download HB16888/MonkeyOCRv2_det --include "*.pth" --local-dir ./model_weight
# ModelScope
modelscope download --model WangXinhan/MonkeyOCRv2_det --local_dir ./model_weight
```

## Environment

The reproduced environment uses Python 3.11, PyTorch 2.9.0, CUDA 12.8,
MMEngine 0.10.7, MMCV 2.0.1, MMOCR 1.0.1, and Transformers 4.57. Baselines
and oCLIP models were trained on 2 GPUs; the MonkeyOCRv2 PSENet models on 4
GPUs (NVIDIA GeForce RTX 3090).

## Installation

This directory is an add-on on top of the official MMOCR v1.0.1. Run:

```bash
bash install.sh            # clones MMOCR v1.0.1 into ./mmocr and patches it
```

## Pretrained Backbone

Download the MonkeyOCRv2-AS visual encoder before training or evaluation:

```bash
hf download zenosai/MonkeyOCRv2-AS --local-dir ./pretrained/monkeyocrv2_as
```

## Datasets

```bash
cd mmocr
bash tools/dataset_converters/prepare_all_datasets.sh
```

This prepares `data/icdar2015`, `data/ctw1500`, and `data/totaltext` in the
MMOCR format. Note that some official CTW1500 download links are currently
unreliable; we also provide the prepared CTW1500 in MMOCR format at
[HB16888/CTW1500](https://huggingface.co/datasets/HB16888/CTW1500)
(HuggingFace) and
[WangXinhan/CTW1500](https://modelscope.cn/datasets/WangXinhan/CTW1500)
(ModelScope):

```bash
# HuggingFace
hf download HB16888/CTW1500 --repo-type dataset --local-dir ./ctw1500_dl
unzip ctw1500_dl/ctw1500_mmocr.zip -d data/
# ModelScope
modelscope download --dataset WangXinhan/CTW1500 --local_dir ./ctw1500_dl
unzip ctw1500_dl/ctw1500_mmocr.zip -d data/
```

Directory layout:

```text
data/ctw1500/
├── textdet_imgs/
│   ├── train/    # 1000 images
│   └── test/     # 500 images
├── textdet_train.json
└── textdet_test.json
```

## Training

All commands are run from the MMOCR root directory. Baselines and oCLIP
models use 2 GPUs (batch size 16 in total); MonkeyOCRv2 PSENet models use 4
GPUs.

```bash
# ---------- DBNet on Total-Text ----------
bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50_1200e_totaltext_2gpu.py 2
bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_totaltext_2gpu.py 2
bash tools/dist_train.sh configs/textdet/dbnet/dbnet_mkv2vitae_1200e_totaltext_2gpu_adamw.py 2

# ---------- PSENet on CTW1500 ----------
bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500_2gpu.py 2
bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_ctw1500_2gpu.py 2
bash tools/dist_train.sh configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_ctw1500_4gpu_adamw.py 4

# ---------- PSENet on ICDAR2015 ----------
bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015_2gpu.py 2
bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_icdar2015_2gpu.py 2
bash tools/dist_train.sh configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_icdar2015_4gpu_adamw.py 4

# ---------- DBNet on ICDAR2015 ----------
bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50_1200e_icdar2015_2gpu.py 2
bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015_2gpu.py 2
bash tools/dist_train.sh configs/textdet/dbnet/dbnet_mkv2vitae_1200e_icdar2015_2gpu_adamw.py 2
```

## Evaluation

```bash
# single GPU
python tools/test.py \
  configs/textdet/dbnet/dbnet_mkv2vitae_1200e_totaltext_2gpu_adamw.py \
  model_weight/dbnet_mkv2vitae_totaltext.pth

# multi GPU
bash tools/dist_test.sh \
  configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_ctw1500_4gpu_adamw.py \
  model_weight/psenet_mkv2vitae_ctw1500.pth 4
```

The evaluation prints `precision / recall / hmean` with the
`HmeanIOUMetric`, matching the tables above.

## Acknowledgements

This project builds on [MMOCR](https://github.com/open-mmlab/mmocr),
[DBNet](https://github.com/MhLiao/DB),
[PSENet](https://github.com/whai362/PSENet),
[oCLIP](https://github.com/bytedance/oclip), and
[MonkeyOCRv2](https://github.com/Yuliang-Liu/MonkeyOCRv2).