--- license: apache-2.0 tags: - ocr - text-detection - scene-text-detection - mmocr - monkeyocr v2 --- # MonkeyOCRv2 Detection This repository provides the text detection experiments from the [MonkeyOCRv2 paper](https://arxiv.org/abs/2607.11562). The visual encoder from [MonkeyOCRv2-AS](https://huggingface.co/zenosai/MonkeyOCRv2-AS) (ViTAEv2-S, 21M parameters) is integrated into **DBNet** and **PSENet** scene text detectors via [MMOCR](https://github.com/open-mmlab/mmocr). The four-stage ViTAEv2 features (strides 4/8/16/32) are exposed as `res2`–`res5` and fed to the standard FPNC / FPNF necks, so no change to the detection heads is required. Training and evaluation follow the official MMOCR protocols on Total-Text, CTW1500, and ICDAR2015. ## Models and Results For each detector, three visual backbones are compared under identical settings: the original ImageNet-pretrained encoder, the text-specific [oCLIP](https://github.com/bytedance/oclip) encoder, and MonkeyOCRv2. MonkeyOCRv2 consistently improves F-score across all datasets and detector architectures. ### Total-Text | Method | P | R | F | | ----------------------- | -------: | ---: | -------: | | DBNet (ResNet-50) | 82.6 | 78.4 | 80.4 | | DBNet + oCLIP | 85.1 | 81.7 | 83.4 | | **DBNet + MonkeyOCRv2** | **87.7** | 80.1 | **83.7** | ### CTW1500 | Method | P | R | F | | ------------------------ | -------: | ---: | -------: | | PSENet (ResNet-50) | 80.1 | 82.7 | 81.4 | | PSENet + oCLIP | 82.1 | 85.5 | 83.8 | | **PSENet + MonkeyOCRv2** | **88.3** | 82.0 | **85.1** | ### ICDAR2015 | Method | P | R | F | | ------------------------ | -------: | -------: | -------: | | PSENet (ResNet-50) | 84.0 | 76.2 | 79.9 | | PSENet + oCLIP | 87.3 | 82.6 | 84.9 | | **PSENet + MonkeyOCRv2** | **90.4** | 80.3 | **85.0** | | DBNet (ResNet-50) | 88.8 | 81.5 | 85.0 | | DBNet + oCLIP | 90.9 | 84.1 | 87.4 | | **DBNet + MonkeyOCRv2** | **91.2** | **86.0** | **88.5** | ### Checkpoints Download the checkpoints from [HB16888/MonkeyOCRv2\_det](https://huggingface.co/HB16888/MonkeyOCRv2_det) (HuggingFace) or [WangXinhan/MonkeyOCRv2\_det](https://modelscope.cn/models/WangXinhan/MonkeyOCRv2_det) (ModelScope): ```bash # HuggingFace hf download HB16888/MonkeyOCRv2_det --include "*.pth" --local-dir ./model_weight # ModelScope modelscope download --model WangXinhan/MonkeyOCRv2_det --local_dir ./model_weight ``` ## Environment The reproduced environment uses Python 3.11, PyTorch 2.9.0, CUDA 12.8, MMEngine 0.10.7, MMCV 2.0.1, MMOCR 1.0.1, and Transformers 4.57. Baselines and oCLIP models were trained on 2 GPUs; the MonkeyOCRv2 PSENet models on 4 GPUs (NVIDIA GeForce RTX 3090). ## Installation This directory is an add-on on top of the official MMOCR v1.0.1. Run: ```bash bash install.sh # clones MMOCR v1.0.1 into ./mmocr and patches it ``` ## Pretrained Backbone Download the MonkeyOCRv2-AS visual encoder before training or evaluation: ```bash hf download zenosai/MonkeyOCRv2-AS --local-dir ./pretrained/monkeyocrv2_as ``` ## Datasets ```bash cd mmocr bash tools/dataset_converters/prepare_all_datasets.sh ``` This prepares `data/icdar2015`, `data/ctw1500`, and `data/totaltext` in the MMOCR format. Note that some official CTW1500 download links are currently unreliable; we also provide the prepared CTW1500 in MMOCR format at [HB16888/CTW1500](https://huggingface.co/datasets/HB16888/CTW1500) (HuggingFace) and [WangXinhan/CTW1500](https://modelscope.cn/datasets/WangXinhan/CTW1500) (ModelScope): ```bash # HuggingFace hf download HB16888/CTW1500 --repo-type dataset --local-dir ./ctw1500_dl unzip ctw1500_dl/ctw1500_mmocr.zip -d data/ # ModelScope modelscope download --dataset WangXinhan/CTW1500 --local_dir ./ctw1500_dl unzip ctw1500_dl/ctw1500_mmocr.zip -d data/ ``` Directory layout: ```text data/ctw1500/ ├── textdet_imgs/ │ ├── train/ # 1000 images │ └── test/ # 500 images ├── textdet_train.json └── textdet_test.json ``` ## Training All commands are run from the MMOCR root directory. Baselines and oCLIP models use 2 GPUs (batch size 16 in total); MonkeyOCRv2 PSENet models use 4 GPUs. ```bash # ---------- DBNet on Total-Text ---------- bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50_1200e_totaltext_2gpu.py 2 bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_totaltext_2gpu.py 2 bash tools/dist_train.sh configs/textdet/dbnet/dbnet_mkv2vitae_1200e_totaltext_2gpu_adamw.py 2 # ---------- PSENet on CTW1500 ---------- bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500_2gpu.py 2 bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_ctw1500_2gpu.py 2 bash tools/dist_train.sh configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_ctw1500_4gpu_adamw.py 4 # ---------- PSENet on ICDAR2015 ---------- bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015_2gpu.py 2 bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_icdar2015_2gpu.py 2 bash tools/dist_train.sh configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_icdar2015_4gpu_adamw.py 4 # ---------- DBNet on ICDAR2015 ---------- bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50_1200e_icdar2015_2gpu.py 2 bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015_2gpu.py 2 bash tools/dist_train.sh configs/textdet/dbnet/dbnet_mkv2vitae_1200e_icdar2015_2gpu_adamw.py 2 ``` ## Evaluation ```bash # single GPU python tools/test.py \ configs/textdet/dbnet/dbnet_mkv2vitae_1200e_totaltext_2gpu_adamw.py \ model_weight/dbnet_mkv2vitae_totaltext.pth # multi GPU bash tools/dist_test.sh \ configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_ctw1500_4gpu_adamw.py \ model_weight/psenet_mkv2vitae_ctw1500.pth 4 ``` The evaluation prints `precision / recall / hmean` with the `HmeanIOUMetric`, matching the tables above. ## Acknowledgements This project builds on [MMOCR](https://github.com/open-mmlab/mmocr), [DBNet](https://github.com/MhLiao/DB), [PSENet](https://github.com/whai362/PSENet), [oCLIP](https://github.com/bytedance/oclip), and [MonkeyOCRv2](https://github.com/Yuliang-Liu/MonkeyOCRv2).