Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,220 @@
+# MonkeyOCRv2 Detection
+
+MonkeyOCRv2 Detection integrates the visual encoder from
+[MonkeyOCRv2-AS](https://huggingface.co/zenosai/MonkeyOCRv2-AS) (ViTAEv2-S,
+21M parameters) into **DBNet** and **PSENet** scene text detectors via
+[MMOCR](https://github.com/open-mmlab/mmocr). The four-stage ViTAEv2 features
+(strides 4/8/16/32) are exposed as `res2`–`res5` and fed to the standard FPNC /
+FPNF necks, so no change to the detection heads is required.
+
+Training and evaluation follow the official MMOCR protocols on Total-Text,
+CTW1500, and ICDAR2015.
+
+## Models and Results
+
+Replacing the ImageNet-pretrained ResNet-50 with the MonkeyOCRv2-AS encoder
+consistently improves F-score over both the baseline and the
+[oCLIP](https://github.com/alkan25/oclip)-pretrained backbone. `*` marks
+results reproduced by us with MMOCR.
+
+### Total-Text
+
+| Method | P | R | F |
+| --- | ---: | ---: | ---: |
+| DBNet* (ResNet-50) | 82.6 | 78.4 | 80.4 |
+| DBNet + oCLIP | 85.1 | 81.7 | 83.4 |
+| **DBNet + MonkeyOCRv2** | **87.7** | 80.1 | **83.7** |
+
+### CTW1500
+
+| Method | P | R | F |
+| --- | ---: | ---: | ---: |
+| PSENet* (ResNet-50) | 80.1 | 82.7 | 81.4 |
+| PSENet + oCLIP | 82.1 | 85.5 | 83.8 |
+| **PSENet + MonkeyOCRv2** | **88.3** | 82.0 | **85.1** |
+
+### ICDAR2015
+
+| Method | P | R | F |
+| --- | ---: | ---: | ---: |
+| PSENet* (ResNet-50) | 84.0 | 76.2 | 79.9 |
+| PSENet + oCLIP | 87.3 | 82.6 | 84.9 |
+| **PSENet + MonkeyOCRv2** | **90.4** | 80.3 | **85.0** |
+| DBNet* (ResNet-50) | 88.8 | 81.5 | 85.0 |
+| DBNet + oCLIP | 90.9 | 84.1 | 87.4 |
+| **DBNet + MonkeyOCRv2** | **91.2** | **86.0** | **88.5** |
+
+### Checkpoints
+
+Download the checkpoints from
+[HB16888/MonkeyOCRv2_det](https://huggingface.co/HB16888/MonkeyOCRv2_det)
+(HuggingFace) or
+[WangXinhan/MonkeyOCRv2_det](https://modelscope.cn/models/WangXinhan/MonkeyOCRv2_det)
+(ModelScope):
+
+```bash
+# HuggingFace
+hf download HB16888/MonkeyOCRv2_det --include "*.pth" --local-dir ./model_weight
+# ModelScope
+modelscope download --model WangXinhan/MonkeyOCRv2_det --local_dir ./model_weight
+```
+
+Each checkpoint is the best epoch on the test set, i.e. exactly the row
+reported in the tables above.
+
+| Checkpoint | Method | Dataset | Epoch | Config |
+| --- | --- | --- | ---: | --- |
+| dbnet_r50_totaltext.pth | DBNet baseline | Total-Text | 580 | configs/textdet/dbnet/dbnet_resnet50_1200e_totaltext_2gpu.py |
+| dbnet_r50-oclip_totaltext.pth | DBNet + oCLIP | Total-Text | 740 | configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_totaltext_2gpu.py |
+| dbnet_mkv2vitae_totaltext.pth | DBNet + MonkeyOCRv2 | Total-Text | 1000 | configs/textdet/dbnet/dbnet_mkv2vitae_1200e_totaltext_2gpu_adamw.py |
+| psenet_r50_ctw1500.pth | PSENet baseline | CTW1500 | 280 | configs/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500_2gpu.py |
+| psenet_r50-oclip_ctw1500.pth | PSENet + oCLIP | CTW1500 | 280 | configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_ctw1500_2gpu.py |
+| psenet_mkv2vitae_ctw1500.pth | PSENet + MonkeyOCRv2 | CTW1500 | 120 | configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_ctw1500_4gpu_adamw.py |
+| psenet_r50_icdar2015.pth | PSENet baseline | ICDAR2015 | 400 | configs/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015_2gpu.py |
+| psenet_r50-oclip_icdar2015.pth | PSENet + oCLIP | ICDAR2015 | 520 | configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_icdar2015_2gpu.py |
+| psenet_mkv2vitae_icdar2015.pth | PSENet + MonkeyOCRv2 | ICDAR2015 | 160 | configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_icdar2015_4gpu_adamw.py |
+| dbnet_r50_icdar2015.pth | DBNet baseline | ICDAR2015 | 980 | configs/textdet/dbnet/dbnet_resnet50_1200e_icdar2015_2gpu.py |
+| dbnet_r50-oclip_icdar2015.pth | DBNet + oCLIP | ICDAR2015 | 1100 | configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015_2gpu.py |
+| dbnet_mkv2vitae_icdar2015.pth | DBNet + MonkeyOCRv2 | ICDAR2015 | 420 | configs/textdet/dbnet/dbnet_mkv2vitae_1200e_icdar2015_2gpu_adamw.py |
+
+## Environment
+
+The reproduced environment uses Python 3.11, PyTorch 2.9.0, CUDA 12.8,
+MMEngine 0.10.7, MMCV 2.0.1, MMOCR 1.0.1, and Transformers 4.57. Baselines
+and oCLIP models were trained on 2 GPUs; the MonkeyOCRv2 PSENet models on 4
+GPUs (NVIDIA GeForce RTX 3090).
+
+## Installation
+
+This directory is an add-on on top of the official MMOCR v1.0.1. Run:
+
+```bash
+bash install.sh            # clones MMOCR v1.0.1 into ./mmocr and patches it
+```
+
+The add-on contains:
+
+- `mmocr/models/common/backbones/monkeyocr_v2_vitae.py` — MMEngine wrapper
+  that loads the HuggingFace MonkeyOCRv2-AS encoder (via `transformers`
+  `AutoModel`, `trust_remote_code=True`) and exposes its four stages as
+  `res2`–`res5`. Image normalization (ViTAE mean/std) and patch splitting are
+  performed inside the backbone; the data preprocessor must use
+  `mean=None`, `std=None`, `bgr_to_rgb=True`, `pad_size_divisor=32`.
+- `mmocr/models/textdet/postprocessors/pse_postprocessor.py` — one-line fix
+  for the PSE kernel-growing condition (`or` → `and` when filtering by
+  `score_threshold`), which improves the PSENet postprocessing quality.
+- `dataset_zoo/ctw1500/textdet.py` — updated MD5 sums and archive layouts
+  for the CTW1500 download sources (the official Box/CloudStor archives were
+  re-uploaded and no longer match the upstream checksums).
+- `tools/test.py` — accepts `--out` when `test_evaluator` is a plain dict.
+- `tools/dataset_converters/prepare_all_datasets.sh` — prepares ICDAR2015,
+  CTW1500, and Total-Text in one go.
+- `configs/textdet/{dbnet,psenet}/` — the 12 training configs listed above.
+
+## Pretrained Backbone
+
+Download the MonkeyOCRv2-AS visual encoder before training or evaluation:
+
+```bash
+hf download zenosai/MonkeyOCRv2-AS --local-dir ./pretrained/monkeyocrv2_as
+```
+
+The configs reference `pretrained/monkeyocrv2_as` relative to the MMOCR
+working directory. The weights are identical to the ones used to produce the
+results above (verified by MD5).
+
+## Datasets
+
+```bash
+cd mmocr
+bash tools/dataset_converters/prepare_all_datasets.sh
+```
+
+This prepares `data/icdar2015`, `data/ctw1500`, and `data/totaltext` in the
+MMOCR format. Note that some official CTW1500 download links are currently
+unreliable; we also provide the prepared CTW1500 in MMOCR format at
+[HB16888/CTW1500](https://huggingface.co/datasets/HB16888/CTW1500)
+(HuggingFace) and
+[WangXinhan/CTW1500](https://modelscope.cn/datasets/WangXinhan/CTW1500)
+(ModelScope):
+
+```bash
+# HuggingFace
+hf download HB16888/CTW1500 --repo-type dataset --local-dir ./ctw1500_dl
+unzip ctw1500_dl/ctw1500_mmocr.zip -d data/
+# ModelScope
+modelscope download --dataset WangXinhan/CTW1500 --local_dir ./ctw1500_dl
+unzip ctw1500_dl/ctw1500_mmocr.zip -d data/
+```
+
+Directory layout:
+
+```text
+data/ctw1500/
+├── textdet_imgs/
+│   ├── train/    # 1000 images
+│   └── test/     # 500 images
+├── textdet_train.json
+└── textdet_test.json
+```
+
+## Training
+
+All commands are run from the MMOCR root directory. Baselines and oCLIP
+models use 2 GPUs (batch size 16 in total); MonkeyOCRv2 PSENet models use 4
+GPUs.
+
+```bash
+# ---------- DBNet on Total-Text ----------
+bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50_1200e_totaltext_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_totaltext_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/dbnet/dbnet_mkv2vitae_1200e_totaltext_2gpu_adamw.py 2
+
+# ---------- PSENet on CTW1500 ----------
+bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50_fpnf_600e_ctw1500_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_ctw1500_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_ctw1500_4gpu_adamw.py 4
+
+# ---------- PSENet on ICDAR2015 ----------
+bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50_fpnf_600e_icdar2015_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/psenet/psenet_resnet50-oclip_fpnf_600e_icdar2015_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_icdar2015_4gpu_adamw.py 4
+
+# ---------- DBNet on ICDAR2015 ----------
+bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50_1200e_icdar2015_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/dbnet/dbnet_resnet50-oclip_1200e_icdar2015_2gpu.py 2
+bash tools/dist_train.sh configs/textdet/dbnet/dbnet_mkv2vitae_1200e_icdar2015_2gpu_adamw.py 2
+```
+
+Notes:
+
+- The oCLIP backbone weights (`resnet50-oclip-7ba0c533.pth`) are fetched
+  automatically from `download.openmmlab.com` via `init_cfg`.
+- The MonkeyOCRv2 configs load the encoder from
+  `pretrained/monkeyocrv2_as` (see [Pretrained Backbone](#pretrained-backbone)).
+- Every config sets `randomness = dict(seed=42)`; results are selected by the
+  best test-set hmean over the training run, evaluated every 20 epochs.
+
+## Evaluation
+
+```bash
+# single GPU
+python tools/test.py \
+  configs/textdet/dbnet/dbnet_mkv2vitae_1200e_totaltext_2gpu_adamw.py \
+  model_weight/dbnet_mkv2vitae_totaltext.pth
+
+# multi GPU
+bash tools/dist_test.sh \
+  configs/textdet/psenet/psenet_mkv2vitae_fpnf_600e_ctw1500_4gpu_adamw.py \
+  model_weight/psenet_mkv2vitae_ctw1500.pth 1
+```
+
+The evaluation prints `precision / recall / hmean` with the
+`HmeanIOUMetric`, matching the tables above.
+
+## Acknowledgements
+
+This project builds on [MMOCR](https://github.com/open-mmlab/mmocr),
+[DBNet](https://arxiv.org/abs/1911.08947),
+[PSENet](https://arxiv.org/abs/1806.02559), and
+[oCLIP](https://arxiv.org/abs/2303.06995).