Commit ·
ccbaadb
1
Parent(s): 38987a6
Update Jev evaluation, screening, model-selection, and examples copy.
Browse files
README.md
CHANGED
|
@@ -55,11 +55,10 @@ supply the original region image, the candidate text, and the region `category`
|
|
| 55 |
(`text` / `formula` / `table`) — the service renders internally. A service without a renderer
|
| 56 |
requires you to supply the rendered image as well.
|
| 57 |
|
| 58 |
-
### Evaluation
|
| 59 |
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
humans; the metric is AUROC (shown ×100).
|
| 63 |
|
| 64 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-benchmark-score-table.png" alt="Overall, text, formula, and table AUROC on JevOCRBench, highlighting MiniCPM-OCR Jev." width="100%"></p>
|
| 65 |
|
|
@@ -72,17 +71,21 @@ humans; the metric is AUROC (shown ×100).
|
|
| 72 |
Same 200 samples drawn at random from JevOCRBench; four models parse them independently, every
|
| 73 |
candidate is hand-labelled, then scored by MiniCPM-OCR Jev. Maximum gap **1.5 points**.
|
| 74 |
|
| 75 |
-
###
|
| 76 |
|
| 77 |
-
|
| 78 |
-
automatically
|
|
|
|
|
|
|
| 79 |
|
| 80 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/review-free-en.png" alt="Accuracy versus review-free rate for legal documents, financial receipts, and complex documents." width="100%"></p>
|
| 81 |
|
| 82 |
-
###
|
| 83 |
|
| 84 |
-
|
| 85 |
-
|
|
|
|
|
|
|
| 86 |
|
| 87 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/multi-models-improvements-en.png" alt="Parsing-quality gains from MiniCPM-OCR Jev scoring for MiniCPM-OCR 1.0, Xiaomi OCR, and PaddleOCR-VL 1.6." width="100%"></p>
|
| 88 |
|
|
@@ -91,6 +94,9 @@ existing candidates, so quality is bounded by their coverage and correctness.
|
|
| 91 |
|
| 92 |
### Examples
|
| 93 |
|
|
|
|
|
|
|
|
|
|
| 94 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-case-en.png" alt="Three examples for text, formulas, and tables: original, candidate 1 and candidate 2, with MiniCPM-OCR Jev scores." width="100%"></p>
|
| 95 |
|
| 96 |
### Choosing a review threshold
|
|
|
|
| 55 |
(`text` / `formula` / `table`) — the service renders internally. A service without a renderer
|
| 56 |
requires you to supply the rendered image as well.
|
| 57 |
|
| 58 |
+
### Evaluation Results
|
| 59 |
|
| 60 |
+
The table below reports overall and per-category AUROC (×100) on JevOCRBench. Higher values
|
| 61 |
+
indicate better discrimination between correct and incorrect parsing results.
|
|
|
|
| 62 |
|
| 63 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-benchmark-score-table.png" alt="Overall, text, formula, and table AUROC on JevOCRBench, highlighting MiniCPM-OCR Jev." width="100%"></p>
|
| 64 |
|
|
|
|
| 71 |
Same 200 samples drawn at random from JevOCRBench; four models parse them independently, every
|
| 72 |
candidate is hand-labelled, then scored by MiniCPM-OCR Jev. Maximum gap **1.5 points**.
|
| 73 |
|
| 74 |
+
### Automatic screening to reduce human review
|
| 75 |
|
| 76 |
+
Score the parsed result for each region and choose a threshold using samples from your
|
| 77 |
+
application. Results above the threshold proceed automatically; the rest go to human review.
|
| 78 |
+
A higher threshold typically reduces the number of results that pass automatically, so monitor
|
| 79 |
+
both the accuracy of automatically accepted results and the volume of human review.
|
| 80 |
|
| 81 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/review-free-en.png" alt="Accuracy versus review-free rate for legal documents, financial receipts, and complex documents." width="100%"></p>
|
| 82 |
|
| 83 |
+
### Selecting across models to improve parsing quality
|
| 84 |
|
| 85 |
+
Use the same layout segmentation to have several OCR models parse each region. MiniCPM-OCR Jev
|
| 86 |
+
scores the candidates for the same region, selects the highest-scoring candidate, and combines
|
| 87 |
+
the selected results in reading order. If all candidates fall below the application threshold,
|
| 88 |
+
the region can still be sent for human review.
|
| 89 |
|
| 90 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/multi-models-improvements-en.png" alt="Parsing-quality gains from MiniCPM-OCR Jev scoring for MiniCPM-OCR 1.0, Xiaomi OCR, and PaddleOCR-VL 1.6." width="100%"></p>
|
| 91 |
|
|
|
|
| 94 |
|
| 95 |
### Examples
|
| 96 |
|
| 97 |
+
The figure below compares the original image with candidates 1 and 2, showing how MiniCPM-OCR
|
| 98 |
+
Jev scores text, formula, and table candidates.
|
| 99 |
+
|
| 100 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-case-en.png" alt="Three examples for text, formulas, and tables: original, candidate 1 and candidate 2, with MiniCPM-OCR Jev scores." width="100%"></p>
|
| 101 |
|
| 102 |
### Choosing a review threshold
|