asdfasdfqrqwer commited on
Commit
ccbaadb
·
1 Parent(s): 38987a6

Update Jev evaluation, screening, model-selection, and examples copy.

Browse files
Files changed (1) hide show
  1. README.md +16 -10
README.md CHANGED
@@ -55,11 +55,10 @@ supply the original region image, the candidate text, and the region `category`
55
  (`text` / `formula` / `table`) — the service renders internally. A service without a renderer
56
  requires you to supply the rendered image as well.
57
 
58
- ### Evaluation
59
 
60
- [JevOCRBench](https://huggingface.co/datasets/openbmb/JevOCRBench) is a block-level benchmark:
61
- each sample is a region image, one candidate, and its rendering, labelled correct/incorrect by
62
- humans; the metric is AUROC (shown ×100).
63
 
64
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-benchmark-score-table.png" alt="Overall, text, formula, and table AUROC on JevOCRBench, highlighting MiniCPM-OCR Jev." width="100%"></p>
65
 
@@ -72,17 +71,21 @@ humans; the metric is AUROC (shown ×100).
72
  Same 200 samples drawn at random from JevOCRBench; four models parse them independently, every
73
  candidate is hand-labelled, then scored by MiniCPM-OCR Jev. Maximum gap **1.5 points**.
74
 
75
- ### Use case 1 — auto-accept and reduce manual review
76
 
77
- Pick a threshold for your scenario and target accuracy; regions above it are stored
78
- automatically, the rest go to review.
 
 
79
 
80
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/review-free-en.png" alt="Accuracy versus review-free rate for legal documents, financial receipts, and complex documents." width="100%"></p>
81
 
82
- ### Use case 2 — pick the better model per region
83
 
84
- One layout, several OCR models parse each region; MiniCPM-OCR Jev scores the candidates, the
85
- highest-scoring one is selected, and results are combined in reading order.
 
 
86
 
87
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/multi-models-improvements-en.png" alt="Parsing-quality gains from MiniCPM-OCR Jev scoring for MiniCPM-OCR 1.0, Xiaomi OCR, and PaddleOCR-VL 1.6." width="100%"></p>
88
 
@@ -91,6 +94,9 @@ existing candidates, so quality is bounded by their coverage and correctness.
91
 
92
  ### Examples
93
 
 
 
 
94
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-case-en.png" alt="Three examples for text, formulas, and tables: original, candidate 1 and candidate 2, with MiniCPM-OCR Jev scores." width="100%"></p>
95
 
96
  ### Choosing a review threshold
 
55
  (`text` / `formula` / `table`) — the service renders internally. A service without a renderer
56
  requires you to supply the rendered image as well.
57
 
58
+ ### Evaluation Results
59
 
60
+ The table below reports overall and per-category AUROC (×100) on JevOCRBench. Higher values
61
+ indicate better discrimination between correct and incorrect parsing results.
 
62
 
63
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-benchmark-score-table.png" alt="Overall, text, formula, and table AUROC on JevOCRBench, highlighting MiniCPM-OCR Jev." width="100%"></p>
64
 
 
71
  Same 200 samples drawn at random from JevOCRBench; four models parse them independently, every
72
  candidate is hand-labelled, then scored by MiniCPM-OCR Jev. Maximum gap **1.5 points**.
73
 
74
+ ### Automatic screening to reduce human review
75
 
76
+ Score the parsed result for each region and choose a threshold using samples from your
77
+ application. Results above the threshold proceed automatically; the rest go to human review.
78
+ A higher threshold typically reduces the number of results that pass automatically, so monitor
79
+ both the accuracy of automatically accepted results and the volume of human review.
80
 
81
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/review-free-en.png" alt="Accuracy versus review-free rate for legal documents, financial receipts, and complex documents." width="100%"></p>
82
 
83
+ ### Selecting across models to improve parsing quality
84
 
85
+ Use the same layout segmentation to have several OCR models parse each region. MiniCPM-OCR Jev
86
+ scores the candidates for the same region, selects the highest-scoring candidate, and combines
87
+ the selected results in reading order. If all candidates fall below the application threshold,
88
+ the region can still be sent for human review.
89
 
90
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/multi-models-improvements-en.png" alt="Parsing-quality gains from MiniCPM-OCR Jev scoring for MiniCPM-OCR 1.0, Xiaomi OCR, and PaddleOCR-VL 1.6." width="100%"></p>
91
 
 
94
 
95
  ### Examples
96
 
97
+ The figure below compares the original image with candidates 1 and 2, showing how MiniCPM-OCR
98
+ Jev scores text, formula, and table candidates.
99
+
100
  <p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/jev-case-en.png" alt="Three examples for text, formulas, and tables: original, candidate 1 and candidate 2, with MiniCPM-OCR Jev scores." width="100%"></p>
101
 
102
  ### Choosing a review threshold