ReadME / ModelCard 优化
#8
by HwwwH - opened
README.md
CHANGED
|
@@ -16,7 +16,7 @@ tags:
|
|
| 16 |
|
| 17 |
<div align="center">
|
| 18 |
|
| 19 |
-
[GitHub](https://github.com/OpenBMB/MiniCPM-OCR) · [API](https://
|
| 20 |
|
| 21 |
</div>
|
| 22 |
|
|
@@ -39,8 +39,10 @@ and balances quality against speed on real-world captures.
|
|
| 39 |
information extraction (KIE) for downstream automation.
|
| 40 |
- 🛠️ **Easy to deploy and develop**: **vLLM, SGLang, llama.cpp, and Ollama from day one**, with
|
| 41 |
quantized variants and multi-chip builds. Supports further fine-tuning, and ships a pip SDK,
|
| 42 |
-
a full API, and agent workflows. Online demos: [
|
| 43 |
-
[
|
|
|
|
|
|
|
| 44 |
|
| 45 |
## SOTA quality at higher throughput
|
| 46 |
|
|
@@ -92,7 +94,8 @@ Left to right: original document, detected layout, and rendered Markdown.
|
|
| 92 |
|
| 93 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/minicpm-chinese-examples-3-column.gif" alt="Chinese document animation: original, detected layout, and rendered Markdown." width="100%"></p>
|
| 94 |
|
| 95 |
-
|
|
|
|
| 96 |
|
| 97 |
Original documents and parsed results, side by side.
|
| 98 |
|
|
@@ -104,6 +107,8 @@ Original documents and parsed results, side by side.
|
|
| 104 |
|
| 105 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/minicpm-ocr-chinese-cases.png" alt="Chinese examples comparing inputs and results: handwritten notes, complex formulas, newspapers, financial statements, stamped documents, illustrations, and key information extraction." width="100%"></p>
|
| 106 |
|
|
|
|
|
|
|
| 107 |
## Model architecture
|
| 108 |
|
| 109 |
| Item | Description |
|
|
@@ -121,77 +126,17 @@ deployment options, and benchmark breakdowns.
|
|
| 121 |
|
| 122 |
## Quick start
|
| 123 |
|
| 124 |
-
###
|
| 125 |
-
|
| 126 |
-
The SDK parses whole documents; the model recognises regions:
|
| 127 |
-
|
| 128 |
-
1. **Read** — accept PDF/PNG/JPG and rasterise PDF pages.
|
| 129 |
-
2. **Layout** — PP-DocLayoutV3 detects regions and reading order, then crops them.
|
| 130 |
-
3. **Recognise** — regions are sent to the MiniCPM-OCR 1.0 service concurrently.
|
| 131 |
-
4. **Assemble** — results are merged into Markdown in reading order; tables become HTML,
|
| 132 |
-
formulas stay LaTeX.
|
| 133 |
|
| 134 |
-
|
| 135 |
-
See the [SDK tutorial](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/minicpmvocr_tutorial_en.md).
|
| 136 |
-
|
| 137 |
-
### vLLM
|
| 138 |
-
|
| 139 |
-
```bash
|
| 140 |
-
vllm serve openbmb/MiniCPM-OCR-1.0 \
|
| 141 |
-
--served-model-name MiniCPM-OCR \
|
| 142 |
-
--trust-remote-code \
|
| 143 |
-
--max-model-len 40960
|
| 144 |
-
```
|
| 145 |
|
| 146 |
-
|
| 147 |
-
[vLLM deploy guide](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/vllm_deploy_en.md).
|
| 148 |
|
| 149 |
-
#
|
| 150 |
|
| 151 |
-
|
| 152 |
-
| :--- | :--- |
|
| 153 |
-
| vLLM | [API service and single-region recognition](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/vllm_deploy_en.md) |
|
| 154 |
-
| SGLang | [Deployment guide](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/sglang_deploy.md) |
|
| 155 |
-
| llama.cpp | [GGUF deployment guide](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/llamacpp_deploy.md) |
|
| 156 |
-
| Ollama | [SDK service connection and backend configuration](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/minicpmvocr_api_en.md) |
|
| 157 |
-
| Transformers | [Service deployment](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.md#43-transformers-serve) |
|
| 158 |
-
| FlagOS (multi-chip) | [Deployment guide](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/flagos_deploy_en.md) |
|
| 159 |
-
|
| 160 |
-
### Prompts
|
| 161 |
-
|
| 162 |
-
The prompt selects the task and must match the region you cropped:
|
| 163 |
-
|
| 164 |
-
| Region | Prompt | Output |
|
| 165 |
-
| :--- | :--- | :--- |
|
| 166 |
-
| Text block / paragraph / title / page | `Text Recognition:` | plain text |
|
| 167 |
-
| Table | `Table Recognition:` | HTML (via SDK post-processing) |
|
| 168 |
-
| Formula | `Formula Recognition:` | LaTeX |
|
| 169 |
-
| Inline formula | `Text Recognition:` | LaTeX |
|
| 170 |
-
| Seal | `Seal Recognition:` | seal text |
|
| 171 |
-
| Chart | `Chart to Table:` | table |
|
| 172 |
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
### Key information extraction (KIE)
|
| 176 |
-
|
| 177 |
-
MiniCPM-OCR 1.0 can pull specified fields straight out of an image — for example the store name
|
| 178 |
-
and amount on a receipt. Send the image with a field-extraction prompt and ask for a JSON object;
|
| 179 |
-
you do not need to produce page Markdown first.
|
| 180 |
-
|
| 181 |
-
Use the [image recognition API example](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/vllm_deploy_en.md#22-image-recognition) and
|
| 182 |
-
replace `Text Recognition:` with:
|
| 183 |
-
|
| 184 |
-
```text
|
| 185 |
-
Extract the "store_name", "amount". Format the output as a JSON object with keys: "store_name", "amount". If an item is not in the image, use "###".
|
| 186 |
-
```
|
| 187 |
-
|
| 188 |
-
Replace both field lists with your own field names (Chinese or English). The response is
|
| 189 |
-
model-generated text: parse the JSON and check fields and values before using it downstream.
|
| 190 |
-
|
| 191 |
-
### Agent skill
|
| 192 |
-
|
| 193 |
-
Have a coding agent follow the skill to set up the environment, download the model, and parse
|
| 194 |
-
a first document:
|
| 195 |
|
| 196 |
```text
|
| 197 |
Read and execute https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.md
|
|
@@ -203,6 +148,58 @@ Read and execute https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.
|
|
| 203 |
[ms-swift](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/swift_minicpm_ocr_french_en.md) — data preparation, full-parameter SFT with
|
| 204 |
the multimodal projector frozen, training, and evaluation (French handwritten text lines).
|
| 205 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 206 |
## Related checkpoints
|
| 207 |
|
| 208 |
| Format | Hugging Face | ModelScope |
|
|
|
|
| 16 |
|
| 17 |
<div align="center">
|
| 18 |
|
| 19 |
+
[GitHub](https://github.com/OpenBMB/MiniCPM-OCR) · [API](https://platform.modelbest.ai/console/docs/api/document-parse) · [Official Demo](https://platform.modelbest.ai/console/playground/document-parse) · [HF Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-OCR) · [Usage](https://github.com/OpenBMB/MiniCPM-OCR#using-minicpm-ocr) · [DSpark](https://huggingface.co/openbmb/MiniCPM-OCR-1.0-DSpark) · [Skill](https://github.com/OpenBMB/MiniCPM-OCR#skill)
|
| 20 |
|
| 21 |
</div>
|
| 22 |
|
|
|
|
| 39 |
information extraction (KIE) for downstream automation.
|
| 40 |
- 🛠️ **Easy to deploy and develop**: **vLLM, SGLang, llama.cpp, and Ollama from day one**, with
|
| 41 |
quantized variants and multi-chip builds. Supports further fine-tuning, and ships a pip SDK,
|
| 42 |
+
a full API, and agent workflows. Online demos: [HF Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-OCR) ·
|
| 43 |
+
[CN](http://82.157.64.212:10111/playground/document-parse) ·
|
| 44 |
+
[Official CN](https://platform.modelbest.cn/console/playground/document-parse) ·
|
| 45 |
+
[Official Global](https://platform.modelbest.ai/console/playground/document-parse).
|
| 46 |
|
| 47 |
## SOTA quality at higher throughput
|
| 48 |
|
|
|
|
| 94 |
|
| 95 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/minicpm-chinese-examples-3-column.gif" alt="Chinese document animation: original, detected layout, and rendered Markdown." width="100%"></p>
|
| 96 |
|
| 97 |
+
<details>
|
| 98 |
+
<summary><b>More examples</b></summary>
|
| 99 |
|
| 100 |
Original documents and parsed results, side by side.
|
| 101 |
|
|
|
|
| 107 |
|
| 108 |
<p align="center"><img src="https://raw.githubusercontent.com/OpenBMB/MiniCPM-OCR/main/assets/minicpm-ocr/minicpm-ocr-chinese-cases.png" alt="Chinese examples comparing inputs and results: handwritten notes, complex formulas, newspapers, financial statements, stamped documents, illustrations, and key information extraction." width="100%"></p>
|
| 109 |
|
| 110 |
+
</details>
|
| 111 |
+
|
| 112 |
## Model architecture
|
| 113 |
|
| 114 |
| Item | Description |
|
|
|
|
| 126 |
|
| 127 |
## Quick start
|
| 128 |
|
| 129 |
+
### Online Demo
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
+
To quickly try text-centric document understanding and structured information extraction, we recommend [HF Spaces](https://huggingface.co/spaces/openbmb/MiniCPM-OCR), [CN](http://82.157.64.212:10111/playground/document-parse), [Official CN](https://platform.modelbest.cn/console/playground/document-parse), [Official Global](https://platform.modelbest.ai/console/playground/document-parse) / [API](https://platform.modelbest.ai/console/docs/api/document-parse).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
|
| 133 |
+
### Local deployment
|
|
|
|
| 134 |
|
| 135 |
+
For local deployment of MiniCPM-OCR, we recommend our SDK ([tutorial](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/minicpmvocr_tutorial_en.md)) with [DSpark](https://huggingface.co/openbmb/MiniCPM-OCR-1.0-DSpark) enabled by default. For document parsing task, It reads a PDF or image, detects layout with PP-DocLayoutV3, recognises each region, and assembles Markdown in reading order. Our SDK supports different inference backends, including [vLLM](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/vllm_deploy_en.md), [SGLang](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/sglang_deploy.md), [llama.cpp](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/llamacpp_deploy.md), [Ollama](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/minicpmvocr_api_en.md), [Transformers](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.md#43-transformers-serve), and [FlagOS](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/flagos_deploy_en.md).
|
| 136 |
|
| 137 |
+
### Agent Skill
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 138 |
|
| 139 |
+
You can also have a coding agent follow the [Agent Skill](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.md) to set up the environment, download the model, and parse a first document.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
|
| 141 |
```text
|
| 142 |
Read and execute https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/SKILL_en.md
|
|
|
|
| 148 |
[ms-swift](https://github.com/OpenBMB/MiniCPM-OCR/blob/main/docs/swift_minicpm_ocr_french_en.md) — data preparation, full-parameter SFT with
|
| 149 |
the multimodal projector frozen, training, and evaluation (French handwritten text lines).
|
| 150 |
|
| 151 |
+
### Recognition guide
|
| 152 |
+
|
| 153 |
+
<table>
|
| 154 |
+
<thead>
|
| 155 |
+
<tr>
|
| 156 |
+
<th>Task</th>
|
| 157 |
+
<th>Region</th>
|
| 158 |
+
<th>Prompt</th>
|
| 159 |
+
<th>Output</th>
|
| 160 |
+
</tr>
|
| 161 |
+
</thead>
|
| 162 |
+
<tbody>
|
| 163 |
+
<tr>
|
| 164 |
+
<td rowspan="6">Text-centric document understanding</td>
|
| 165 |
+
<td>Text block / Paragraph / Title / Page</td>
|
| 166 |
+
<td><code>Text Recognition:</code></td>
|
| 167 |
+
<td>Plain text</td>
|
| 168 |
+
</tr>
|
| 169 |
+
<tr>
|
| 170 |
+
<td>Table</td>
|
| 171 |
+
<td><code>Table Recognition:</code></td>
|
| 172 |
+
<td>Table in HTML format(via SDK post-processing)</td>
|
| 173 |
+
</tr>
|
| 174 |
+
<tr>
|
| 175 |
+
<td>Formula</td>
|
| 176 |
+
<td><code>Formula Recognition:</code></td>
|
| 177 |
+
<td>LaTeX</td>
|
| 178 |
+
</tr>
|
| 179 |
+
<tr>
|
| 180 |
+
<td>Inline formula</td>
|
| 181 |
+
<td><code>Text Recognition:</code></td>
|
| 182 |
+
<td>LaTeX</td>
|
| 183 |
+
</tr>
|
| 184 |
+
<tr>
|
| 185 |
+
<td>Seal</td>
|
| 186 |
+
<td><code>Seal Recognition:</code></td>
|
| 187 |
+
<td>Plain text</td>
|
| 188 |
+
</tr>
|
| 189 |
+
<tr>
|
| 190 |
+
<td>Chart</td>
|
| 191 |
+
<td><code>Chart to Table:</code></td>
|
| 192 |
+
<td>Table in HTML format(via SDK post-processing)</td>
|
| 193 |
+
</tr>
|
| 194 |
+
<tr>
|
| 195 |
+
<td>Structured information extraction</td>
|
| 196 |
+
<td>Specified fields</td>
|
| 197 |
+
<td><code>Extract the "field", .... Format the output as a JSON object with keys: "field", .... If an item is not in the image, use "###".</code></td>
|
| 198 |
+
<td>JSON</td>
|
| 199 |
+
</tr>
|
| 200 |
+
</tbody>
|
| 201 |
+
</table>
|
| 202 |
+
|
| 203 |
## Related checkpoints
|
| 204 |
|
| 205 |
| Format | Hugging Face | ModelScope |
|