rabbit commited on
Commit ·
e241053
1
Parent(s): f4708fd
update
Browse files
README.md
CHANGED
|
@@ -3,11 +3,8 @@
|
|
| 3 |
A Vision-Language Model (VLM) specialized in **document understanding and information extraction**, supporting both **unstructured information** and **structured data (tables)** from document images.
|
| 4 |
|
| 5 |
This model is optimized for production usage via **vLLM serving** with an **OpenAI-compatible API**.
|
| 6 |
-
|
| 7 |
---
|
| 8 |
-
|
| 9 |
## 🚀 Features
|
| 10 |
-
|
| 11 |
- Vision-Language Model for document images
|
| 12 |
- Extracts **unstructured key–value information**
|
| 13 |
- Extracts **structured table data**, including **column-wise extraction**
|
|
@@ -15,11 +12,8 @@ This model is optimized for production usage via **vLLM serving** with an **Open
|
|
| 15 |
- Strict output formatting (no hallucination)
|
| 16 |
- Compatible with **vLLM OpenAI-style API**
|
| 17 |
- Prompting optimized for **Vietnamese instructions**
|
| 18 |
-
|
| 19 |
---
|
| 20 |
-
|
| 21 |
## 📌 Supported Data Types
|
| 22 |
-
|
| 23 |
### 1. Unstructured Information
|
| 24 |
Extract specific fields defined by the user, such as:
|
| 25 |
- Invoice number
|
|
@@ -28,59 +22,46 @@ Extract specific fields defined by the user, such as:
|
|
| 28 |
- Address
|
| 29 |
- Total amount
|
| 30 |
- Custom document attributes
|
| 31 |
-
|
| 32 |
---
|
| 33 |
-
|
| 34 |
### 2. Structured Table Data
|
| 35 |
Designed for extracting **individual columns** from tables, especially product tables.
|
| 36 |
-
|
| 37 |
Capabilities:
|
| 38 |
- Column-level extraction
|
| 39 |
- Ignore non-product rows
|
| 40 |
- Markdown-formatted output
|
| 41 |
- Clean and deterministic structure
|
| 42 |
-
|
| 43 |
---
|
| 44 |
-
|
| 45 |
## 🔧 Deployment (vLLM)
|
| 46 |
-
|
| 47 |
This model is intended to be deployed using **vLLM** with an OpenAI-compatible interface.
|
| 48 |
-
|
| 49 |
Example:
|
| 50 |
-
|
| 51 |
```bash
|
| 52 |
vllm serve <model-path-or-name> \
|
| 53 |
--served-model-name document-vlm \
|
| 54 |
--port 8000
|
| 55 |
```
|
| 56 |
-
|
| 57 |
-
|
| 58 |
Prompt Construction (Python)
|
| 59 |
```bash
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
-
|
| 64 |
-
-
|
| 65 |
-
- không
|
| 66 |
-
|
| 67 |
-
"""
|
| 68 |
```
|
| 69 |
-
|
| 70 |
Expected Output
|
| 71 |
```bash
|
| 72 |
1. Số hóa đơn: INV-001
|
| 73 |
2. Ngày phát hành: 12/03/2024
|
| 74 |
3. Tổng tiền: 1.250.000 VND
|
| 75 |
```
|
| 76 |
-
|
| 77 |
If data cannot be extracted:
|
| 78 |
```bash
|
| 79 |
tôi không thể tìm thấy dữ liệu này
|
| 80 |
```
|
| 81 |
-
|
| 82 |
-
🔹 Structured Table (Column-wise) Extraction Prompt
|
| 83 |
-
|
| 84 |
Used to extract data for a specific column from a product table.
|
| 85 |
|
| 86 |
```bash
|
|
@@ -92,9 +73,8 @@ prompt = (
|
|
| 92 |
f"Yêu cầu tiêu đề cột là |{col_name}|.\n"
|
| 93 |
)
|
| 94 |
```
|
| 95 |
-
|
| 96 |
-
```bash
|
| 97 |
Expected Output
|
|
|
|
| 98 |
|Tên sản phẩm|
|
| 99 |
|------------|
|
| 100 |
|Sản phẩm A|
|
|
|
|
| 3 |
A Vision-Language Model (VLM) specialized in **document understanding and information extraction**, supporting both **unstructured information** and **structured data (tables)** from document images.
|
| 4 |
|
| 5 |
This model is optimized for production usage via **vLLM serving** with an **OpenAI-compatible API**.
|
|
|
|
| 6 |
---
|
|
|
|
| 7 |
## 🚀 Features
|
|
|
|
| 8 |
- Vision-Language Model for document images
|
| 9 |
- Extracts **unstructured key–value information**
|
| 10 |
- Extracts **structured table data**, including **column-wise extraction**
|
|
|
|
| 12 |
- Strict output formatting (no hallucination)
|
| 13 |
- Compatible with **vLLM OpenAI-style API**
|
| 14 |
- Prompting optimized for **Vietnamese instructions**
|
|
|
|
| 15 |
---
|
|
|
|
| 16 |
## 📌 Supported Data Types
|
|
|
|
| 17 |
### 1. Unstructured Information
|
| 18 |
Extract specific fields defined by the user, such as:
|
| 19 |
- Invoice number
|
|
|
|
| 22 |
- Address
|
| 23 |
- Total amount
|
| 24 |
- Custom document attributes
|
|
|
|
| 25 |
---
|
|
|
|
| 26 |
### 2. Structured Table Data
|
| 27 |
Designed for extracting **individual columns** from tables, especially product tables.
|
|
|
|
| 28 |
Capabilities:
|
| 29 |
- Column-level extraction
|
| 30 |
- Ignore non-product rows
|
| 31 |
- Markdown-formatted output
|
| 32 |
- Clean and deterministic structure
|
|
|
|
| 33 |
---
|
|
|
|
| 34 |
## 🔧 Deployment (vLLM)
|
|
|
|
| 35 |
This model is intended to be deployed using **vLLM** with an OpenAI-compatible interface.
|
|
|
|
| 36 |
Example:
|
|
|
|
| 37 |
```bash
|
| 38 |
vllm serve <model-path-or-name> \
|
| 39 |
--served-model-name document-vlm \
|
| 40 |
--port 8000
|
| 41 |
```
|
| 42 |
+
## Prompt Usage
|
| 43 |
+
Unstructured Data Extraction Prompt
|
| 44 |
Prompt Construction (Python)
|
| 45 |
```bash
|
| 46 |
+
prompt = f"""QUERY Trích xuất thông tin: {field_names}.
|
| 47 |
+
INSTRUCTION:
|
| 48 |
+
- Bắt buộc dữ liệu trả về theo format <index>. <key>:<value>, trong đó <index> là số thứ tự (1, 2, 3, 4, ...)
|
| 49 |
+
- key lấy chính xác từ trong QUERY của tôi
|
| 50 |
+
- không tự bịa dữ liệu và coi đó là điều hiển nhiên
|
| 51 |
+
- nếu không thể trích xuất thì hãy trả lời: tôi không thể tìm thấy dữ liệu này
|
| 52 |
+
"""
|
|
|
|
| 53 |
```
|
|
|
|
| 54 |
Expected Output
|
| 55 |
```bash
|
| 56 |
1. Số hóa đơn: INV-001
|
| 57 |
2. Ngày phát hành: 12/03/2024
|
| 58 |
3. Tổng tiền: 1.250.000 VND
|
| 59 |
```
|
|
|
|
| 60 |
If data cannot be extracted:
|
| 61 |
```bash
|
| 62 |
tôi không thể tìm thấy dữ liệu này
|
| 63 |
```
|
| 64 |
+
Structured Table (Column-wise) Extraction Prompt
|
|
|
|
|
|
|
| 65 |
Used to extract data for a specific column from a product table.
|
| 66 |
|
| 67 |
```bash
|
|
|
|
| 73 |
f"Yêu cầu tiêu đề cột là |{col_name}|.\n"
|
| 74 |
)
|
| 75 |
```
|
|
|
|
|
|
|
| 76 |
Expected Output
|
| 77 |
+
```bash
|
| 78 |
|Tên sản phẩm|
|
| 79 |
|------------|
|
| 80 |
|Sản phẩm A|
|