| # Data format |
|
|
| Training examples are UTF-8 JSONL: one JSON object per line. Each object contains one or more images, a user |
| instruction, and a manually verified assistant answer. |
|
|
| ## Canonical format |
|
|
| ```json |
| { |
| "images": ["dataset/images/page-0001.jpg"], |
| "messages": [ |
| { |
| "role": "user", |
| "content": [ |
| {"type": "image", "image": "dataset/images/page-0001.jpg"}, |
| {"type": "text", "text": "Extract only visible document text exactly. Preserve RTL order and printed digits. Ignore watermarks and stamps. Never repeat text."} |
| ] |
| }, |
| { |
| "role": "assistant", |
| "content": [ |
| {"type": "text", "text": "{\"full_text\": \"النص المصحح\", \"has_tables\": false, \"has_lists\": false}"} |
| ] |
| } |
| ], |
| "source": "verified:document.pdf:1", |
| "trusted": true |
| } |
| ``` |
|
|
| ## Quality rules |
|
|
| - The target must be transcribed from the image and reviewed by a person. |
| - Keep Arabic right-to-left reading order, line breaks, punctuation, and printed digit shapes. |
| - Do not include watermarks, logos, stamps, signatures, borders, or scanner artifacts unless the task explicitly |
| asks for them. |
| - Keep tables and lists in a consistent representation; do not flatten a table differently from one example to |
| the next. |
| - Do not use model output as ground truth without human verification. |
| - Keep validation pages separate from training pages. Do not reuse the test pages during fine-tuning. |
| - Avoid absolute paths containing a user's home directory when publishing a dataset. |
|
|
| ## Dataset manifests |
|
|
| For each collection, keep a small `manifest.json` with the source, row count, verification method, license, |
| and whether images are safe to redistribute. This makes it possible to audit the model lineage without uploading |
| private source documents. |
|
|
| The existing datasets under `dataset/` remain in place and are not reorganized by this repository layout. |
|
|
