File size: 1,904 Bytes
cbe2185 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | # Data format
Training examples are UTF-8 JSONL: one JSON object per line. Each object contains one or more images, a user
instruction, and a manually verified assistant answer.
## Canonical format
```json
{
"images": ["dataset/images/page-0001.jpg"],
"messages": [
{
"role": "user",
"content": [
{"type": "image", "image": "dataset/images/page-0001.jpg"},
{"type": "text", "text": "Extract only visible document text exactly. Preserve RTL order and printed digits. Ignore watermarks and stamps. Never repeat text."}
]
},
{
"role": "assistant",
"content": [
{"type": "text", "text": "{\"full_text\": \"النص المصحح\", \"has_tables\": false, \"has_lists\": false}"}
]
}
],
"source": "verified:document.pdf:1",
"trusted": true
}
```
## Quality rules
- The target must be transcribed from the image and reviewed by a person.
- Keep Arabic right-to-left reading order, line breaks, punctuation, and printed digit shapes.
- Do not include watermarks, logos, stamps, signatures, borders, or scanner artifacts unless the task explicitly
asks for them.
- Keep tables and lists in a consistent representation; do not flatten a table differently from one example to
the next.
- Do not use model output as ground truth without human verification.
- Keep validation pages separate from training pages. Do not reuse the test pages during fine-tuning.
- Avoid absolute paths containing a user's home directory when publishing a dataset.
## Dataset manifests
For each collection, keep a small `manifest.json` with the source, row count, verification method, license,
and whether images are safe to redistribute. This makes it possible to audit the model lineage without uploading
private source documents.
The existing datasets under `dataset/` remain in place and are not reorganized by this repository layout.
|