docs/data_format.md

Data format

Training examples are UTF-8 JSONL: one JSON object per line. Each object contains one or more images, a user instruction, and a manually verified assistant answer.

Canonical format

{
  "images": ["dataset/images/page-0001.jpg"],
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "image", "image": "dataset/images/page-0001.jpg"},
        {"type": "text", "text": "Extract only visible document text exactly. Preserve RTL order and printed digits. Ignore watermarks and stamps. Never repeat text."}
      ]
    },
    {
      "role": "assistant",
      "content": [
        {"type": "text", "text": "{\"full_text\": \"النص المصحح\", \"has_tables\": false, \"has_lists\": false}"}
      ]
    }
  ],
  "source": "verified:document.pdf:1",
  "trusted": true
}

Quality rules

• The target must be transcribed from the image and reviewed by a person.
• Keep Arabic right-to-left reading order, line breaks, punctuation, and printed digit shapes.
• Do not include watermarks, logos, stamps, signatures, borders, or scanner artifacts unless the task explicitly asks for them.
• Keep tables and lists in a consistent representation; do not flatten a table differently from one example to the next.
• Do not use model output as ground truth without human verification.
• Keep validation pages separate from training pages. Do not reuse the test pages during fine-tuning.
• Avoid absolute paths containing a user's home directory when publishing a dataset.

Dataset manifests

For each collection, keep a small manifest.json with the source, row count, verification method, license, and whether images are safe to redistribute. This makes it possible to audit the model lineage without uploading private source documents.

The existing datasets under dataset/ remain in place and are not reorganized by this repository layout.