File size: 1,347 Bytes
92134b1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | # Evaluation guide
Compare models on the same rendered pages, prompt, resolution, and output budget. A single successful page is not
enough to rank OCR systems.
## Recommended protocol
1. Keep a held-out set of Arabic scans with verified text.
2. Render each PDF page once and use the same image for every model.
3. Use deterministic decoding and the same instruction: exact visible transcription, preserve RTL order and digit
shapes, ignore watermarks and signatures, and never repeat text.
4. Compare normalized character error rate (CER), word error rate (WER), digit accuracy, and table-cell accuracy.
5. Review layout separately from text accuracy. A model can have good text but poor reading order.
## Reproducible Abjad command
```bash
.venv/bin/python mlx_ocr_pipeline.py page.pdf \
-o outputs/eval_abjad.txt \
--model /path/to/Abjad_0.2 \
--no-adapter \
--regions \
--preserve-digit-shapes \
--max-tokens 4096
```
Keep the raw output. Post-processing that collapses a runaway repetition is a safety filter, not evidence that the
model recognized the original page correctly.
## Reporting
Report the page count, source type, whether labels were human verified, prompt, resolution, model revision, and
the exact metrics. If no ground truth exists, describe the result qualitatively and do not assign a numeric score.
|