Abjad_0.2 / docs /evaluation.md
Alsamir's picture
Add Hunyuan-style repository documentation and tooling
92134b1 verified
|
Raw History Blame Contribute Delete
1.35 kB

Evaluation guide

Compare models on the same rendered pages, prompt, resolution, and output budget. A single successful page is not enough to rank OCR systems.

Recommended protocol

  1. Keep a held-out set of Arabic scans with verified text.
  2. Render each PDF page once and use the same image for every model.
  3. Use deterministic decoding and the same instruction: exact visible transcription, preserve RTL order and digit shapes, ignore watermarks and signatures, and never repeat text.
  4. Compare normalized character error rate (CER), word error rate (WER), digit accuracy, and table-cell accuracy.
  5. Review layout separately from text accuracy. A model can have good text but poor reading order.

Reproducible Abjad command

.venv/bin/python mlx_ocr_pipeline.py page.pdf \
  -o outputs/eval_abjad.txt \
  --model /path/to/Abjad_0.2 \
  --no-adapter \
  --regions \
  --preserve-digit-shapes \
  --max-tokens 4096

Keep the raw output. Post-processing that collapses a runaway repetition is a safety filter, not evidence that the model recognized the original page correctly.

Reporting

Report the page count, source type, whether labels were human verified, prompt, resolution, model revision, and the exact metrics. If no ground truth exists, describe the result qualitatively and do not assign a numeric score.