ocr
handwriting-recognition
evaluation
ukrainian

RUKOPYS evaluation code

Code behind the RUKOPYS Leaderboard: full-page OCR of Ukrainian handwritten documents on the public test split of RUKOPYS. Results and raw replies: rukopys-leaderboard-results.

file role
kaggle_metric.py official competition metric, unchanged
rukopys_normalize.py parses a model reply into [{bbox, type, text}]
prompts.py the prompt (one text, two box formats: pixels or fractions)
data.py loads the split from the Hub and prepares the exact image sent
run_api.py runner for Anthropic, Gemini (Vertex AI) and OpenAI
run_vllm.py runner for open models behind an OpenAI-compatible server
hf_openai_server.py transformers server, used only for Gemma 4 12B at 1120 image tokens
ocr_page_runner.py runner for OCR models (Chandra, dots.mocr): their own layout prompt, blocks split into lines
score.py scores runs: overall and per-source metrics
reproduce.sh exact server and run command of every published run
dev_pages.txt 48 train pages used to choose the prompt variant

Usage

pip install -r requirements.txt            # scoring
pip install -r requirements-api.txt        # API runners

# re-score a published run (no API calls)
python score.py rukopys-leaderboard-results/predictions/gemini-4-argon__high.parquet

# run an API model (keys from the environment: ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_APPLICATION_CREDENTIALS + GOOGLE_CLOUD_PROJECT)
python run_api.py --provider gemini --model gemini-4-argon --reasoning high --prompt spec-nfrac-geom --max-tokens 65000
python score.py preds/test/gemini-4-argon__spec-nfrac-geom__r-high__s1568__m65k/

# run an open model: start the server line from reproduce.sh, then
python run_vllm.py --model google/gemma-4-31B-it --family gemma --reasoning off --prompt spec-npx-geom

Data is downloaded from the Hub (dataset revision 79fb7009); --data-dir uses a local copy.

Protocol

  • One request per page: the image (1568 px long side, JPEG q92) and the prompt. No system prompt, tools or examples.
  • Other sizes: --size 2048; native with run_vllm.py --size 0 (the Hub image, downscaled only above 16.8 MP; vLLM --max-model-len 65536) and, for Gemini, run_api.py --size 2560 --media-resolution ultra_high.
  • The prompt variant per model was chosen on dev_pages.txt (at 2048 px for models added before 2026-11-11, at 1568 px since), never on test.
  • Open models: greedy without reasoning; with reasoning, the model card's sampling. API models: provider defaults (Gemini without reasoning: temperature 0).
  • Reasoning: --reasoning off, an effort level (low, medium, high, ...) or budget:<tokens> (older models). Claude off sends thinking: disabled (between_tools on Sonnet 5.5) to models that think by default when the parameter is omitted; Claude Opus 5.5 and Fable cannot run without thinking, so off stops with an error there (since 2026-11-16; earlier runs omitted the parameter, see the results changelog).
  • A reply with no parseable region gets one retry; if it fails again the page is scored as empty. Infrastructure failures are re-requested (re-running a command skips finished pages).
  • Metric: 0.15·DetF1 + 0.05·ClassAcc + 0.30·(1 − RegionCER) + 0.50·(1 − PageCER), computed by kaggle_metric.py.

Re-scoring the published replies reproduces every published number exactly. API models can change behind the same name, so new API runs give close but not identical results.

License

CC BY 4.0, like the dataset.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train UkrainianCatholicUniversity/rukopys-eval