UkrainianCatholicUniversity/rukopys
Viewer • Updated • 10.6k • 5.98k • 28
Code behind the RUKOPYS Leaderboard:
full-page OCR of Ukrainian handwritten documents on the public test split of
RUKOPYS.
Results and raw replies: rukopys-leaderboard-results.
| file | role |
|---|---|
kaggle_metric.py |
official competition metric, unchanged |
rukopys_normalize.py |
parses a model reply into [{bbox, type, text}] |
prompts.py |
the prompt (one text, two box formats: pixels or fractions) |
data.py |
loads the split from the Hub and prepares the exact image sent |
run_api.py |
runner for Anthropic, Gemini (Vertex AI) and OpenAI |
run_vllm.py |
runner for open models behind an OpenAI-compatible server |
hf_openai_server.py |
transformers server, used only for Gemma 4 12B at 1120 image tokens |
ocr_page_runner.py |
runner for OCR models (Chandra, dots.mocr): their own layout prompt, blocks split into lines |
score.py |
scores runs: overall and per-source metrics |
reproduce.sh |
exact server and run command of every published run |
dev_pages.txt |
48 train pages used to choose the prompt variant |
pip install -r requirements.txt # scoring
pip install -r requirements-api.txt # API runners
# re-score a published run (no API calls)
python score.py rukopys-leaderboard-results/predictions/gemini-4-argon__high.parquet
# run an API model (keys from the environment: ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_APPLICATION_CREDENTIALS + GOOGLE_CLOUD_PROJECT)
python run_api.py --provider gemini --model gemini-4-argon --reasoning high --prompt spec-nfrac-geom --max-tokens 65000
python score.py preds/test/gemini-4-argon__spec-nfrac-geom__r-high__s1568__m65k/
# run an open model: start the server line from reproduce.sh, then
python run_vllm.py --model google/gemma-4-31B-it --family gemma --reasoning off --prompt spec-npx-geom
Data is downloaded from the Hub (dataset revision 79fb7009); --data-dir uses a local copy.
--size 2048; native with run_vllm.py --size 0 (the Hub image, downscaled only above 16.8 MP; vLLM
--max-model-len 65536) and, for Gemini, run_api.py --size 2560 --media-resolution ultra_high.dev_pages.txt (at 2048 px for models added before 2026-11-11, at 1568 px
since), never on test.--reasoning off, an effort level (low, medium, high, ...) or budget:<tokens> (older models).
Claude off sends thinking: disabled (between_tools on Sonnet 5.5) to models that think by default when the
parameter is omitted; Claude Opus 5.5 and Fable cannot run without thinking, so off stops with an error there
(since 2026-11-16; earlier runs omitted the parameter, see the results changelog).0.15·DetF1 + 0.05·ClassAcc + 0.30·(1 − RegionCER) + 0.50·(1 − PageCER), computed by kaggle_metric.py.Re-scoring the published replies reproduces every published number exactly. API models can change behind the same name, so new API runs give close but not identical results.
CC BY 4.0, like the dataset.