trocr-large-rukopys

A TrOCR-Large (VisionEncoderDecoderModel, ~558M params) line recognizer for handwritten Ukrainian document text, fine-tuned on the Rukopys dataset.

The training data included handwritten, printed, annotation and table region crops from Rukopys.

TL;DR

	value
Architecture	TrOCR-Large (ViT-L/16 encoder @ 384px + TrOCR decoder), VisionEncoderDecoderModel
Parameters	~558M
Init from	Hukyl/trocr-large-uk-handwritten (broad mixed pretrain)
Tokenizer	extended Cyrillic vocab, size 50336 (ships with this repo)
Handles	handwritten, printed, table, annotation
Trained on	nearly all the gold data + weight EMA
Representative gold-val CER / WER	0.1072 / 0.2943
Input	a single cropped text region/line (RGB)
Output	the transcribed string (metric-normalized character set)

Intended use

Recognizing cropped line/region images from handwritten Ukrainian documents, downstream of a layout detector. This recognizer was tuned for robustness across all text region classes.

For better handwritten class CER (~0.075), see sibling model Hukyl/trocr-large-rukopys-hw.

How to use

Note: load the processor from this repo, not from microsoft/trocr-large-handwritten — the tokenizer here is an extended Cyrillic vocabulary (50336 vs stock 50265).

from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-large-rukopys"
processor = TrOCRProcessor.from_pretrained(repo)          # load from THIS repo
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("region_crop.png").convert("RGB")        # one detected line/region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, max_new_tokens=256, num_beams=1)  # greedy
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

Lineage & training curriculum

Rukopys gold-only fine-tuning directly from broad pretrain, with no silver/pseudo-labels. Full fine-tune, optimizer: AdamW via the 🤗 Seq2SeqTrainer, cosine LR decay, max_target_length 256, online augmentation, seed 42, single L40S.

1. Base — Kansallisarkisto/cyrillic-htr-model (Apache-2.0): microsoft/trocr-large-handwritten fine-tuned on ~30k historical Cyrillic rows by the Finnish National Archives; extended tokenizer (vocab 50336).
2. Mixed pretrain — Hukyl/trocr-large-uk-handwritten (6 ep, LR 4e-5): UkrHandwritten ×3 + Cyrillic Handwriting + part of pumb-ai/synthetic-cyrillic-large. The general Ukrainian/Cyrillic handwriting reader.
3. Rukopys gold fine-tune (22 ep, LR 2e-5, warmup 0.05, weight decay 0.01 + weight EMA): the human-labeled gold training data, with page-level validation split. Note: to incorporate as much of the Rukopys gold split as possible, this checkpoint had a near-zero validation set (~5 pages): 22,330 train / 105 val crops. The published weights are the EMA average of the training tail.

Weight EMA (why the published weights are an average)

At first, an original model with proper validation split (20%) was tested on Rukopys. After ablation on the held out test set (measured by Kaggle leaderboard), smaller validation splits produced better tests results. Therefore, to incorporate the gold-split data into training, we used the best-CER epoch from previous training, and added 2 more epochs (20 -> 22) to accommodate for more training data. The achieved EMA near-zero val model produced best results on the test split.

Hyperparameters (as launched)

hyperparameter	value
epochs	22 (= N+2, N=20)
batch size	28
learning rate	2e-5
warmup ratio	0.05
weight decay	0.01
weight EMA	decay 0.9995, start at 50% of training
max target length	256 tokens
optimizer / schedule	AdamW, cosine decay
online augmentation	on (per-region profile)
eval	greedy (num_beams=1)
precision / device	bf16, CUDA (single L40S)
seed	42

Fine-tune ran 17,556 steps in ~2.5h.

Online data augmentation

Online augmentation was applied during training. Only training crops were augmented (no augmented validation was measured).

Each crop was transformed once per epoch by one geometric + one or two photometric operations at random. Each class received a separate augmentation profile that was selected to minimize the distribution shift.

region type	geometric (one)	photometric (one or two)
handwritten, annotation	margin pad 2–15% / trim 1–5%, rotation ±1–5°, elastic distortion (α=25, σ=5), baseline warp (amp 2–8 px, freq 0.5–2.0)	paper-colour shift (LAB a±10 / b±15), Gaussian noise (σ 5–15), JPEG recompression (q 30–65), contrast/gamma (0.7–1.3 / 0.6–1.5), morphological erode/dilate (kernel 2)
printed, table	margin pad 2–15% / trim 1–5%, rotation ±1–3°	paper-colour shift (a±5 / b±10), Gaussian noise (σ 3–10), JPEG recompression (q 40–70), contrast/gamma (0.8–1.2 / 0.8–1.3), morphological erode/dilate (kernel 2)

These simulate scanner/paper variation, ink thinning/bleed, and natural handwriting deformation, widening the range of appearances beyond the raw training crops.

Results

Protocol: greedy decode, max_new_tokens=512, page-level gold validation split, 1000-sample draw, seed 42. Labels are metric-normalized (LaTeX → Unicode, table PSV canonicalized) so CER/WER reflect the scored character set.

Why the numbers come from a sibling

Since this checkpoint included almost all of the gold split, measuring CER on the tiny val split would result in meaningless metrics. Instead, we report CER on the model that used a 20% validation split, i.e. 185 pages. We suspect that this model is strictly equal or better than the reported numbers.

metric	training-time best	standalone predictor eval (n=1,000)
CER	0.1058	0.1072
WER	0.3001	0.2943
exact-match accuracy	—	0.3780

Per class

class	CER	WER	accuracy	n
handwritten	0.0975	0.2864	0.3721	954
annotation	0.1920	0.3514	0.6667	27
printed	0.1929	0.4351	0.3125	16
table	0.7764	1.0000	0.0000	3

We also acknowledge that printed/table/annotation n is quite small, so measuring CER against them is quite noisy.

Limitations & biases

• Handwritten Ukrainian archival document material only; expect degradation on other scripts, languages, or modern born-digital text.
• The character set and normalization are tuned to the document-OCR metric — outputs are normalized text, not faithful transcription.
• table is weak and annotation/printed are high-variance (small n).
• Headline metrics are from a same-recipe sibling, not this exact checkpoint (its own holdout is too small to score); EMA weights are an average, not a single training step.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

dataset	source	license	role
Rukopys	UkrainianCatholicUniversity/rukopys	CC BY 4.0	gold fine-tune
UkrHandwritten	Kaggle annyhnatiuk/ukrainian-handwritten-text	CC BY-SA 4.0	pretrain (lineage)
Cyrillic Handwriting Dataset	Kaggle constantinwerner/cyrillic-handwriting-dataset	CC0	pretrain (lineage)
pumb-ai/synthetic-cyrillic-large	HF	Apache-2.0	pretrain (lineage)