trocr-rukopys

A TrOCR-base (VisionEncoderDecoderModel, ~334M params) line recognizer for handwritten Ukrainian document text, fine-tuned on the Rukopys dataset.

The training data included handwritten, printed, annotation and table region crops from Rukopys.

TL;DR

	value
Architecture	TrOCR-base (ViT-B/16 encoder + TrOCR decoder), VisionEncoderDecoderModel
Parameters	~334M
Init from	kazars24/trocr-base-handwritten-ru
Handles	handwritten, printed, table, annotation
Gold-val CER / WER	0.1146 / 0.3089
Input	a single cropped text region/line (RGB)
Output	the transcribed string (metric-normalized character set)

Intended use

Recognizing cropped line/region images from handwritten Ukrainian documents, downstream of a layout detector.

For a stronger recognizer from a later, larger training generation, see Hukyl/trocr-large-rukopys.

How to use

from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-rukopys"
processor = TrOCRProcessor.from_pretrained(repo)
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("region_crop.png").convert("RGB")          # one detected line/region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, max_new_tokens=256)  # greedy
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

Lineage & training curriculum

Two-stage silver→gold fine-tuning from a Russian handwriting base. Optimizer: AdamW via the 🤗 Seq2SeqTrainer, linear LR decay, max_target_length 256, online augmentation, seed 42.

1. Base — kazars24/trocr-base-handwritten-ru: microsoft/trocr-base-handwritten fine-tuned for Russian handwriting.
2. Silver pretrain (5 ep): partial-freeze pretrain on the Rukopys silver split (~119.8k auto-labeled crops).
3. Gold fine-tune (up to 15 ep, early stopping with patience 3): full encoder + decoder unfreeze on the human-labeled gold train split, 19,069 train / 3,366 val crops, stratified at crop level.

Hyperparameters (as launched)

hyperparameter	value
epochs (max)	15 (early stopping, patience 3)
batch size	32
learning rate	1e-4
warmup ratio	0.1
max target length	256 tokens
optimizer / schedule	AdamW, linear decay
online augmentation	on (default profile)
eval	greedy (num_beams=1), eval batch 64
precision / device	bf16, CUDA
seed	42

The full recorded recipe ships in this repo's training_meta.json.

Online data augmentation

Online augmentation was applied during training. Only training crops were augmented (no augmented validation was measured).

Each crop was transformed once per epoch by one geometric + one or two photometric operations at random. Each class received a separate augmentation profile that was selected to minimize the distribution shift.

pool	handwritten / annotation	printed
geometric (pick 1)	margin pad/trim, rotation ±1–5°, elastic distortion, baseline warp	margin pad/trim, rotation ±1–3°
photometric (pick 1–2)	paper colour shift, Gaussian noise, JPEG recompression (q30–65), contrast/gamma, morphological op	paper colour shift, Gaussian noise, JPEG recompression (q40–70), contrast/gamma, morphological op

These simulate scanner/paper variation, ink thinning/bleed, and natural handwriting deformation, widening the range of appearances beyond the raw training crops.

Results

Protocol: greedy decode, max_new_tokens=512, 1000-sample draw from the gold validation split, seed 42. Labels are metric-normalized (LaTeX → Unicode, table PSV canonicalized) so CER/WER reflect the scored character set.

Overall

metric	value
CER	0.1146
WER	0.3089
exact-match accuracy	0.3830
n_samples	1000

Per class

class	CER	WER	accuracy	n
handwritten	0.1126	0.3036	0.3841	958
printed	0.0987	0.2826	0.4167	12
table	0.1915	0.5000	0.5000	2
annotation	0.4278	0.9268	0.3214	28

We also acknowledge that printed/table/annotation n is quite small, so measuring CER against them is quite noisy.

Limitations & biases

• Handwritten Ukrainian archival document material only; expect degradation on other scripts, languages, or modern born-digital text.
• The character set and normalization are tuned to the document-OCR metric — outputs are normalized text, not faithful transcription.
• The train/val split is stratified at crop level, not page level — crops from the same page (and writer) appear on both sides, so the reported val CER is optimistic compared to a page-level protocol.
• annotation and table classes are weak.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

dataset	source	license	role
Rukopys	UkrainianCatholicUniversity/rukopys	CC BY 4.0	silver pretrain (auto-labeled) + gold fine-tune

The base model kazars24/trocr-base-handwritten-ru declares no explicit license — verify upstream terms before redistribution.