trocr-base-rukopys-formula

A TrOCR-base (VisionEncoderDecoderModel, ~334M params) recognizer for handwritten mathematical expressions in Ukrainian documents, fine-tuned on the formula regions of the Rukopys dataset through a staged formula curriculum: HME100K → CROHME → synthetic Cyrillic formulas (noisy profile) → Rukopys silver → Rukopys gold.

TL;DR

	value
Architecture	TrOCR-base (ViT-B/16 encoder @ 384px + TrOCR decoder), VisionEncoderDecoderModel
Parameters	~334M
Init from	Hukyl/trocr-rukopys (Rukopys line recognizer)
Tokenizer	stock RoBERTa vocab, size 50265
Handles	formula region crops
Gold-val CER / WER	0.2079 / 0.4455
Input	a single cropped formula region (RGB)
Output	normalized formula text — Unicode math symbols, structural LaTeX, Cyrillic words/units

Intended use

Recognizing cropped formula regions from handwritten Ukrainian documents (school-style math, chemistry, unit arithmetic), downstream of a layout detector. The output space is the normalized Rukopys formula convention.

For the plain-text region classes, see Hukyl/trocr-rukopys and Hukyl/trocr-large-rukopys.

How to use

from PIL import Image
from transformers import VisionEncoderDecoderModel, TrOCRProcessor

repo = "Hukyl/trocr-base-rukopys-formula"
processor = TrOCRProcessor.from_pretrained(repo)
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

crop = Image.open("formula_crop.png").convert("RGB")   # one detected formula region
pixel_values = processor(images=crop, return_tensors="pt").pixel_values
generated = model.generate(pixel_values, num_beams=4, max_length=256)
text = processor.batch_decode(generated, skip_special_tokens=True)[0]
print(text)

Output format

Rukopys formula labels are dual-mode: mostly plain linear math (R = 5, 7/10 y - 2/10 y = 5,5), with a minority of structural LaTeX (\frac, \int, \begin{array}, Ukrainian inside \text{…}). Training targets are normalized: LaTeX symbol commands become Unicode (\pi → π, \cdot → ·) while structural commands survive. The model emits that normalized space — Unicode symbols, slash fractions, Cyrillic words and units, and structural LaTeX where the source notation is genuinely two-dimensional.

Lineage & training curriculum

Staged curriculum; each stage fine-tunes the previous checkpoint. Optimizer: AdamW via the 🤗 Seq2SeqTrainer, linear LR decay, warmup ratio 0.1, max_target_length 256, seed 42. Every fine-tune stage uses the hybrid partial freeze: trainable are the upper half of the encoder blocks (plus the final encoder layernorm), the upper half of the decoder self-attentions, and all decoder cross-attentions; the lower blocks, embeddings, and LM head stay frozen. Intermediate stages select checkpoints by validation loss, the gold stage by validation CER.

1. Base — kazars24/trocr-base-handwritten-ru: microsoft/trocr-base-handwritten fine-tuned for Russian handwriting.
2. Rukopys line recognizer — Hukyl/trocr-rukopys: silver→gold fine-tune on the handwritten/printed/annotation/table crops (gold-val CER 0.1146).
3. HME100K (3 ep): handwritten mathematical expressions with LaTeX targets, normalized into the target space above.
4. CROHME (3 ep): the ICDAR/ICFHR competition corpus; InkML stroke traces rendered to 384px raster images, labels normalized like HME100K.
5. Synthetic Cyrillic formulas, noisy profile (3 ep): an in-house generator font-renders template-sampled formula strings — fractions, equations, square roots, angles, unit arithmetic, sequences, chemistry, mixed fractions, multi-step expressions, matrices — with Cyrillic variables and Ukrainian unit words (см, кг, год, …), and render-time degradation (rotation ±2°, Gaussian blur, additive noise, baseline jitter, random fonts and sizes). The noisy profile weights the mix toward the harder families (chemistry, mixed fractions, multi-step, matrices, deliberately ambiguous short forms) that the balanced profile leaves out.
6. Rukopys silver formula pretrain (5 ep): auto-labeled formula crops from the Rukopys silver split.
7. Rukopys gold formula fine-tune (15 ep): the human-labeled formula crops — the published weights.

Hyperparameters (gold stage, as launched)

hyperparameter	value
epochs (max)	15 (early stopping, patience 3)
batch size	16
learning rate	5e-5
warmup ratio	0.1
freeze	hybrid (see curriculum)
max target length	256 tokens
optimizer / schedule	AdamW, linear decay
eval	greedy (num_beams=1), full val split
seed	42

The full recorded recipe ships in this repo's training_meta.json.

Results

Protocol: greedy decode, the formula validation split of the Rukopys gold train data (seed 42, crop-level split), full split. Labels are metric-normalized as described in Output format.

metric	value
CER	0.2079
WER	0.4455

Compared runs

Same curriculum and gold-stage recipe unless noted; all compared checkpoints are internal and unpublished.

run	CER	WER
this model (noisy synthetic stage)	0.2079	0.4455
balanced synthetic stage	0.2082	0.4477
no synthetic stage	0.2178	0.4576
no silver stage	0.2489	0.5017
gold stage as LoRA / DoRA adaptation	~0.2233	—

This model was evaluated on the full validation split; the compared runs on a 2,000-sample subsample. The gap to the balanced-profile run is within evaluation noise.

Limitations & biases

• Outputs are normalized formula text, not faithful LaTeX transcription — not a general image-to-LaTeX model.
• Handwritten Ukrainian school-style material only; expect degradation on other scripts, printed math, or clean typeset input.
• The train/val split is stratified at crop level, not page level — crops from the same page (and writer) appear on both sides, so the reported val CER is optimistic compared to a page-level protocol.
• Single seed and validation split — no across-run variance estimate.

Training data & attribution

dataset	source	license	role
Rukopys	UkrainianCatholicUniversity/rukopys	CC BY 4.0	silver formula pretrain + gold formula fine-tune
HME100K	"When Counting Meets HMER" (ECCV 2022), TAL	not restated here — verify with the source	formula pretrain
CROHME	ICDAR/ICFHR CROHME competition corpus	not restated here — verify with the source	formula pretrain
synthetic Cyrillic formulas	generated in-house (font-rendered)	—	synthetic pretrain stage

The base model kazars24/trocr-base-handwritten-ru declares no explicit license — verify upstream terms before redistribution.