--- license: apache-2.0 base_model: PaddlePaddle/PaddleOCR-VL-1.6 language: - bo tags: - ocr - tibetan - vision-language - paddleocr-vl - pecha pipeline_tag: image-text-to-text library_name: transformers metrics: - cer --- # Tibetan OCR — Yigdzin 1 **A vision-first OCR model for Tibetan pecha** — modern publications, woodblock prints, and manuscripts (uchen and u-med), including pecha layout and orthographic shorthands. Built by the [Buddhist Digital Resource Center (BDRC)](https://bdrc.io) and trained on the `bec_mixed_elie_v8` mix of curated real + synthetic pages, with the **evaluation-benchmark pages held out of training**. - **Repository:** `BDRC/tibetan-ocr` (display name **Yigdzin 1**) - **Base model:** [`PaddlePaddle/PaddleOCR-VL-1.6`](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) (Apache-2.0) - **Live demo:** [ocr.bdrc.io](https://ocr.bdrc.io) · **Desktop app:** [buda-base/tibetan-ocr-app](https://github.com/buda-base/tibetan-ocr-app) ## TL;DR On a 1,070-page hand-transcribed benchmark (production serving: vLLM + sequential regime + DRY guard + temperature-retry), **median page CER is ~1.4%** and clean-page (non-catastrophic) mean CER is **~4.0%**, with **0 hard repetition loops**. The benchmark pages are held out of training. It is a *specialized* OCR model, not a general chat VLM. ## Model details | | | |---|---| | Architecture | PaddleOCR-VL-1.6 (vision encoder + depth-upscaled decoder) | | Parameters | ~810M (~413M vision · ~360M 26-layer decoder · ~4M embeddings/head) | | Vision share | ~51–59% of parameters ("big eyes, small mouth" — vision-first) | | Tokenizer | Tibetan **unicode-stack** tokenizer, vocab **3,560** (pruned + BoCorpus-warmed) | | Decoder growth | grow26: SOLAR-style depth-upscale 18L→26L, then annealed | | Position regime | **sequential** image-token M-RoPE (1-D); `max_pixels` 1280 budget | | Languages | Tibetan (`bo`); Tibetan script | | License | Apache-2.0 (derivative of PaddleOCR-VL-1.6; upstream `NOTICE` retained) | | Funder | Khyentse Foundation ("The BDRC Etext Corpus") | ## Intended use - OCR of Tibetan pecha pages: **uchen** and **u-med** (dbu med) scripts across **modern print, woodblock, and manuscript** sources. - Batch OCR at corpus scale (BDRC is applying it to ~25M scanned pages). **Out of scope:** general vision-language chat / VQA; non-Tibetan scripts; layout analysis of illustrations, tables, or diagrams; line/region detection (this model transcribes page or line crops it is given). ## How to use Prompt (used at training and inference), rendered through the repo's `chat_template.jinja`: ``` Extract all Tibetan text. Preserve line breaks. ``` This checkpoint uses the **native** PaddleOCR-VL architecture (`model_type: paddleocr_vl`, `PaddleOCRVLForConditionalGeneration`) — **no `trust_remote_code` needed**, but it requires **transformers ≥ 5.15** (native `paddleocr_vl` support) or **vLLM ≥ 0.26**. Decode **greedy** (`temperature=0`). ```bash pip install "transformers>=5.15" torch torchvision accelerate pillow # torchvision is required by the PaddleOCR-VL image processor. ``` > **⚠ Position regime — required for correct output.** This model was trained in the > **sequential** image-token M-RoPE regime (1-D positions), *not* the default "grid" > regime. If the processor emits a non-zero `mm_token_type_ids`, structured pages > loop badly (some pages regress from ~0.10 to ~0.62 CER). Serve it sequential: > - **HF:** zero the mask before the forward — > `inputs["mm_token_type_ids"] = torch.zeros_like(inputs["input_ids"])`. > - **vLLM:** install the `vllm_paddleocr_seqpos` plugin and set > `OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential`. ### HF transformers (reference / accuracy baseline) ```python import torch from PIL import Image from transformers import AutoProcessor, AutoModelForImageTextToText model_id = "BDRC/tibetan-ocr" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="bfloat16", device_map="cuda") image = Image.open("page.jpg").convert("RGB") messages = [{"role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "Extract all Tibetan text. Preserve line breaks."}]}] prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) inputs = processor(text=[prompt], images=[image], return_tensors="pt").to("cuda") inputs["mm_token_type_ids"] = torch.zeros_like(inputs["input_ids"]) # sequential regime (required) out = model.generate(**inputs, do_sample=False, max_new_tokens=4096) print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]) ``` To add the **DRY** anti-loop guard under HF too (optional; `dry_logits_processor.py` ships in this repo): ```python from transformers import LogitsProcessorList from dry_logits_processor import make_hf_dry_processor prompt_len = inputs["input_ids"].shape[1] out = model.generate(**inputs, do_sample=False, max_new_tokens=4096, logits_processor=LogitsProcessorList([make_hf_dry_processor( prompt_len=prompt_len, multiplier=0.8, base=1.75, allowed_length=12)])) ``` ### vLLM (production) Serve with **vLLM ≥ 0.26**. Two things matter for production quality: **1. Sequential image-token positions (required).** Stock vLLM serves PaddleOCR-VL in the *grid* regime and never reads `mm_token_type_ids`; this model needs the *sequential* regime. A tiny vLLM plugin ships in this repo under `vllm_paddleocr_seqpos/`: ```bash pip install "git+https://huggingface.co/BDRC/tibetan-ocr#subdirectory=vllm_paddleocr_seqpos" export OCR_VLLM_IMAGE_TOKEN_POSITIONS=sequential # 'grid' / unset = no-op ``` Without it, structured (book/list) pages skip or merge lines and CER regresses badly. **2. DRY anti-loop guard + temperature retry (recommended).** Greedy decoding runs away into repetition on a small fraction of pages. The surgical fix is the **DRY** ("Don't Repeat Yourself") penalty: it ties an n-gram ban on corpus CER, zeroes hard loops, and — unlike `repetition_penalty` / `no_repeat_ngram_size` — barely touches clean pages or legitimate repetitive scripture (mantras, litanies). It ships here as the self-contained `dry_logits_processor.py` (torch-only): ```python from vllm import LLM, SamplingParams from dry_logits_processor import DRYLogitsProcessor # ships in this repo llm = LLM(model="BDRC/tibetan-ocr", logits_processors=[DRYLogitsProcessor]) # + seqpos env above params = SamplingParams(temperature=0, max_tokens=4096, extra_args={ "dry_multiplier": 0.8, "dry_base": 1.75, "dry_allowed_length": 12, # production config, no breakers }) ``` For the pages that still loop under greedy+DRY, **re-decode at temperature and keep the cleanest sample.** The strong signal is *how often DRY fired*: the measured knee is **≥ 100 fires** — re-decode only those pages at `temperature=0.4`, `n=3` (DRY still on) and pick the sample with the lowest leftover repetition. On this benchmark that is ~2% of pages and clears the residual hard loops. `dry_logits_processor.py` writes per-request fire counts (pass `dry_stats_id` / `dry_stats_path` in `extra_args`, read them back with `load_dry_stats_dir`); `deploy/fast_inference/bench.py` in the training repo is the reference implementation of the full greedy → fire-count gate → temperature-retry loop. Do **not** reach for aggressive `repetition_penalty` / `no_repeat_ngram_size`: they "correct away" legitimate repeated scripture. DRY with the config above is the tested, surgical choice. ## Training data Curated mix (`bec_mixed_elie_v8`), assembled from BDRC alignment collections + BoCorpus-rendered synthetic pages, with an easy→hard curriculum, u-med over-weighting, and repetition-aware filtering. **The evaluation-benchmark pages are held out of training** (`images_exclude_from_train.csv`). Released component datasets: - Real transcriptions: [ALL-BDRC (ACIP Sungbum)](https://huggingface.co/datasets/BDRC/ALL-BDRC-alignments), [PalriParkhang](https://huggingface.co/datasets/BDRC/palri-parkhang), [Berkeley](https://huggingface.co/datasets/BDRC/berkeley), [MonlamAI-transcriptions](https://huggingface.co/datasets/BDRC/monlamai-transcriptions), [MonlamAI-handwritten](https://huggingface.co/datasets/BDRC/monlamai-handwritten), [Stok](https://huggingface.co/datasets/openpecha/stok), [TibSchol](https://huggingface.co/datasets/BDRC/tibschol) *(gated)*. - Synthetic: [Tibetan OCR synthetic v5](https://huggingface.co/datasets/BDRC/tibetan-ocr-synthetic). ## Evaluation Evaluated on the BDRC hand-transcribed benchmark `20260315` (1,070 pages) in the **production configuration** (vLLM + `vllm_paddleocr_seqpos` sequential regime + DRY guard `mult=0.8 base=1.75 allowed_length=12` + temperature-retry `temp=0.4 n=3` on pages with ≥100 DRY fires). Metrics from the benchmark's `compute_cer.py` (botok normalization, whitespace stripping, tsheg folding, placeholder removal); pages with CER > 50% are counted as *catastrophic* and reported separately. | Metric (1,070 pages) | **Yigdzin 1** | |---|---:| | Median page CER | **0.0142** | | Mean page CER (all) | 0.0628 | | Catastrophic (>50%) share | 3.3% (35) | | Clean mean CER (non-catastrophic) | **0.0404** | | Clean mean SER | **0.0764** | | Hard repetition loops | **0** | Full method + the multi-system leaderboard: **[BDRC Tibetan OCR benchmark](https://huggingface.co/datasets/BDRC/tibetan-ocr-benchmark)** and the **[leaderboard](https://huggingface.co/spaces/BDRC/tibetan-ocr-leaderboard)**. ## Limitations & recommendations - **Residual repetition loops** on a small fraction of pages. Production uses the **DRY** guard + temperature retry (see How to use), which clears them (0 hard loops on the benchmark) while leaving legitimate repetitive scripture (mantras) intact; keep any anti-loop mitigation this gentle. - **Line-break joins** on some dense (6-line) woodblock pecha. - Struggles on illustrations, non-Tibetan text, and illegible/damaged pages. ## Citation A comprehensive article is forthcoming (Springer *Language Resources and Evaluation*). Until then, please cite the model and BDRC: ```bibtex @misc{bdrc_tibetan_ocr_2026, title = {Tibetan OCR (Yigdzin 1)}, author = {Roux, Elie and Werner, Eric}, year = {2026}, howpublished = {Buddhist Digital Resource Center, Hugging Face}, note = {https://huggingface.co/BDRC/tibetan-ocr} } ``` ## Acknowledgements Built on [PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) (Apache-2.0). Funded by the **Khyentse Foundation**. Thanks to the BDRC, Dharmaduta, and collaborating transcription teams.