README.md

metadata

title: HTR · VLM as Annotator
emoji: 📜
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: HTR via VLM and MoE post-correction

HTR VLM + MoE Post-correction

Custom HuggingFace-style web app for Handwritten Text Recognition by Vision-Language Models (via OpenRouter) and Mixture-of-Experts post-correction.

Logic : LLM as annotator — the model proposes a transcription, the human validates / corrects, the validated examples feed an in-context-learning (ICL) pool that is re-injected as few-shot for subsequent pages.

Two modes

1. OCR

• Upload one or more page images (manuscripts, prints, …).
• Select an active page, run the OCR with a single VLM of your choice (any OpenRouter model slug).
• Edit the result inline. After "Validate & score", edited words are highlighted in light green and CER / WER are computed (corrected = reference).
• "Add to ICL" stores {image, corrected_text} in the in-memory pool. Future OCR calls inject up to N recent examples as few-shot, so the VLM learns the hand / orthography on the fly.
• A quick-add tool also lets you paste a validated transcription directly without going through the editor.

2. Post-correction (MoE)

• Image + OCR text (paste your own, or come from mode 1).
• Two expert VLMs are called in parallel with the same prompt.
• If they agree → consensus result. Otherwise → a third "judge" VLM arbitrates.
• Defaults : google/gemini-3-flash-preview, qwen/qwen3.5-flash-02-23, judge mistralai/mistral-large-2512 (all editable from Settings).

Customisation

• Language / script field and free-text guidelines are merged into the prompts.
• Any model slug from OpenRouter can be entered manually.
• Advanced : custom system / user prompt overrides for the OCR call.
• All settings are reachable from the header menus (no hidden CLI flags).

Run locally

conda activate research
cd Webapp
pip install -r requirements.txt
uvicorn app:app --reload --port 7860

Then open http://localhost:7860. Open the API key modal, paste your OpenRouter key (or export OPENROUTER_API_KEY in the environment as fallback), click Test, then start uploading images.

Run in Docker / HF Spaces

docker build -t htr-vlm .
docker run -p 7860:7860 -e OPENROUTER_API_KEY=sk-or-v1-… htr-vlm

Pushing the directory to a HuggingFace Space (Docker SDK) deploys it as-is.

Notes

• State is in-memory and single-process (intended single-user demo / annotator workflow). Use the Export JSONL button on the ICL pool to persist before reloading.
• Images are normalised to JPEG, ≤ 2200 px on the longest side, sent as base64 data-URLs (OpenRouter multimodal request shape).
• No telemetry, no persistence on disk except what you explicitly download.

Architecture

• app.py — FastAPI routes, in-memory session.
• provider.py — async OpenRouter client (multimodal).
• prompts.py + data/prompts/*.txt — system / user templates and ICL pool.
• moe.py — two-experts + judge pipeline.
• io_utils.py — image normalisation, word-level diff, CER / WER (jiwer-backed).
• schemas.py — Pydantic request shapes.
• static/ — Alpine.js + Tailwind single-page UI.