---
title: HTR · VLM as Annotator
emoji: 📜
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: HTR via VLM and MoE post-correction
---

# HTR VLM + MoE Post-correction

Custom HuggingFace-style web app for Handwritten Text Recognition by Vision-Language Models (via OpenRouter) and Mixture-of-Experts post-correction.

Logic : *LLM as annotator* — the model proposes a transcription, the human validates / corrects, the validated examples feed an in-context-learning (ICL) pool that is re-injected as few-shot for subsequent pages.

## Two modes

### 1. OCR
- Upload one or more page images (manuscripts, prints, …).
- Select an active page, run the OCR with a single VLM of your choice (any OpenRouter model slug).
- Edit the result inline. After "Validate & score", edited words are highlighted in light green and CER / WER are computed (corrected = reference).
- "Add to ICL" stores `{image, corrected_text}` in the in-memory pool. Future OCR calls inject up to *N* recent examples as few-shot, so the VLM learns the hand / orthography on the fly.
- A quick-add tool also lets you paste a validated transcription directly without going through the editor.

### 2. Post-correction (MoE)
- Image + OCR text (paste your own, or come from mode 1).
- Two expert VLMs are called **in parallel** with the same prompt.
- If they agree → consensus result. Otherwise → a third "judge" VLM arbitrates.
- Defaults : `google/gemini-3-flash-preview`, `qwen/qwen3.5-flash-02-23`, judge `mistralai/mistral-large-2512` (all editable from Settings).

## Customisation
- Language / script field and free-text guidelines are merged into the prompts.
- Any model slug from OpenRouter can be entered manually.
- Advanced : custom system / user prompt overrides for the OCR call.
- All settings are reachable from the header menus (no hidden CLI flags).

## Run locally

```bash
conda activate research
cd Webapp
pip install -r requirements.txt
uvicorn app:app --reload --port 7860
```

Then open <http://localhost:7860>. Open the *API key* modal, paste your OpenRouter key (or export `OPENROUTER_API_KEY` in the environment as fallback), click *Test*, then start uploading images.

## Run in Docker / HF Spaces

```bash
docker build -t htr-vlm .
docker run -p 7860:7860 -e OPENROUTER_API_KEY=sk-or-v1-… htr-vlm
```

Pushing the directory to a HuggingFace Space (Docker SDK) deploys it as-is.

## Notes
- State is **in-memory and single-process** (intended single-user demo / annotator workflow). Use the *Export JSONL* button on the ICL pool to persist before reloading.
- Images are normalised to JPEG, ≤ 2200 px on the longest side, sent as base64 data-URLs (OpenRouter multimodal request shape).
- No telemetry, no persistence on disk except what you explicitly download.

## Architecture
- `app.py` — FastAPI routes, in-memory session.
- `provider.py` — async OpenRouter client (multimodal).
- `prompts.py` + `data/prompts/*.txt` — system / user templates and ICL pool.
- `moe.py` — two-experts + judge pipeline.
- `io_utils.py` — image normalisation, word-level diff, CER / WER (jiwer-backed).
- `schemas.py` — Pydantic request shapes.
- `static/` — Alpine.js + Tailwind single-page UI.