Spaces:
Running on Zero
Running on Zero
One stream each: TrOCR reads its lines one by one (its one-batch time beside it), the replay starts all three together, our server's vLLM timings under the race, one region box, Jev's rows folded with marked errors
Browse files- HOW_IT_WORKS.md +56 -21
- README.md +10 -6
- app.py +288 -116
- examples/19a1b2195bab_right_0/thumb.jpg +0 -0
- examples/35ffe6b8b600_left_0/thumb.jpg +0 -0
- examples/38c4ce6331ad_left_1/thumb.jpg +0 -0
- examples/8d24e7df7dea_2/thumb.jpg +0 -0
- examples/eed569a6699c_left_3/thumb.jpg +0 -0
- examples/f970d28ac08b_left_0/thumb.jpg +0 -0
- measure_reference.py +147 -0
- race_view.css +65 -14
- race_view.js +211 -104
- race_view.py +8 -6
- reference.json +55 -0
- reference.py +27 -0
HOW_IT_WORKS.md
CHANGED
|
@@ -1,24 +1,39 @@
|
|
| 1 |
### What is timed
|
| 2 |
|
| 3 |
-
All three read the same region, one after the other on the same GPU (ZeroGPU: half an RTX PRO 6000 Blackwell)
|
| 4 |
-
is timed from the images to the text, preprocessing included, after a
|
|
|
|
| 5 |
|
| 6 |
- **Swedish Lion TrOCR** (Riksarkivet's Swedish Lion Libre 2026, large, character-based) cannot read a region. It
|
| 7 |
-
gets the region as its text lines, cut out beforehand along the ground truth's line outlines, and reads them
|
| 8 |
-
|
| 9 |
-
step is not timed here.
|
| 10 |
- **MiniCPM-OCR reader alone** reads the region as one image, one token per pass, greedily.
|
| 11 |
- **Reader + DSpark draft**: the draft proposes 15 tokens at a time; the reader checks them in one pass and keeps the
|
| 12 |
ones it agrees with, plus one of its own. Greedy decoding, so the text is the reader's own.
|
| 13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
### What you see
|
| 15 |
|
| 16 |
-
Each reader notes when every token became known
|
| 17 |
-
|
| 18 |
-
slows no reader down.
|
| 19 |
|
| 20 |
-
- On each manuscript a highlighter follows the reader: TrOCR's
|
| 21 |
-
|
| 22 |
- The strip above a reader's text has one block per reader pass, as wide as the tokens that pass produced: many thin
|
| 23 |
ones alone, a few wide ones with the draft.
|
| 24 |
- In the DSpark lane, shaded text was proposed by the draft and kept; underlined text is the reader's own token, one
|
|
@@ -45,23 +60,41 @@ the time. So:
|
|
| 45 |
cannot be cut back after rejected guesses. The checking pass keeps each such layer's inputs for the block, and the
|
| 46 |
state is rebuilt over the accepted tokens only (a few kernels instead of a second pass).
|
| 47 |
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
|
| 52 |
-
|
| 53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
-
|
|
|
|
|
|
|
| 56 |
|
| 57 |
-
On 1,000 test lines and 200 dev regions, one
|
|
|
|
| 58 |
|
| 59 |
-
| | one at a time |
|
| 60 |
|---|---|---|
|
| 61 |
-
| lines:
|
| 62 |
-
|
|
|
|
|
| 63 |
|
| 64 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
### Jev
|
| 67 |
|
|
@@ -81,3 +114,5 @@ does not prove a reading right.
|
|
| 81 |
where the loop starts.
|
| 82 |
- Timings vary with the GPU slice and the queue. A fresh GPU worker captures its CUDA graphs first (a few seconds, not
|
| 83 |
counted in the timings shown).
|
|
|
|
|
|
|
|
|
| 1 |
### What is timed
|
| 2 |
|
| 3 |
+
All three read the same region, one after the other on the same GPU (ZeroGPU: half an RTX PRO 6000 Blackwell), with
|
| 4 |
+
**one stream each and nothing batched**. Each is timed from the images to the text, preprocessing included, after a
|
| 5 |
+
warm-up that is not timed:
|
| 6 |
|
| 7 |
- **Swedish Lion TrOCR** (Riksarkivet's Swedish Lion Libre 2026, large, character-based) cannot read a region. It
|
| 8 |
+
gets the region as its text lines, cut out beforehand along the ground truth's line outlines, and reads them one
|
| 9 |
+
after the other, greedily, in float32 as Riksarkivet runs it. Finding the lines is a model of its own in practice;
|
| 10 |
+
that step is not timed here.
|
| 11 |
- **MiniCPM-OCR reader alone** reads the region as one image, one token per pass, greedily.
|
| 12 |
- **Reader + DSpark draft**: the draft proposes 15 tokens at a time; the reader checks them in one pass and keeps the
|
| 13 |
ones it agrees with, plus one of its own. Greedy decoding, so the text is the reader's own.
|
| 14 |
|
| 15 |
+
### Comparing them fairly
|
| 16 |
+
|
| 17 |
+
A batch lets a GPU decode many sequences side by side. TrOCR can batch a region's lines. The reader reads a region as
|
| 18 |
+
one sequence, which has nothing to batch; what it batches is whole regions, in vLLM, which this Space cannot run.
|
| 19 |
+
Racing a batch of 8 to 18 lines against one sequence would compare two different things, so there are three views,
|
| 20 |
+
each with one rule for everyone:
|
| 21 |
+
|
| 22 |
+
- **Nothing batched** (the race): TrOCR reads its lines one after the other, the reader one region.
|
| 23 |
+
- **TrOCR's lines as one batch** is timed too and drawn as the second bar, so nothing is hidden: on these regions it
|
| 24 |
+
is about as fast as the reader alone in this Space's PyTorch, and 2 to 4 times slower than the reader with its
|
| 25 |
+
draft.
|
| 26 |
+
- **Everything batched** (the three rates under the race): on our server, TrOCR in batches of 32 lines against the
|
| 27 |
+
reader with 64 requests at a time in vLLM, which reads about ten times as many lines a second.
|
| 28 |
+
|
| 29 |
### What you see
|
| 30 |
|
| 31 |
+
Each reader notes when every token became known. Once all three are timed, the page replays their readings from the
|
| 32 |
+
same start, each at its measured pace; the buttons replay it, also at half and quarter speed. Nothing is sent to the
|
| 33 |
+
page inside a timed stretch, so the display slows no reader down.
|
| 34 |
|
| 35 |
+
- On each manuscript a highlighter follows the reader: TrOCR's goes down the lines one by one, a character per
|
| 36 |
+
decoding step; the reader's follows its tokens through the region.
|
| 37 |
- The strip above a reader's text has one block per reader pass, as wide as the tokens that pass produced: many thin
|
| 38 |
ones alone, a few wide ones with the draft.
|
| 39 |
- In the DSpark lane, shaded text was proposed by the draft and kept; underlined text is the reader's own token, one
|
|
|
|
| 60 |
cannot be cut back after rejected guesses. The checking pass keeps each such layer's inputs for the block, and the
|
| 61 |
state is rebuilt over the accepted tokens only (a few kernels instead of a second pass).
|
| 62 |
|
| 63 |
+
### The same region on our server, and vLLM
|
| 64 |
+
|
| 65 |
+
The outlined bars under the race were measured beforehand on our server (`measure_reference.py`, `reference.json`):
|
| 66 |
+
one whole RTX PRO 6000 Blackwell with nothing else running on it, TrOCR in transformers as here, the reader served by
|
| 67 |
+
vLLM alone and with its draft, one request at a time, the median of 5 runs. vLLM keeps the processed form of an image
|
| 68 |
+
it has seen, which saves 0.05 to 0.1 s a region, so every request sent the region with one pixel changed by one
|
| 69 |
+
level: new to the server, as a region is in production. Over the six example regions:
|
| 70 |
|
| 71 |
+
| a region read by | seconds |
|
| 72 |
+
|---|---|
|
| 73 |
+
| TrOCR, one line after the other | 1.25 – 3.92 |
|
| 74 |
+
| TrOCR, its lines as one batch | 0.60 – 1.54 |
|
| 75 |
+
| the reader in vLLM | 0.38 – 0.73 |
|
| 76 |
+
| the reader with its draft in vLLM | 0.27 – 0.45 |
|
| 77 |
+
| the reader in this Space's PyTorch, on that GPU | 0.53 – 1.21 |
|
| 78 |
+
| the reader with its draft in this Space's PyTorch, on that GPU | 0.23 – 0.41 |
|
| 79 |
|
| 80 |
+
So with the draft this Space's decoding is as fast as vLLM's for one region at a time, and alone it is 1.2 to 1.8
|
| 81 |
+
times slower; the live bars are slower again because ZeroGPU gives half a GPU. (The PyTorch timings start from the
|
| 82 |
+
decoded image, vLLM's from the request.) The reader alone writes the same tokens as transformers' own `generate`.
|
| 83 |
|
| 84 |
+
What vLLM adds is reading many requests at once. On the first 1,000 test lines and 200 dev regions, one RTX PRO
|
| 85 |
+
6000 Blackwell:
|
| 86 |
|
| 87 |
+
| | one at a time | many at a time |
|
| 88 |
|---|---|---|
|
| 89 |
+
| lines: Swedish Lion TrOCR | 4.3 lines/s (batch 1) | 6.5 lines/s (batch 32) |
|
| 90 |
+
| lines: reader alone → with DSpark | 13.3 → 18.2 lines/s (1.37×) | 67 → 77 lines/s (64 at a time) |
|
| 91 |
+
| regions: reader alone → with DSpark | 1.95 → 4.08 regions/s (2.09×) | 11.9 → 12.7 regions/s (64 at a time) |
|
| 92 |
|
| 93 |
+
The 64-at-a-time rows are from a run where every image was new to the servers (two servers on one GPU, asked in
|
| 94 |
+
turn); with each server on a GPU of its own they were 75 → 82 lines/s and 12.4 → 13.2 regions/s. Acceptance length
|
| 95 |
+
8.72; the character error rate is the same with and without the draft. OpenBMB's deployment guide also serves the
|
| 96 |
+
reader and a DSpark draft with SGLang (through a pull request still open there in October 2026); we have measured
|
| 97 |
+
vLLM only.
|
| 98 |
|
| 99 |
### Jev
|
| 100 |
|
|
|
|
| 114 |
where the loop starts.
|
| 115 |
- Timings vary with the GPU slice and the queue. A fresh GPU worker captures its CUDA graphs first (a few seconds, not
|
| 116 |
counted in the timings shown).
|
| 117 |
+
- With the draft the reader's text can differ from its text alone by a token where two tokens were nearly tied: a
|
| 118 |
+
check of 16 tokens rounds differently from a pass of one. vLLM shows the same between 1 and 64 requests at a time.
|
README.md
CHANGED
|
@@ -19,16 +19,20 @@ models:
|
|
| 19 |
|
| 20 |
# Swedish HTR speed and Jev
|
| 21 |
|
| 22 |
-
Reads the same handwritten text region
|
| 23 |
-
Swedish Lion TrOCR on the region's text lines, the
|
| 24 |
[MiniCPM-OCR HTR reader](https://huggingface.co/Gabriel/diffu-minicpm-ocr-htr) on the region as one image, and the same
|
| 25 |
-
reader with its [DSpark draft](https://huggingface.co/Gabriel/diffu-minicpm-ocr-htr-dspark).
|
| 26 |
-
|
| 27 |
-
|
|
|
|
| 28 |
|
| 29 |
- `app.py`: the Gradio app; the readers run one after the other and the page replays them at their measured pace.
|
| 30 |
- `dspark_hf.py`: DSpark speculative decoding in PyTorch on CUDA graphs (ZeroGPU cannot host vLLM).
|
| 31 |
-
- `race_view.py`, `race_view.js`, `race_view.css`: the custom component that draws the race
|
|
|
|
|
|
|
|
|
|
| 32 |
- `check_decode.py`: compares the reader alone and with its draft on a GPU, token for token, and times them.
|
| 33 |
|
| 34 |
See the app's "How it works" tab. Part of [diffu](https://github.com/Borg93/diffu2).
|
|
|
|
| 19 |
|
| 20 |
# Swedish HTR speed and Jev
|
| 21 |
|
| 22 |
+
Reads the same handwritten text region with one stream each on the same GPU, times each and replays them from the same
|
| 23 |
+
start: the Swedish Lion TrOCR on the region's text lines, one after the other (and once more as one batch), the
|
| 24 |
[MiniCPM-OCR HTR reader](https://huggingface.co/Gabriel/diffu-minicpm-ocr-htr) on the region as one image, and the same
|
| 25 |
+
reader with its [DSpark draft](https://huggingface.co/Gabriel/diffu-minicpm-ocr-htr-dspark). Under the race: the same
|
| 26 |
+
region's timings on our server, where vLLM serves the reader, and what the readers do there when everything is batched.
|
| 27 |
+
Then [Jev](https://huggingface.co/Gabriel/diffu-minicpm-ocr-htr-jev), the reader's proof-reader, scores both readers'
|
| 28 |
+
reading of each line.
|
| 29 |
|
| 30 |
- `app.py`: the Gradio app; the readers run one after the other and the page replays them at their measured pace.
|
| 31 |
- `dspark_hf.py`: DSpark speculative decoding in PyTorch on CUDA graphs (ZeroGPU cannot host vLLM).
|
| 32 |
+
- `race_view.py`, `race_view.js`, `race_view.css`: the custom component that draws the race, our server's timings and
|
| 33 |
+
Jev's rows.
|
| 34 |
+
- `reference.json`, `reference.py`, `measure_reference.py`: each example region's timings on our server (TrOCR in
|
| 35 |
+
transformers, the reader in vLLM), and the script that measured them.
|
| 36 |
- `check_decode.py`: compares the reader alone and with its draft on a GPU, token for token, and times them.
|
| 37 |
|
| 38 |
See the app's "How it works" tab. Part of [diffu](https://github.com/Borg93/diffu2).
|
app.py
CHANGED
|
@@ -3,14 +3,18 @@ draft, timed on the same region and replayed side by side; and Jev, the reader's
|
|
| 3 |
|
| 4 |
Everything runs in PyTorch on ZeroGPU (vLLM, where the draft normally runs, cannot run here): the reader's passes and
|
| 5 |
the draft's proposals are replayed from CUDA graphs (dspark_hf.py), for the reader alone and with the draft alike.
|
| 6 |
-
The readers run one after the other
|
| 7 |
-
|
|
|
|
|
|
|
|
|
|
| 8 |
"""
|
| 9 |
|
| 10 |
# ZeroGPU patches CUDA when `spaces` is imported, so it comes before torch and anything that imports torch.
|
| 11 |
import spaces # isort: skip
|
| 12 |
|
| 13 |
import base64
|
|
|
|
| 14 |
import io
|
| 15 |
import json
|
| 16 |
import os
|
|
@@ -18,14 +22,14 @@ import threading
|
|
| 18 |
import time
|
| 19 |
from collections.abc import Callable, Iterator
|
| 20 |
from pathlib import Path
|
| 21 |
-
from typing import Any, cast
|
| 22 |
from uuid import uuid4
|
| 23 |
|
| 24 |
import gradio as gr
|
| 25 |
import torch
|
| 26 |
from huggingface_hub import snapshot_download
|
| 27 |
from huggingface_hub.errors import HfHubHTTPError
|
| 28 |
-
from PIL import Image
|
| 29 |
from pydantic import BaseModel, Field
|
| 30 |
from rapidfuzz.distance import Levenshtein
|
| 31 |
from transformers import (
|
|
@@ -42,6 +46,7 @@ import dspark_hf as D
|
|
| 42 |
from jev_prompt import model_prompt
|
| 43 |
from jev_render import render_text, renderer_version
|
| 44 |
from race_view import RaceView
|
|
|
|
| 45 |
|
| 46 |
READER = "Gabriel/diffu-minicpm-ocr-htr"
|
| 47 |
DRAFT = "Gabriel/diffu-minicpm-ocr-htr-dspark"
|
|
@@ -51,6 +56,7 @@ TROCR = os.environ.get("TROCR_REPO", "Gabriel/swedish-lion-trocr-large-handwritt
|
|
| 51 |
HERE = Path(__file__).parent
|
| 52 |
FONT = HERE / "DejaVuSans.ttf"
|
| 53 |
EXAMPLES: list[dict[str, Any]] = json.loads((HERE / "examples" / "examples.json").read_text(encoding="utf-8"))
|
|
|
|
| 54 |
MAX_LEN = 4096 # the reader's cache: a region's prompt (up to about 2,600 tokens) and its reading; shorter is faster
|
| 55 |
MINICPM_IMAGES = {"downsample_mode": "4x", "max_slice_nums": 9} # OpenBMB's settings, as the reader was trained
|
| 56 |
# new tokens at most: an example region, an upload (it must fit its GPU slot), a line
|
|
@@ -83,17 +89,20 @@ gr.set_static_paths([HERE / "examples"]) # the line crops in Jev's rows
|
|
| 83 |
|
| 84 |
|
| 85 |
class TrocrLine(BaseModel):
|
| 86 |
-
"""A line's reading
|
|
|
|
| 87 |
|
| 88 |
text: str
|
|
|
|
| 89 |
ends: list[int]
|
| 90 |
|
| 91 |
|
| 92 |
class TrocrLane(BaseModel):
|
| 93 |
-
"""TrOCR's reading of a region's lines
|
|
|
|
| 94 |
|
| 95 |
seconds: float
|
| 96 |
-
|
| 97 |
lines: list[TrocrLine]
|
| 98 |
cer: float | None = None
|
| 99 |
|
|
@@ -118,24 +127,53 @@ class ReaderLane(BaseModel):
|
|
| 118 |
return "".join(piece for _, piece, _ in self.pieces).strip()
|
| 119 |
|
| 120 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
class JevRow(BaseModel):
|
| 122 |
-
"""One line: its crop, its
|
| 123 |
-
|
| 124 |
|
| 125 |
index: int
|
| 126 |
image: str
|
| 127 |
truth: str
|
| 128 |
-
trocr:
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
minicpm_score: float
|
| 133 |
-
minicpm_exact: bool
|
| 134 |
|
| 135 |
|
| 136 |
class Race(BaseModel):
|
| 137 |
"""What RaceView draws: the region with its lines' outlines (fractions of the image) and each line's cumulative
|
| 138 |
-
share of the transcription, the readers' lanes as they
|
|
|
|
| 139 |
|
| 140 |
run: str = Field(default_factory=lambda: uuid4().hex[:12])
|
| 141 |
image: str | None = None
|
|
@@ -144,26 +182,52 @@ class Race(BaseModel):
|
|
| 144 |
shares: list[float] = Field(default_factory=list)
|
| 145 |
status: str = ""
|
| 146 |
notice: str = ""
|
|
|
|
| 147 |
done: bool = False
|
| 148 |
trocr: TrocrLane | None = None
|
| 149 |
plain: ReaderLane | None = None
|
| 150 |
dspark: ReaderLane | None = None
|
| 151 |
summary: str = ""
|
|
|
|
|
|
|
|
|
|
|
|
|
| 152 |
jev: list[JevRow] = Field(default_factory=list)
|
| 153 |
jev_total: int = 0
|
| 154 |
jev_note: str = ""
|
| 155 |
|
| 156 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
class StepClock(StoppingCriteria):
|
| 158 |
"""Notes when each decoding step ends; it never stops the generation itself."""
|
| 159 |
|
| 160 |
def __init__(self) -> None:
|
| 161 |
self.times: list[float] = []
|
|
|
|
| 162 |
|
| 163 |
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs: Any) -> torch.BoolTensor:
|
| 164 |
self.times.append(time.perf_counter())
|
| 165 |
-
|
| 166 |
-
|
|
|
|
| 167 |
|
| 168 |
|
| 169 |
def _example(example_id: str) -> dict[str, Any]:
|
|
@@ -195,40 +259,51 @@ def _region_inputs(image: Image.Image) -> dict[str, Any]:
|
|
| 195 |
return inputs
|
| 196 |
|
| 197 |
|
| 198 |
-
def
|
| 199 |
-
"""TrOCR's greedy
|
| 200 |
-
|
| 201 |
-
|
| 202 |
model, images = trocr, trocr_processor
|
| 203 |
if model is None or images is None:
|
| 204 |
raise RuntimeError("TrOCR is not loaded")
|
| 205 |
-
torch.cuda.synchronize()
|
| 206 |
-
started = time.perf_counter()
|
| 207 |
-
clock = StepClock()
|
| 208 |
pixels = images(images=lines, return_tensors="pt").pixel_values.to("cuda")
|
| 209 |
-
|
| 210 |
pixels,
|
| 211 |
decoder_start_token_id=model.config.decoder_start_token_id,
|
| 212 |
interpolate_pos_encoding=True,
|
| 213 |
num_beams=1,
|
| 214 |
max_new_tokens=max_new_tokens,
|
| 215 |
-
stopping_criteria=StoppingCriteriaList([clock]),
|
| 216 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 217 |
torch.cuda.synchronize()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
seconds = time.perf_counter() - started
|
| 219 |
-
|
| 220 |
tokenizer = images.tokenizer
|
| 221 |
read = []
|
| 222 |
-
for
|
| 223 |
-
text = tokenizer.decode(
|
| 224 |
ends, shown = [], 0
|
| 225 |
for step in range(1, len(steps) + 1):
|
| 226 |
if shown < len(text):
|
| 227 |
-
partial = tokenizer.decode(
|
| 228 |
shown = max(shown, min(len(text), len(partial)))
|
| 229 |
ends.append(shown)
|
| 230 |
-
read.append(TrocrLine(text=text, ends=ends))
|
| 231 |
-
return TrocrLane(seconds=seconds,
|
| 232 |
|
| 233 |
|
| 234 |
def _reader_lane(decoded: D.Decoded, seconds: float) -> ReaderLane:
|
|
@@ -284,7 +359,7 @@ def _engine() -> D.FastDSpark:
|
|
| 284 |
warm = Image.open(HERE / "examples" / EXAMPLES[0]["id"] / "region.png").convert("RGB")
|
| 285 |
engine.warm(_region_inputs(warm), EOS)
|
| 286 |
if trocr is not None:
|
| 287 |
-
|
| 288 |
_jev_score(_jev_crop(warm.crop((0, 0, 200, 40))), "warm")
|
| 289 |
_engines[os.getpid()] = engine
|
| 290 |
return engine
|
|
@@ -299,51 +374,115 @@ def _timed[T](fn: Callable[[], T]) -> tuple[T, float]:
|
|
| 299 |
|
| 300 |
|
| 301 |
def _summary(race: Race, identical: bool) -> str:
|
| 302 |
-
plain, spec = race.plain, race.dspark
|
| 303 |
if plain is None or spec is None:
|
| 304 |
return ""
|
| 305 |
-
|
| 306 |
-
|
| 307 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 308 |
if identical
|
| 309 |
else "Its reading differs from the reader's alone at a near-tied token: the reader checks 16 tokens at once, "
|
| 310 |
-
"and a batch of another size can tip a tie the other way (the reader alone does the same between batch sizes)
|
| 311 |
-
)
|
| 312 |
-
return (
|
| 313 |
-
f"With the draft, the reader needed {spec.passes} passes instead of {plain.passes} for {spec.tokens} tokens, "
|
| 314 |
-
f"keeping {spec.acceptance or 0:.1f} tokens per check on average (at most 16): "
|
| 315 |
-
f"{plain.seconds / spec.seconds:.1f}× faster than alone{against_trocr}. {same}"
|
| 316 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 317 |
|
| 318 |
|
| 319 |
def _jev_note(race: Race, truth: list[str]) -> str:
|
|
|
|
| 320 |
whole = " ".join(truth)
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
| 324 |
-
|
| 325 |
-
|
| 326 |
-
|
| 327 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 328 |
)
|
|
|
|
| 329 |
|
| 330 |
|
| 331 |
-
def _duration(example_id: str, upload: Image.Image | None, with_jev: bool) -> int:
|
| 332 |
-
return 75 if upload is not None else
|
| 333 |
|
| 334 |
|
| 335 |
@spaces.GPU(duration=_duration)
|
| 336 |
-
def compare(example_id: str, upload: Image.Image | None, with_jev: bool) -> Iterator[Race]:
|
| 337 |
-
"""Read one region
|
| 338 |
-
|
| 339 |
-
graphs). For an example region, the reader then reads each line on
|
| 340 |
-
TrOCR's. Yields the race so far between the timed stretches."""
|
| 341 |
race = Race(status="Preparing the GPU worker: a fresh one captures its CUDA graphs first…")
|
| 342 |
yield race
|
| 343 |
with _lock:
|
| 344 |
engine = _engine()
|
| 345 |
-
if upload is not None:
|
| 346 |
-
region, lines, truth, limit =
|
| 347 |
else:
|
| 348 |
folder = HERE / "examples" / example_id
|
| 349 |
truth = _example(example_id)["lines"]
|
|
@@ -355,7 +494,7 @@ def compare(example_id: str, upload: Image.Image | None, with_jev: bool) -> Iter
|
|
| 355 |
if trocr is None:
|
| 356 |
race.notice = NO_TROCR
|
| 357 |
elif lines:
|
| 358 |
-
race.status = f"TrOCR is reading the region's {len(lines)} lines…"
|
| 359 |
yield race
|
| 360 |
race.trocr = _trocr_read(lines)
|
| 361 |
race.trocr.cer = _cer("\n".join(race.trocr.texts), whole) if whole else None
|
|
@@ -372,63 +511,84 @@ def compare(example_id: str, upload: Image.Image | None, with_jev: bool) -> Iter
|
|
| 372 |
race.dspark = _reader_lane(spec, seconds)
|
| 373 |
race.dspark.cer = _cer(race.dspark.text, whole) if whole else None
|
| 374 |
race.summary = _summary(race, identical=plain.tokens == spec.tokens)
|
|
|
|
| 375 |
|
| 376 |
-
if with_jev and race.trocr is not None and truth is not None:
|
| 377 |
race.jev_total = len(lines)
|
| 378 |
race.status = "The reader reads each line on its own, and Jev scores its reading and TrOCR's…"
|
| 379 |
yield race
|
| 380 |
for i, line in enumerate(lines):
|
| 381 |
-
crop = _jev_crop(line)
|
| 382 |
-
guess = race.trocr.lines[i].text
|
| 383 |
own = _text(engine.decode(_region_inputs(line), LINE_TOKENS, EOS).tokens)
|
| 384 |
-
|
| 385 |
-
|
| 386 |
-
image=f"/gradio_api/file={HERE / 'examples' / example_id / f'line_{i:02d}.jpg'}",
|
| 387 |
-
truth=truth[i],
|
| 388 |
-
trocr=guess,
|
| 389 |
-
trocr_score=_jev_score(crop, guess),
|
| 390 |
-
trocr_exact=_same(guess, truth[i]),
|
| 391 |
-
minicpm=own,
|
| 392 |
-
minicpm_score=_jev_score(crop, own),
|
| 393 |
-
minicpm_exact=_same(own, truth[i]),
|
| 394 |
-
)
|
| 395 |
-
race.jev.append(row)
|
| 396 |
yield race
|
| 397 |
race.jev_note = _jev_note(race, truth)
|
| 398 |
race.status, race.done = "", True
|
| 399 |
yield race
|
| 400 |
|
| 401 |
|
| 402 |
-
def _stage(example_id: str, upload: Image.Image | None) -> Race:
|
| 403 |
-
"""The race before anyone has read: the region's image and, for an example, its lines' outlines
|
| 404 |
-
|
| 405 |
-
|
|
|
|
| 406 |
shown.thumbnail((1000, 1000))
|
| 407 |
buffer = io.BytesIO()
|
| 408 |
shown.save(buffer, format="JPEG", quality=85)
|
| 409 |
image = "data:image/jpeg;base64," + base64.b64encode(buffer.getvalue()).decode()
|
| 410 |
-
return Race(image=image, aspect=shown.width / shown.height)
|
| 411 |
example = _example(example_id)
|
| 412 |
width, height = example["region_size"]
|
| 413 |
lengths = [max(1, len(line)) for line in example["lines"]]
|
| 414 |
shares = [sum(lengths[: i + 1]) / sum(lengths) for i in range(len(lengths))]
|
|
|
|
| 415 |
return Race(
|
| 416 |
image=f"/gradio_api/file={HERE / 'examples' / example_id / 'region.png'}",
|
| 417 |
aspect=width / height,
|
| 418 |
outlines=example["outlines"],
|
| 419 |
shares=shares,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 420 |
)
|
| 421 |
|
| 422 |
|
| 423 |
-
def
|
| 424 |
-
|
| 425 |
-
|
| 426 |
-
|
| 427 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 428 |
stage = _stage(example_id, upload)
|
| 429 |
stage.status = "Waiting for a GPU…"
|
| 430 |
yield stage.model_dump()
|
| 431 |
-
keep = {
|
|
|
|
|
|
|
|
|
|
| 432 |
for race in compare(example_id, upload, with_jev):
|
| 433 |
yield race.model_copy(update=keep).model_dump()
|
| 434 |
|
|
@@ -447,46 +607,58 @@ def score_line(line: Image.Image | None, text: str) -> tuple[dict[str, float], I
|
|
| 447 |
|
| 448 |
def _label(example: dict[str, Any]) -> str:
|
| 449 |
collection = example["collection"].removeprefix("GT-").rsplit("_", 1)[0].replace("_", " ")
|
| 450 |
-
return f"{
|
|
|
|
| 451 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 452 |
|
| 453 |
-
LABELS = {_label(e): e["id"] for e in EXAMPLES}
|
| 454 |
|
| 455 |
with gr.Blocks(title="Swedish HTR: TrOCR, MiniCPM-OCR and DSpark") as demo:
|
| 456 |
gr.Markdown(
|
| 457 |
"# Reading Swedish handwriting faster\n"
|
| 458 |
-
"
|
| 459 |
-
"TrOCR** cannot read a region, so it gets the region
|
| 460 |
-
|
| 461 |
-
f"
|
| 462 |
-
f"
|
| 463 |
-
"
|
| 464 |
-
"
|
|
|
|
| 465 |
)
|
| 466 |
with gr.Tab("Compare"):
|
| 467 |
with gr.Row():
|
| 468 |
-
with gr.Column(scale=
|
| 469 |
-
|
| 470 |
-
|
| 471 |
-
|
| 472 |
-
|
| 473 |
-
|
| 474 |
-
|
| 475 |
-
|
| 476 |
-
|
| 477 |
-
value=str(HERE / "examples" / EXAMPLES[0]["id"] / "region.png"),
|
| 478 |
-
label="Region",
|
| 479 |
-
interactive=False,
|
| 480 |
-
height=330,
|
| 481 |
)
|
| 482 |
-
|
| 483 |
-
|
| 484 |
-
|
| 485 |
-
|
| 486 |
-
|
| 487 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 488 |
)
|
| 489 |
-
|
|
|
|
|
|
|
| 490 |
with gr.Tab("Try Jev"):
|
| 491 |
gr.Markdown(
|
| 492 |
"Jev scores whether a text is exactly what a **single handwritten line** says. Upload a line crop and type "
|
|
|
|
| 3 |
|
| 4 |
Everything runs in PyTorch on ZeroGPU (vLLM, where the draft normally runs, cannot run here): the reader's passes and
|
| 5 |
the draft's proposals are replayed from CUDA graphs (dspark_hf.py), for the reader alone and with the draft alike.
|
| 6 |
+
The readers run one after the other, one stream each: TrOCR reads the region's lines one by one (and once more as
|
| 7 |
+
one batch, shown beside), the reader reads the region as one sequence. Each notes when every token became known, and
|
| 8 |
+
once all are timed the page replays them from the same start at that pace (race_view.py). Nothing is sent to the page
|
| 9 |
+
inside a timed stretch. Under the live timings the page shows the same region's on our server, where vLLM serves the
|
| 10 |
+
reader (reference.py), and what the readers do there when everything is batched.
|
| 11 |
"""
|
| 12 |
|
| 13 |
# ZeroGPU patches CUDA when `spaces` is imported, so it comes before torch and anything that imports torch.
|
| 14 |
import spaces # isort: skip
|
| 15 |
|
| 16 |
import base64
|
| 17 |
+
import filecmp
|
| 18 |
import io
|
| 19 |
import json
|
| 20 |
import os
|
|
|
|
| 22 |
import time
|
| 23 |
from collections.abc import Callable, Iterator
|
| 24 |
from pathlib import Path
|
| 25 |
+
from typing import Any, Literal, cast
|
| 26 |
from uuid import uuid4
|
| 27 |
|
| 28 |
import gradio as gr
|
| 29 |
import torch
|
| 30 |
from huggingface_hub import snapshot_download
|
| 31 |
from huggingface_hub.errors import HfHubHTTPError
|
| 32 |
+
from PIL import Image, ImageOps
|
| 33 |
from pydantic import BaseModel, Field
|
| 34 |
from rapidfuzz.distance import Levenshtein
|
| 35 |
from transformers import (
|
|
|
|
| 46 |
from jev_prompt import model_prompt
|
| 47 |
from jev_render import render_text, renderer_version
|
| 48 |
from race_view import RaceView
|
| 49 |
+
from reference import Reference, RegionTimes
|
| 50 |
|
| 51 |
READER = "Gabriel/diffu-minicpm-ocr-htr"
|
| 52 |
DRAFT = "Gabriel/diffu-minicpm-ocr-htr-dspark"
|
|
|
|
| 56 |
HERE = Path(__file__).parent
|
| 57 |
FONT = HERE / "DejaVuSans.ttf"
|
| 58 |
EXAMPLES: list[dict[str, Any]] = json.loads((HERE / "examples" / "examples.json").read_text(encoding="utf-8"))
|
| 59 |
+
REFERENCE = Reference.model_validate_json((HERE / "reference.json").read_text(encoding="utf-8"))
|
| 60 |
MAX_LEN = 4096 # the reader's cache: a region's prompt (up to about 2,600 tokens) and its reading; shorter is faster
|
| 61 |
MINICPM_IMAGES = {"downsample_mode": "4x", "max_slice_nums": 9} # OpenBMB's settings, as the reader was trained
|
| 62 |
# new tokens at most: an example region, an upload (it must fit its GPU slot), a line
|
|
|
|
| 89 |
|
| 90 |
|
| 91 |
class TrocrLine(BaseModel):
|
| 92 |
+
"""A line's reading, when each of its decoding steps ended (seconds from the start of the region's first line) and
|
| 93 |
+
how many of its characters each step had reached."""
|
| 94 |
|
| 95 |
text: str
|
| 96 |
+
steps: list[float]
|
| 97 |
ends: list[int]
|
| 98 |
|
| 99 |
|
| 100 |
class TrocrLane(BaseModel):
|
| 101 |
+
"""TrOCR's reading of a region's lines, one line after the other: its seconds and each line's steps; and the
|
| 102 |
+
seconds the same lines took as one batch."""
|
| 103 |
|
| 104 |
seconds: float
|
| 105 |
+
batch_seconds: float
|
| 106 |
lines: list[TrocrLine]
|
| 107 |
cer: float | None = None
|
| 108 |
|
|
|
|
| 127 |
return "".join(piece for _, piece, _ in self.pieces).strip()
|
| 128 |
|
| 129 |
|
| 130 |
+
class ServerTimes(RegionTimes):
|
| 131 |
+
"""A region's timings on our server, with its GPU, the day they were measured and the runs they are medians of."""
|
| 132 |
+
|
| 133 |
+
gpu: str
|
| 134 |
+
measured: str
|
| 135 |
+
runs: int
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
class Rate(BaseModel):
|
| 139 |
+
"""What a reader reads a second when everything is batched, and how it was batched."""
|
| 140 |
+
|
| 141 |
+
lane: Literal["trocr", "plain", "dspark"]
|
| 142 |
+
name: str
|
| 143 |
+
how: str
|
| 144 |
+
per_second: float
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
class JevReading(BaseModel):
|
| 148 |
+
"""One reader's reading of a line: its text in stretches, each with whether it agrees with the ground truth, Jev's
|
| 149 |
+
score, and whether the whole reading is exactly the ground truth."""
|
| 150 |
+
|
| 151 |
+
marks: list[tuple[str, bool]]
|
| 152 |
+
score: float
|
| 153 |
+
exact: bool
|
| 154 |
+
|
| 155 |
+
@property
|
| 156 |
+
def text(self) -> str:
|
| 157 |
+
return "".join(piece for piece, _ in self.marks)
|
| 158 |
+
|
| 159 |
+
|
| 160 |
class JevRow(BaseModel):
|
| 161 |
+
"""One line: its crop, its ground truth, TrOCR's reading and the reader's (each of the line on its own) with Jev's
|
| 162 |
+
scores, the reader Jev scores higher ("" where both read the same) and what that amounts to."""
|
| 163 |
|
| 164 |
index: int
|
| 165 |
image: str
|
| 166 |
truth: str
|
| 167 |
+
trocr: JevReading
|
| 168 |
+
minicpm: JevReading
|
| 169 |
+
pick: Literal["trocr", "minicpm", ""]
|
| 170 |
+
verdict: str
|
|
|
|
|
|
|
| 171 |
|
| 172 |
|
| 173 |
class Race(BaseModel):
|
| 174 |
"""What RaceView draws: the region with its lines' outlines (fractions of the image) and each line's cumulative
|
| 175 |
+
share of the transcription, the readers' lanes as they are timed (``ready`` once all are: the replay starts), the
|
| 176 |
+
same region's timings on our server and the batched rates, then Jev's rows."""
|
| 177 |
|
| 178 |
run: str = Field(default_factory=lambda: uuid4().hex[:12])
|
| 179 |
image: str | None = None
|
|
|
|
| 182 |
shares: list[float] = Field(default_factory=list)
|
| 183 |
status: str = ""
|
| 184 |
notice: str = ""
|
| 185 |
+
ready: bool = False
|
| 186 |
done: bool = False
|
| 187 |
trocr: TrocrLane | None = None
|
| 188 |
plain: ReaderLane | None = None
|
| 189 |
dspark: ReaderLane | None = None
|
| 190 |
summary: str = ""
|
| 191 |
+
server: ServerTimes | None = None
|
| 192 |
+
server_said: str = ""
|
| 193 |
+
batched: list[Rate] = Field(default_factory=list)
|
| 194 |
+
batched_note: str = ""
|
| 195 |
jev: list[JevRow] = Field(default_factory=list)
|
| 196 |
jev_total: int = 0
|
| 197 |
jev_note: str = ""
|
| 198 |
|
| 199 |
|
| 200 |
+
# Everything batched, on our server: the first 1,000 test lines, a whole GPU for TrOCR, the two vLLM servers sharing
|
| 201 |
+
# one and asked in turn, every image new to them (docs/HTR_RUNS.md: bench_three.sh on 9 Nov, bench_cold.sh on 11 Nov)
|
| 202 |
+
BATCHED = [
|
| 203 |
+
Rate(lane="trocr", name="Swedish Lion TrOCR", how="batches of 32 lines", per_second=6.54),
|
| 204 |
+
Rate(lane="plain", name="Reader alone, in vLLM", how="64 requests at a time", per_second=67.26),
|
| 205 |
+
Rate(lane="dspark", name="Reader + DSpark, in vLLM", how="64 requests at a time", per_second=77.15),
|
| 206 |
+
]
|
| 207 |
+
# This Space's decoding against vLLM on that server's GPU, the six example regions (docs/HTR_RUNS.md, 11 Nov 2026)
|
| 208 |
+
SAME_GPU = (
|
| 209 |
+
"On that same whole GPU this Space's PyTorch decoding is as fast as vLLM with the draft (0.23–0.41 s a region "
|
| 210 |
+
"against 0.27–0.45 s) and 1.2–1.8× slower alone; the rest of the gap to the live bars is the half GPU."
|
| 211 |
+
)
|
| 212 |
+
BATCHED_NOTE = (
|
| 213 |
+
"The first 1,000 test lines on one RTX PRO 6000 Blackwell, every image new to the server. One at a time on the "
|
| 214 |
+
"same lines: TrOCR 4.3 lines a second, the reader 13.3, with its draft 18.2. Whole regions, 64 at a time: 11.9 a "
|
| 215 |
+
"second with the reader alone, 12.7 with its draft."
|
| 216 |
+
)
|
| 217 |
+
|
| 218 |
+
|
| 219 |
class StepClock(StoppingCriteria):
|
| 220 |
"""Notes when each decoding step ends; it never stops the generation itself."""
|
| 221 |
|
| 222 |
def __init__(self) -> None:
|
| 223 |
self.times: list[float] = []
|
| 224 |
+
self._never: torch.Tensor | None = None # made once: a tensor a step would cost the reader it times
|
| 225 |
|
| 226 |
def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs: Any) -> torch.BoolTensor:
|
| 227 |
self.times.append(time.perf_counter())
|
| 228 |
+
if self._never is None:
|
| 229 |
+
self._never = torch.zeros(input_ids.shape[0], dtype=torch.bool, device=input_ids.device)
|
| 230 |
+
return cast(torch.BoolTensor, self._never)
|
| 231 |
|
| 232 |
|
| 233 |
def _example(example_id: str) -> dict[str, Any]:
|
|
|
|
| 259 |
return inputs
|
| 260 |
|
| 261 |
|
| 262 |
+
def _trocr_ids(lines: list[Image.Image], clock: StepClock | None = None, max_new_tokens: int = 128) -> torch.Tensor:
|
| 263 |
+
"""TrOCR's greedy token ids for a batch of lines, as Riksarkivet's evaluation runs it: the Swedish Lion starts
|
| 264 |
+
decoding from its config's token (its generation_config names another, which costs about 2 points of CER), and
|
| 265 |
+
its 192 x 1024 inputs need the position encoding interpolated."""
|
| 266 |
model, images = trocr, trocr_processor
|
| 267 |
if model is None or images is None:
|
| 268 |
raise RuntimeError("TrOCR is not loaded")
|
|
|
|
|
|
|
|
|
|
| 269 |
pixels = images(images=lines, return_tensors="pt").pixel_values.to("cuda")
|
| 270 |
+
return model.generate(
|
| 271 |
pixels,
|
| 272 |
decoder_start_token_id=model.config.decoder_start_token_id,
|
| 273 |
interpolate_pos_encoding=True,
|
| 274 |
num_beams=1,
|
| 275 |
max_new_tokens=max_new_tokens,
|
| 276 |
+
stopping_criteria=StoppingCriteriaList([clock] if clock else []),
|
| 277 |
)
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
def _trocr_read(lines: list[Image.Image]) -> TrocrLane:
|
| 281 |
+
"""TrOCR's readings of a region's lines, one line after the other, timed from the images to the token ids; then
|
| 282 |
+
the seconds the same lines take as one batch."""
|
| 283 |
+
images = trocr_processor
|
| 284 |
+
if images is None:
|
| 285 |
+
raise RuntimeError("TrOCR is not loaded")
|
| 286 |
torch.cuda.synchronize()
|
| 287 |
+
started = time.perf_counter()
|
| 288 |
+
rows = []
|
| 289 |
+
for line in lines:
|
| 290 |
+
clock = StepClock()
|
| 291 |
+
ids = _trocr_ids([line], clock)[0, 1:].tolist() # after the start token; the copy waits for the GPU
|
| 292 |
+
rows.append((ids, [round(at - started, 4) for at in clock.times]))
|
| 293 |
seconds = time.perf_counter() - started
|
| 294 |
+
_, batch_seconds = _timed(lambda: _trocr_ids(lines))
|
| 295 |
tokenizer = images.tokenizer
|
| 296 |
read = []
|
| 297 |
+
for ids, steps in rows:
|
| 298 |
+
text = tokenizer.decode(ids, skip_special_tokens=True).strip()
|
| 299 |
ends, shown = [], 0
|
| 300 |
for step in range(1, len(steps) + 1):
|
| 301 |
if shown < len(text):
|
| 302 |
+
partial = tokenizer.decode(ids[:step], skip_special_tokens=True).strip()
|
| 303 |
shown = max(shown, min(len(text), len(partial)))
|
| 304 |
ends.append(shown)
|
| 305 |
+
read.append(TrocrLine(text=text, steps=steps, ends=ends))
|
| 306 |
+
return TrocrLane(seconds=seconds, batch_seconds=batch_seconds, lines=read)
|
| 307 |
|
| 308 |
|
| 309 |
def _reader_lane(decoded: D.Decoded, seconds: float) -> ReaderLane:
|
|
|
|
| 359 |
warm = Image.open(HERE / "examples" / EXAMPLES[0]["id"] / "region.png").convert("RGB")
|
| 360 |
engine.warm(_region_inputs(warm), EOS)
|
| 361 |
if trocr is not None:
|
| 362 |
+
_trocr_ids([warm.crop((0, 0, 200, 40))], max_new_tokens=4)
|
| 363 |
_jev_score(_jev_crop(warm.crop((0, 0, 200, 40))), "warm")
|
| 364 |
_engines[os.getpid()] = engine
|
| 365 |
return engine
|
|
|
|
| 374 |
|
| 375 |
|
| 376 |
def _summary(race: Race, identical: bool) -> str:
|
| 377 |
+
plain, spec, lines = race.plain, race.dspark, race.trocr
|
| 378 |
if plain is None or spec is None:
|
| 379 |
return ""
|
| 380 |
+
said = [
|
| 381 |
+
f"One stream each, nothing batched: with its draft the reader read the region in {spec.seconds:.2f} s, "
|
| 382 |
+
f"{plain.seconds / spec.seconds:.1f}× faster than alone"
|
| 383 |
+
]
|
| 384 |
+
if lines is not None:
|
| 385 |
+
said[0] += (
|
| 386 |
+
f" and {lines.seconds / spec.seconds:.1f}× faster than TrOCR reading the {len(lines.lines)} lines one "
|
| 387 |
+
"after the other"
|
| 388 |
+
)
|
| 389 |
+
said.append(
|
| 390 |
+
f"As one batch the lines decode side by side, and TrOCR took {lines.batch_seconds:.2f} s "
|
| 391 |
+
f"({lines.batch_seconds / spec.seconds:.1f}× the reader with its draft); a region is one sequence, so "
|
| 392 |
+
"the reader has nothing to batch here. It batches whole regions, in vLLM"
|
| 393 |
+
)
|
| 394 |
+
said.append(
|
| 395 |
+
f"The draft: {spec.passes} reader passes instead of {plain.passes} for {spec.tokens} tokens, "
|
| 396 |
+
f"{spec.acceptance or 0:.1f} tokens kept per check on average (at most 16)"
|
| 397 |
+
)
|
| 398 |
+
said.append(
|
| 399 |
+
"It wrote exactly what the reader writes alone"
|
| 400 |
if identical
|
| 401 |
else "Its reading differs from the reader's alone at a near-tied token: the reader checks 16 tokens at once, "
|
| 402 |
+
"and a batch of another size can tip a tie the other way (the reader alone does the same between batch sizes)"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 403 |
)
|
| 404 |
+
return ". ".join(said) + "."
|
| 405 |
+
|
| 406 |
+
|
| 407 |
+
def _marks(reading: str, truth: str) -> list[tuple[str, bool]]:
|
| 408 |
+
"""A reading in stretches, each with whether it agrees with the ground truth; where the reading lacks characters,
|
| 409 |
+
a caret stands for them."""
|
| 410 |
+
reading, truth = " ".join(reading.split()), " ".join(truth.split())
|
| 411 |
+
marks = []
|
| 412 |
+
for step in Levenshtein.opcodes(reading, truth):
|
| 413 |
+
if step.tag == "insert":
|
| 414 |
+
marks.append(("‸", False))
|
| 415 |
+
else:
|
| 416 |
+
marks.append((reading[step.src_start : step.src_end], step.tag == "equal"))
|
| 417 |
+
return marks
|
| 418 |
+
|
| 419 |
+
|
| 420 |
+
def _jev_row(index: int, image: str, truth: str, crop: Image.Image, guess: str, own: str) -> JevRow:
|
| 421 |
+
"""A line's row: Jev's score of TrOCR's reading and of the reader's, and what its higher score amounts to."""
|
| 422 |
+
first = JevReading(marks=_marks(guess, truth), score=_jev_score(crop, guess), exact=_same(guess, truth))
|
| 423 |
+
if _same(guess, own):
|
| 424 |
+
second, pick = first, ""
|
| 425 |
+
verdict = "both right" if first.exact else "both wrong, the same way"
|
| 426 |
+
else:
|
| 427 |
+
second = JevReading(marks=_marks(own, truth), score=_jev_score(crop, own), exact=_same(own, truth))
|
| 428 |
+
pick = "minicpm" if second.score > first.score else "trocr"
|
| 429 |
+
if first.exact == second.exact:
|
| 430 |
+
verdict = "both wrong" # two different readings cannot both be the ground truth
|
| 431 |
+
else:
|
| 432 |
+
right = "trocr" if first.exact else "minicpm"
|
| 433 |
+
verdict = "Jev picks the right one" if pick == right else "Jev picks the wrong one"
|
| 434 |
+
return JevRow(index=index, image=image, truth=truth, trocr=first, minicpm=second, pick=pick, verdict=verdict)
|
| 435 |
|
| 436 |
|
| 437 |
def _jev_note(race: Race, truth: list[str]) -> str:
|
| 438 |
+
rows = race.jev
|
| 439 |
whole = " ".join(truth)
|
| 440 |
+
picked = [row.minicpm.text if row.pick == "minicpm" else row.trocr.text for row in rows]
|
| 441 |
+
cer = [
|
| 442 |
+
_cer(" ".join(texts), whole) for texts in ([r.trocr.text for r in rows], [r.minicpm.text for r in rows], picked)
|
| 443 |
+
]
|
| 444 |
+
one_right = [row for row in rows if row.trocr.exact != row.minicpm.exact]
|
| 445 |
+
said = [f"Both readers are right on {sum(row.verdict == 'both right' for row in rows)} of {len(rows)} lines"]
|
| 446 |
+
hits = sum(row.verdict == "Jev picks the right one" for row in one_right)
|
| 447 |
+
if len(one_right) == 1:
|
| 448 |
+
said.append(
|
| 449 |
+
f"On the one where only one of them is right, Jev scored the {'right' if hits else 'wrong'} one higher"
|
| 450 |
+
)
|
| 451 |
+
elif one_right:
|
| 452 |
+
said.append(
|
| 453 |
+
f"On the {len(one_right)} where only one of them is right, Jev scored the right one higher on {hits}"
|
| 454 |
+
)
|
| 455 |
+
readings = [reading for row in rows for reading in (row.trocr, row.minicpm)]
|
| 456 |
+
right = [reading.score for reading in readings if reading.exact]
|
| 457 |
+
wrong = [reading.score for reading in readings if not reading.exact]
|
| 458 |
+
if right and wrong:
|
| 459 |
+
said.append(
|
| 460 |
+
f"Jev's score averages {sum(right) / len(right):.2f} on right readings and "
|
| 461 |
+
f"{sum(wrong) / len(wrong):.2f} on wrong ones"
|
| 462 |
+
)
|
| 463 |
+
said.append(
|
| 464 |
+
f"CER over these lines, each read on its own: TrOCR {100 * cer[0]:.1f} %, MiniCPM-OCR {100 * cer[1]:.1f} %, "
|
| 465 |
+
f"Jev's pick on each line {100 * cer[2]:.1f} %"
|
| 466 |
)
|
| 467 |
+
return ". ".join(said) + "."
|
| 468 |
|
| 469 |
|
| 470 |
+
def _duration(example_id: str | None, upload: Image.Image | None, with_jev: bool) -> int:
|
| 471 |
+
return 75 if upload is not None else 55 + (15 if with_jev else 0)
|
| 472 |
|
| 473 |
|
| 474 |
@spaces.GPU(duration=_duration)
|
| 475 |
+
def compare(example_id: str | None, upload: Image.Image | None, with_jev: bool) -> Iterator[Race]:
|
| 476 |
+
"""Read one region with one stream each, one reader after the other, timing each from the images to the text:
|
| 477 |
+
TrOCR on its lines one by one (and once more as one batch), the MiniCPM-OCR reader alone, and the reader with its
|
| 478 |
+
DSpark draft (the last two on the same CUDA graphs). For an example region, the reader then reads each line on
|
| 479 |
+
its own, and Jev scores that reading and TrOCR's. Yields the race so far between the timed stretches."""
|
| 480 |
race = Race(status="Preparing the GPU worker: a fresh one captures its CUDA graphs first…")
|
| 481 |
yield race
|
| 482 |
with _lock:
|
| 483 |
engine = _engine()
|
| 484 |
+
if upload is not None or example_id is None:
|
| 485 |
+
region, lines, truth, limit = cast(Image.Image, upload), [], None, UPLOAD_TOKENS
|
| 486 |
else:
|
| 487 |
folder = HERE / "examples" / example_id
|
| 488 |
truth = _example(example_id)["lines"]
|
|
|
|
| 494 |
if trocr is None:
|
| 495 |
race.notice = NO_TROCR
|
| 496 |
elif lines:
|
| 497 |
+
race.status = f"TrOCR is reading the region's {len(lines)} lines, one after the other, then as one batch…"
|
| 498 |
yield race
|
| 499 |
race.trocr = _trocr_read(lines)
|
| 500 |
race.trocr.cer = _cer("\n".join(race.trocr.texts), whole) if whole else None
|
|
|
|
| 511 |
race.dspark = _reader_lane(spec, seconds)
|
| 512 |
race.dspark.cer = _cer(race.dspark.text, whole) if whole else None
|
| 513 |
race.summary = _summary(race, identical=plain.tokens == spec.tokens)
|
| 514 |
+
race.ready = True
|
| 515 |
|
| 516 |
+
if with_jev and race.trocr is not None and truth is not None and example_id is not None:
|
| 517 |
race.jev_total = len(lines)
|
| 518 |
race.status = "The reader reads each line on its own, and Jev scores its reading and TrOCR's…"
|
| 519 |
yield race
|
| 520 |
for i, line in enumerate(lines):
|
|
|
|
|
|
|
| 521 |
own = _text(engine.decode(_region_inputs(line), LINE_TOKENS, EOS).tokens)
|
| 522 |
+
image = f"/gradio_api/file={HERE / 'examples' / example_id / f'line_{i:02d}.jpg'}"
|
| 523 |
+
race.jev.append(_jev_row(i, image, truth[i], _jev_crop(line), race.trocr.lines[i].text, own))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 524 |
yield race
|
| 525 |
race.jev_note = _jev_note(race, truth)
|
| 526 |
race.status, race.done = "", True
|
| 527 |
yield race
|
| 528 |
|
| 529 |
|
| 530 |
+
def _stage(example_id: str | None, upload: Image.Image | None) -> Race:
|
| 531 |
+
"""The race before anyone has read: the region's image and, for an example, its lines' outlines and its timings
|
| 532 |
+
on our server."""
|
| 533 |
+
if upload is not None or example_id is None:
|
| 534 |
+
shown = cast(Image.Image, upload).copy()
|
| 535 |
shown.thumbnail((1000, 1000))
|
| 536 |
buffer = io.BytesIO()
|
| 537 |
shown.save(buffer, format="JPEG", quality=85)
|
| 538 |
image = "data:image/jpeg;base64," + base64.b64encode(buffer.getvalue()).decode()
|
| 539 |
+
return Race(image=image, aspect=shown.width / shown.height, batched=BATCHED, batched_note=BATCHED_NOTE)
|
| 540 |
example = _example(example_id)
|
| 541 |
width, height = example["region_size"]
|
| 542 |
lengths = [max(1, len(line)) for line in example["lines"]]
|
| 543 |
shares = [sum(lengths[: i + 1]) / sum(lengths) for i in range(len(lengths))]
|
| 544 |
+
times = REFERENCE.regions.get(example_id)
|
| 545 |
return Race(
|
| 546 |
image=f"/gradio_api/file={HERE / 'examples' / example_id / 'region.png'}",
|
| 547 |
aspect=width / height,
|
| 548 |
outlines=example["outlines"],
|
| 549 |
shares=shares,
|
| 550 |
+
server=ServerTimes(**times.model_dump(), gpu=REFERENCE.gpu, measured=REFERENCE.measured, runs=REFERENCE.runs)
|
| 551 |
+
if times
|
| 552 |
+
else None,
|
| 553 |
+
server_said=_server_said(times) if times else "",
|
| 554 |
+
batched=BATCHED,
|
| 555 |
+
batched_note=BATCHED_NOTE,
|
| 556 |
)
|
| 557 |
|
| 558 |
|
| 559 |
+
def _server_said(times: RegionTimes) -> str:
|
| 560 |
+
return (
|
| 561 |
+
f"Served by vLLM, the reader read this region in {times.vllm:.2f} s alone and {times.vllm_dspark:.2f} s with "
|
| 562 |
+
f"its draft: {times.trocr_single / times.vllm:.1f}× and {times.trocr_single / times.vllm_dspark:.1f}× faster "
|
| 563 |
+
f"than TrOCR reading the {times.lines} lines one after the other, {times.trocr_batch / times.vllm:.1f}× and "
|
| 564 |
+
f"{times.trocr_batch / times.vllm_dspark:.1f}× faster than TrOCR's one batch. {SAME_GPU}"
|
| 565 |
+
)
|
| 566 |
+
|
| 567 |
+
|
| 568 |
+
def _example_of(image: str) -> str | None:
|
| 569 |
+
"""The example whose region is the image in the box, or None for a visitor's own image."""
|
| 570 |
+
for example in EXAMPLES:
|
| 571 |
+
if filecmp.cmp(image, HERE / "examples" / example["id"] / "region.png", shallow=False):
|
| 572 |
+
return example["id"]
|
| 573 |
+
return None
|
| 574 |
+
|
| 575 |
+
|
| 576 |
+
def run(region: str | None, with_jev: bool) -> Iterator[dict[str, Any]]:
|
| 577 |
+
"""Read the region in the box (an example, or a crop of your own) with one stream each: TrOCR on an example's
|
| 578 |
+
lines, the MiniCPM-OCR reader and the reader with its DSpark draft; then let Jev score each line of an example.
|
| 579 |
+
Streams the race as each reader is timed: every reading with the time each token became known, the summary, the
|
| 580 |
+
same region's timings on our server, and Jev's rows."""
|
| 581 |
+
if region is None:
|
| 582 |
+
raise gr.Error("Pick an example region, or drop a region's crop on the box.")
|
| 583 |
+
example_id = _example_of(region)
|
| 584 |
+
upload = None if example_id else ImageOps.exif_transpose(Image.open(region)).convert("RGB")
|
| 585 |
stage = _stage(example_id, upload)
|
| 586 |
stage.status = "Waiting for a GPU…"
|
| 587 |
yield stage.model_dump()
|
| 588 |
+
keep = {
|
| 589 |
+
name: getattr(stage, name)
|
| 590 |
+
for name in ("run", "image", "aspect", "outlines", "shares", "server", "server_said", "batched", "batched_note")
|
| 591 |
+
}
|
| 592 |
for race in compare(example_id, upload, with_jev):
|
| 593 |
yield race.model_copy(update=keep).model_dump()
|
| 594 |
|
|
|
|
| 607 |
|
| 608 |
def _label(example: dict[str, Any]) -> str:
|
| 609 |
collection = example["collection"].removeprefix("GT-").rsplit("_", 1)[0].replace("_", " ")
|
| 610 |
+
return f"{len(example['lines'])} lines · {collection}"
|
| 611 |
+
|
| 612 |
|
| 613 |
+
def _region(example: dict[str, Any]) -> str:
|
| 614 |
+
return str(HERE / "examples" / example["id"] / "region.png")
|
| 615 |
+
|
| 616 |
+
|
| 617 |
+
def _picked(event: gr.SelectData) -> str:
|
| 618 |
+
return _region(EXAMPLES[event.index])
|
| 619 |
|
|
|
|
| 620 |
|
| 621 |
with gr.Blocks(title="Swedish HTR: TrOCR, MiniCPM-OCR and DSpark") as demo:
|
| 622 |
gr.Markdown(
|
| 623 |
"# Reading Swedish handwriting faster\n"
|
| 624 |
+
"One text region from an archive volume, read on the same GPU with **one stream each and nothing batched**. "
|
| 625 |
+
"Riksarkivet's **Swedish Lion TrOCR** cannot read a region, so it gets the region's text lines, cut out "
|
| 626 |
+
"beforehand, and reads them one after the other (its time for all of them as one batch is shown beside). Our "
|
| 627 |
+
f"**[MiniCPM-OCR reader](https://huggingface.co/{READER})** reads the region as one image, alone and with "
|
| 628 |
+
f"its **[DSpark draft](https://huggingface.co/{DRAFT})**, which guesses 15 tokens at a time for the reader "
|
| 629 |
+
f"to check in one pass. Then **[Jev](https://huggingface.co/{JEV})**, the reader's proof-reader, scores both "
|
| 630 |
+
"readers' reading of each line. Under the race: the same region on our server, where **vLLM** serves the "
|
| 631 |
+
"reader, and what the readers do there when everything is batched."
|
| 632 |
)
|
| 633 |
with gr.Tab("Compare"):
|
| 634 |
with gr.Row():
|
| 635 |
+
with gr.Column(scale=3):
|
| 636 |
+
region = gr.Image(
|
| 637 |
+
value=_region(EXAMPLES[0]),
|
| 638 |
+
type="filepath",
|
| 639 |
+
image_mode=None,
|
| 640 |
+
sources=["upload", "clipboard"],
|
| 641 |
+
label="The region to read: an example from below, or your own crop (upload or paste it)",
|
| 642 |
+
height=340,
|
| 643 |
+
buttons=["fullscreen"],
|
|
|
|
|
|
|
|
|
|
|
|
|
| 644 |
)
|
| 645 |
+
with gr.Column(scale=1, min_width=260):
|
| 646 |
+
with_jev = gr.Checkbox(value=True, label="Score each line with Jev (examples only)")
|
| 647 |
+
go = gr.Button("Read", variant="primary")
|
| 648 |
+
gr.Markdown("Only the reader reads a crop of your own: TrOCR and Jev need the region's lines.")
|
| 649 |
+
gallery = gr.Gallery(
|
| 650 |
+
[(str(HERE / "examples" / e["id"] / "thumb.jpg"), _label(e)) for e in EXAMPLES],
|
| 651 |
+
label="Example regions, from volumes none of the models trained on: click one to put it in the box",
|
| 652 |
+
columns=6,
|
| 653 |
+
height=230,
|
| 654 |
+
object_fit="cover",
|
| 655 |
+
allow_preview=False,
|
| 656 |
+
interactive=False,
|
| 657 |
+
buttons=[],
|
| 658 |
)
|
| 659 |
+
race_view = RaceView()
|
| 660 |
+
gallery.select(_picked, None, region, api_visibility="private")
|
| 661 |
+
go.click(run, [region, with_jev], race_view, api_name="compare", show_progress="minimal")
|
| 662 |
with gr.Tab("Try Jev"):
|
| 663 |
gr.Markdown(
|
| 664 |
"Jev scores whether a text is exactly what a **single handwritten line** says. Upload a line crop and type "
|
examples/19a1b2195bab_right_0/thumb.jpg
ADDED
|
|
examples/35ffe6b8b600_left_0/thumb.jpg
ADDED
|
|
examples/38c4ce6331ad_left_1/thumb.jpg
ADDED
|
|
examples/8d24e7df7dea_2/thumb.jpg
ADDED
|
|
examples/eed569a6699c_left_3/thumb.jpg
ADDED
|
|
examples/f970d28ac08b_left_0/thumb.jpg
ADDED
|
|
measure_reference.py
ADDED
|
@@ -0,0 +1,147 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The Space's reference timings: each example region read on this machine by the Swedish Lion TrOCR (transformers,
|
| 2 |
+
its lines one after the other and as one batch, as the app runs it) and by the reader in vLLM, alone and with its
|
| 3 |
+
DSpark draft, one request at a time. The app shows them beside its live race, which runs in PyTorch because ZeroGPU
|
| 4 |
+
cannot host vLLM.
|
| 5 |
+
|
| 6 |
+
Serve the reader twice first, as the draft's card says (prefix caching off), the second server once the first
|
| 7 |
+
answers, on a GPU nothing else is using:
|
| 8 |
+
|
| 9 |
+
vllm serve READER --served-model-name htr ... --port 8057
|
| 10 |
+
vllm serve READER --served-model-name htr ... --port 8056 --speculative-config '{"method": "dspark", ...}'
|
| 11 |
+
python measure_reference.py --examples examples --gpu "RTX PRO 6000 Blackwell" --out reference.json
|
| 12 |
+
|
| 13 |
+
Each reading is timed from the encoded images to the reading: the median of RUNS runs after one that is not timed.
|
| 14 |
+
vLLM keeps the processed form of an image it has seen, so every request sends the region with one pixel, drawn at
|
| 15 |
+
random, changed by one level: the region is new to the server each time, as a region is in production.
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
|
| 20 |
+
import argparse
|
| 21 |
+
import base64
|
| 22 |
+
import io
|
| 23 |
+
import json
|
| 24 |
+
import secrets
|
| 25 |
+
import statistics
|
| 26 |
+
import time
|
| 27 |
+
from collections.abc import Callable
|
| 28 |
+
from datetime import date
|
| 29 |
+
from pathlib import Path
|
| 30 |
+
from typing import Any
|
| 31 |
+
|
| 32 |
+
import httpx
|
| 33 |
+
import torch
|
| 34 |
+
from PIL import Image
|
| 35 |
+
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
|
| 36 |
+
|
| 37 |
+
from reference import Reference, RegionTimes
|
| 38 |
+
|
| 39 |
+
SAMPLING = {"temperature": 0.01, "top_p": 0.001, "top_k": 1, "repetition_penalty": 1.0} # OpenBMB's, near greedy
|
| 40 |
+
IMAGES = {"downsample_mode": "4x", "max_slice_nums": 9}
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def median_seconds(read: Callable[[int], None], runs: int) -> float:
|
| 44 |
+
"""The median time of `runs` calls of `read`, after one that is not timed; each call gets its own number."""
|
| 45 |
+
read(0)
|
| 46 |
+
times = []
|
| 47 |
+
for run in range(1, runs + 1):
|
| 48 |
+
started = time.perf_counter()
|
| 49 |
+
read(run)
|
| 50 |
+
times.append(time.perf_counter() - started)
|
| 51 |
+
return round(statistics.median(times), 3)
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def requests_for(region: Path, count: int) -> list[dict[str, Any]]:
|
| 55 |
+
"""`count` requests for one region, each with one pixel, drawn at random, changed by one level."""
|
| 56 |
+
image = Image.open(region).convert("RGB")
|
| 57 |
+
pixels = image.tobytes()
|
| 58 |
+
bodies = []
|
| 59 |
+
for _ in range(count):
|
| 60 |
+
changed = bytearray(pixels)
|
| 61 |
+
changed[3 * secrets.randbelow(image.width * image.height)] ^= 1 # a pixel's red
|
| 62 |
+
buffer = io.BytesIO()
|
| 63 |
+
Image.frombytes("RGB", image.size, bytes(changed)).save(buffer, format="PNG")
|
| 64 |
+
url = "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode()
|
| 65 |
+
content = [{"type": "image_url", "image_url": {"url": url}}, {"type": "text", "text": "Text Recognition:"}]
|
| 66 |
+
bodies.append(
|
| 67 |
+
{
|
| 68 |
+
"model": "htr",
|
| 69 |
+
**SAMPLING,
|
| 70 |
+
"max_tokens": 4096,
|
| 71 |
+
"chat_template_kwargs": {"enable_thinking": False},
|
| 72 |
+
"mm_processor_kwargs": IMAGES,
|
| 73 |
+
"messages": [{"role": "user", "content": content}],
|
| 74 |
+
}
|
| 75 |
+
)
|
| 76 |
+
return bodies
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
class Vllm:
|
| 80 |
+
"""One vLLM server, asked for one region at a time; it keeps what the region's requests answered."""
|
| 81 |
+
|
| 82 |
+
def __init__(self, endpoint: str, client: httpx.Client) -> None:
|
| 83 |
+
self.url = f"{endpoint}/v1/chat/completions"
|
| 84 |
+
self.client = client
|
| 85 |
+
self.texts: set[str] = set()
|
| 86 |
+
self.tokens = 0
|
| 87 |
+
|
| 88 |
+
def read(self, body: dict[str, Any]) -> None:
|
| 89 |
+
reply = self.client.post(self.url, json=body)
|
| 90 |
+
reply.raise_for_status()
|
| 91 |
+
answer = reply.json()
|
| 92 |
+
self.texts.add(answer["choices"][0]["message"]["content"])
|
| 93 |
+
self.tokens = answer["usage"]["completion_tokens"]
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def main() -> None:
|
| 97 |
+
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 98 |
+
parser.add_argument("--examples", type=Path, required=True, help="the folder build_examples.py wrote")
|
| 99 |
+
parser.add_argument("--trocr", default="Gabriel/swedish-lion-trocr-large-handwritten-char")
|
| 100 |
+
parser.add_argument("--alone", default="http://localhost:8057", help="vLLM serving the reader alone")
|
| 101 |
+
parser.add_argument("--draft", default="http://localhost:8056", help="vLLM serving the reader with its draft")
|
| 102 |
+
parser.add_argument("--gpu", required=True, help="the GPU's name, as the app will show it")
|
| 103 |
+
parser.add_argument("--runs", type=int, default=5)
|
| 104 |
+
parser.add_argument("--out", type=Path, required=True)
|
| 105 |
+
args = parser.parse_args()
|
| 106 |
+
|
| 107 |
+
processor = TrOCRProcessor.from_pretrained(args.trocr)
|
| 108 |
+
trocr = VisionEncoderDecoderModel.from_pretrained(args.trocr).to("cuda").eval()
|
| 109 |
+
|
| 110 |
+
def read_lines(files: list[bytes], batch: int) -> None:
|
| 111 |
+
lines = [Image.open(io.BytesIO(data)).convert("RGB") for data in files]
|
| 112 |
+
for first in range(0, len(lines), batch):
|
| 113 |
+
pixels = processor(images=lines[first : first + batch], return_tensors="pt").pixel_values.to("cuda")
|
| 114 |
+
with torch.inference_mode():
|
| 115 |
+
trocr.generate(
|
| 116 |
+
pixels,
|
| 117 |
+
decoder_start_token_id=trocr.config.decoder_start_token_id,
|
| 118 |
+
interpolate_pos_encoding=True,
|
| 119 |
+
num_beams=1,
|
| 120 |
+
max_new_tokens=128,
|
| 121 |
+
)
|
| 122 |
+
torch.cuda.synchronize()
|
| 123 |
+
|
| 124 |
+
regions = {}
|
| 125 |
+
with httpx.Client(timeout=600) as client:
|
| 126 |
+
for example in json.loads((args.examples / "examples.json").read_text(encoding="utf-8")):
|
| 127 |
+
folder = args.examples / example["id"]
|
| 128 |
+
files = [(folder / f"line_{i:02d}.jpg").read_bytes() for i in range(len(example["lines"]))]
|
| 129 |
+
bodies = requests_for(folder / "region.png", args.runs + 1)
|
| 130 |
+
alone, draft = Vllm(args.alone, client), Vllm(args.draft, client)
|
| 131 |
+
times = RegionTimes(
|
| 132 |
+
lines=len(files),
|
| 133 |
+
trocr_single=median_seconds(lambda _, files=files: read_lines(files, 1), args.runs),
|
| 134 |
+
trocr_batch=median_seconds(lambda _, files=files: read_lines(files, len(files)), args.runs),
|
| 135 |
+
vllm=median_seconds(lambda run, alone=alone, bodies=bodies: alone.read(bodies[run]), args.runs),
|
| 136 |
+
vllm_dspark=median_seconds(lambda run, draft=draft, bodies=bodies: draft.read(bodies[run]), args.runs),
|
| 137 |
+
tokens=alone.tokens,
|
| 138 |
+
)
|
| 139 |
+
regions[example["id"]] = times
|
| 140 |
+
same = "the same text" if alone.texts == draft.texts and len(alone.texts) == 1 else "texts differ"
|
| 141 |
+
print(example["id"], times.model_dump(), f"{len(alone.texts)} and {len(draft.texts)} readings, {same}")
|
| 142 |
+
reference = Reference(gpu=args.gpu, measured=date.today().isoformat(), runs=args.runs, regions=regions)
|
| 143 |
+
args.out.write_text(reference.model_dump_json(indent=1) + "\n", encoding="utf-8")
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
if __name__ == "__main__":
|
| 147 |
+
main()
|
race_view.css
CHANGED
|
@@ -2,10 +2,11 @@
|
|
| 2 |
--rv-trocr: #b07a2a;
|
| 3 |
--rv-plain: #1f62b0;
|
| 4 |
--rv-dspark: #1a8a5a;
|
|
|
|
| 5 |
--rv-muted: color-mix(in srgb, var(--body-text-color) 70%, transparent);
|
| 6 |
display: flex;
|
| 7 |
flex-direction: column;
|
| 8 |
-
gap:
|
| 9 |
color: var(--body-text-color);
|
| 10 |
font-size: 14px;
|
| 11 |
line-height: 1.5;
|
|
@@ -14,6 +15,7 @@
|
|
| 14 |
--rv-trocr: #b8842f;
|
| 15 |
--rv-plain: #4f8fdb;
|
| 16 |
--rv-dspark: #3fa97a;
|
|
|
|
| 17 |
}
|
| 18 |
.rv-root [hidden] { display: none; }
|
| 19 |
.rv-hint, .rv-status { color: var(--rv-muted); margin: 0; }
|
|
@@ -26,11 +28,15 @@
|
|
| 26 |
max-width: 100ch;
|
| 27 |
}
|
| 28 |
.rv-status::before { content: "● "; animation: rv-pulse 1.2s ease-in-out infinite; }
|
| 29 |
-
.rv-
|
|
|
|
|
|
|
|
|
|
| 30 |
.rv-k { font-size: 12px; color: var(--rv-muted); letter-spacing: 0.02em; }
|
| 31 |
|
|
|
|
| 32 |
.rv-track { display: flex; flex-direction: column; gap: 8px; }
|
| 33 |
-
.rv-row { display: grid; grid-template-columns: minmax(
|
| 34 |
.rv-name, .rv-head { display: flex; align-items: center; gap: 8px; min-width: 0; }
|
| 35 |
.rv-swatch { width: 10px; height: 10px; border-radius: 2px; flex: none; background: var(--rv-lane); }
|
| 36 |
[data-lane="trocr"] { --rv-lane: var(--rv-trocr); }
|
|
@@ -39,6 +45,27 @@
|
|
| 39 |
.rv-bar { height: 14px; border-radius: 4px; background: var(--background-fill-secondary); overflow: hidden; }
|
| 40 |
.rv-fillbar { height: 100%; width: 0; border-radius: 4px; background: var(--rv-lane); }
|
| 41 |
.rv-time { text-align: right; font-variant-numeric: tabular-nums; }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
.rv-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; }
|
| 44 |
.rv-button {
|
|
@@ -53,7 +80,7 @@
|
|
| 53 |
.rv-button:hover { border-color: var(--rv-muted); }
|
| 54 |
.rv-main { font-weight: 600; color: #fff; background: var(--rv-dspark); border-color: var(--rv-dspark); padding: 5px 16px; }
|
| 55 |
.rv-main:hover { filter: brightness(1.08); border-color: var(--rv-dspark); }
|
| 56 |
-
.rv-button:focus-visible { outline: 2px solid var(--rv-plain); outline-offset: 2px; }
|
| 57 |
|
| 58 |
.rv-lanes { display: grid; grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); gap: 12px; }
|
| 59 |
.rv-lane {
|
|
@@ -106,18 +133,40 @@
|
|
| 106 |
.rv-text .rv-draft { animation: rv-flash 0.5s ease-out; }
|
| 107 |
.rv-own { text-decoration: underline; text-decoration-color: var(--rv-dspark); text-decoration-thickness: 2px; text-underline-offset: 3px; }
|
| 108 |
|
| 109 |
-
|
| 110 |
-
.rv-
|
| 111 |
-
.rv-
|
| 112 |
-
.rv-
|
| 113 |
-
.rv-
|
| 114 |
-
.rv-
|
| 115 |
-
.rv-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
.rv-reading { overflow-wrap: anywhere; }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
.rv-meter { height: 8px; border-radius: 4px; background: var(--background-fill-secondary); overflow: hidden; }
|
| 118 |
.rv-meter i { display: block; height: 100%; border-radius: 4px; background: var(--rv-lane); }
|
| 119 |
.rv-score { text-align: right; font-variant-numeric: tabular-nums; }
|
| 120 |
-
.rv-tag { font-size: 12px; color: var(--rv-muted); }
|
| 121 |
.rv-pick { font-size: 12px; font-weight: 600; }
|
| 122 |
.rv-picked .rv-reading { font-weight: 600; }
|
| 123 |
|
|
@@ -126,9 +175,11 @@
|
|
| 126 |
@media (max-width: 640px) {
|
| 127 |
.rv-row { grid-template-columns: minmax(0, 1fr) 64px; }
|
| 128 |
.rv-row .rv-bar { grid-column: 1 / -1; grid-row: 2; }
|
| 129 |
-
.rv-
|
| 130 |
-
.rv-
|
|
|
|
| 131 |
}
|
| 132 |
@media (prefers-reduced-motion: reduce) {
|
| 133 |
.rv-text .rv-draft, .rv-status::before { animation: none; }
|
|
|
|
| 134 |
}
|
|
|
|
| 2 |
--rv-trocr: #b07a2a;
|
| 3 |
--rv-plain: #1f62b0;
|
| 4 |
--rv-dspark: #1a8a5a;
|
| 5 |
+
--rv-bad: #c4342d;
|
| 6 |
--rv-muted: color-mix(in srgb, var(--body-text-color) 70%, transparent);
|
| 7 |
display: flex;
|
| 8 |
flex-direction: column;
|
| 9 |
+
gap: 18px;
|
| 10 |
color: var(--body-text-color);
|
| 11 |
font-size: 14px;
|
| 12 |
line-height: 1.5;
|
|
|
|
| 15 |
--rv-trocr: #b8842f;
|
| 16 |
--rv-plain: #4f8fdb;
|
| 17 |
--rv-dspark: #3fa97a;
|
| 18 |
+
--rv-bad: #ff8a7a;
|
| 19 |
}
|
| 20 |
.rv-root [hidden] { display: none; }
|
| 21 |
.rv-hint, .rv-status { color: var(--rv-muted); margin: 0; }
|
|
|
|
| 28 |
max-width: 100ch;
|
| 29 |
}
|
| 30 |
.rv-status::before { content: "● "; animation: rv-pulse 1.2s ease-in-out infinite; }
|
| 31 |
+
.rv-group { display: flex; flex-direction: column; gap: 10px; }
|
| 32 |
+
.rv-title { margin: 0; font-size: 15px; font-weight: 600; }
|
| 33 |
+
.rv-summary, .rv-said, .rv-note, .rv-jintro { margin: 0; max-width: 100ch; }
|
| 34 |
+
.rv-note, .rv-jintro { font-size: 13px; color: var(--rv-muted); }
|
| 35 |
.rv-k { font-size: 12px; color: var(--rv-muted); letter-spacing: 0.02em; }
|
| 36 |
|
| 37 |
+
/* a bar per timed stretch; the live bars and our server's share one scale */
|
| 38 |
.rv-track { display: flex; flex-direction: column; gap: 8px; }
|
| 39 |
+
.rv-row { display: grid; grid-template-columns: minmax(170px, 250px) minmax(0, 1fr) 64px; gap: 10px; align-items: center; }
|
| 40 |
.rv-name, .rv-head { display: flex; align-items: center; gap: 8px; min-width: 0; }
|
| 41 |
.rv-swatch { width: 10px; height: 10px; border-radius: 2px; flex: none; background: var(--rv-lane); }
|
| 42 |
[data-lane="trocr"] { --rv-lane: var(--rv-trocr); }
|
|
|
|
| 45 |
.rv-bar { height: 14px; border-radius: 4px; background: var(--background-fill-secondary); overflow: hidden; }
|
| 46 |
.rv-fillbar { height: 100%; width: 0; border-radius: 4px; background: var(--rv-lane); }
|
| 47 |
.rv-time { text-align: right; font-variant-numeric: tabular-nums; }
|
| 48 |
+
/* measured beforehand, not here: tinted and outlined instead of solid */
|
| 49 |
+
.rv-server .rv-fillbar {
|
| 50 |
+
background: color-mix(in srgb, var(--rv-lane) 38%, transparent);
|
| 51 |
+
box-shadow: inset 0 0 0 1.5px var(--rv-lane);
|
| 52 |
+
transition: width 0.4s ease;
|
| 53 |
+
}
|
| 54 |
+
.rv-server .rv-swatch { background: color-mix(in srgb, var(--rv-lane) 38%, transparent); box-shadow: inset 0 0 0 1.5px var(--rv-lane); }
|
| 55 |
+
|
| 56 |
+
.rv-tiles { display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 10px; }
|
| 57 |
+
.rv-tile {
|
| 58 |
+
display: flex;
|
| 59 |
+
flex-direction: column;
|
| 60 |
+
gap: 2px;
|
| 61 |
+
padding: 10px 12px;
|
| 62 |
+
border: 1px solid var(--border-color-primary);
|
| 63 |
+
border-top: 3px solid var(--rv-lane);
|
| 64 |
+
border-radius: 6px;
|
| 65 |
+
}
|
| 66 |
+
.rv-tilename { font-weight: 600; }
|
| 67 |
+
.rv-rate { font-size: 30px; font-weight: 650; line-height: 1.15; font-variant-numeric: tabular-nums; }
|
| 68 |
+
.rv-unit { font-size: 13px; font-weight: 400; color: var(--rv-muted); }
|
| 69 |
|
| 70 |
.rv-controls { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; }
|
| 71 |
.rv-button {
|
|
|
|
| 80 |
.rv-button:hover { border-color: var(--rv-muted); }
|
| 81 |
.rv-main { font-weight: 600; color: #fff; background: var(--rv-dspark); border-color: var(--rv-dspark); padding: 5px 16px; }
|
| 82 |
.rv-main:hover { filter: brightness(1.08); border-color: var(--rv-dspark); }
|
| 83 |
+
.rv-button:focus-visible, .rv-foldhead:focus-visible { outline: 2px solid var(--rv-plain); outline-offset: 2px; }
|
| 84 |
|
| 85 |
.rv-lanes { display: grid; grid-template-columns: repeat(auto-fit, minmax(250px, 1fr)); gap: 12px; }
|
| 86 |
.rv-lane {
|
|
|
|
| 133 |
.rv-text .rv-draft { animation: rv-flash 0.5s ease-out; }
|
| 134 |
.rv-own { text-decoration: underline; text-decoration-color: var(--rv-dspark); text-decoration-thickness: 2px; text-underline-offset: 3px; }
|
| 135 |
|
| 136 |
+
/* Jev: what it found in a sentence, then the lines, folded away */
|
| 137 |
+
.rv-jnote { margin: 0; max-width: 100ch; font-weight: 600; }
|
| 138 |
+
.rv-fold { border: 1px solid var(--border-color-primary); border-radius: 6px; padding: 0 12px; }
|
| 139 |
+
.rv-foldhead { padding: 9px 0; cursor: pointer; font-weight: 600; }
|
| 140 |
+
.rv-fold[open] .rv-foldhead { border-bottom: 1px solid var(--border-color-primary); margin-bottom: 10px; }
|
| 141 |
+
.rv-jr { display: grid; grid-template-columns: 92px minmax(0, 1fr) 68px 138px 72px; gap: 10px; align-items: center; }
|
| 142 |
+
.rv-jhdr { margin-top: 10px; padding-bottom: 6px; font-size: 12px; font-weight: 600; color: var(--rv-muted); }
|
| 143 |
+
.rv-jlist { display: flex; flex-direction: column; }
|
| 144 |
+
.rv-jrow { display: grid; gap: 5px; padding: 10px 0; border-top: 1px solid var(--border-color-primary); }
|
| 145 |
+
.rv-jtop { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 8px; }
|
| 146 |
+
.rv-jimg { max-width: min(100%, 720px); max-height: 64px; border-radius: 3px; background: #fff; }
|
| 147 |
+
.rv-verdict {
|
| 148 |
+
font-size: 12px;
|
| 149 |
+
padding: 1px 9px;
|
| 150 |
+
border-radius: 999px;
|
| 151 |
+
border: 1px solid var(--border-color-primary);
|
| 152 |
+
color: var(--rv-muted);
|
| 153 |
+
white-space: nowrap;
|
| 154 |
+
}
|
| 155 |
+
.rv-verdict[data-kind="good"] { color: var(--body-text-color); border-color: var(--rv-dspark); font-weight: 600; }
|
| 156 |
+
.rv-verdict[data-kind="bad"] { color: var(--body-text-color); border-color: var(--rv-bad); font-weight: 600; }
|
| 157 |
.rv-reading { overflow-wrap: anywhere; }
|
| 158 |
+
.rv-bad {
|
| 159 |
+
background: color-mix(in srgb, var(--rv-bad) 20%, transparent);
|
| 160 |
+
text-decoration: underline wavy var(--rv-bad);
|
| 161 |
+
text-underline-offset: 3px;
|
| 162 |
+
border-radius: 2px;
|
| 163 |
+
}
|
| 164 |
+
.rv-right { font-size: 12px; color: var(--rv-muted); }
|
| 165 |
+
.rv-wrong { font-size: 12px; font-weight: 600; color: var(--rv-bad); }
|
| 166 |
+
.rv-scorebox { display: grid; grid-template-columns: minmax(0, 1fr) 42px; gap: 8px; align-items: center; }
|
| 167 |
.rv-meter { height: 8px; border-radius: 4px; background: var(--background-fill-secondary); overflow: hidden; }
|
| 168 |
.rv-meter i { display: block; height: 100%; border-radius: 4px; background: var(--rv-lane); }
|
| 169 |
.rv-score { text-align: right; font-variant-numeric: tabular-nums; }
|
|
|
|
| 170 |
.rv-pick { font-size: 12px; font-weight: 600; }
|
| 171 |
.rv-picked .rv-reading { font-weight: 600; }
|
| 172 |
|
|
|
|
| 175 |
@media (max-width: 640px) {
|
| 176 |
.rv-row { grid-template-columns: minmax(0, 1fr) 64px; }
|
| 177 |
.rv-row .rv-bar { grid-column: 1 / -1; grid-row: 2; }
|
| 178 |
+
.rv-jr { grid-template-columns: 76px minmax(0, 1fr) 60px; }
|
| 179 |
+
.rv-jr .rv-scorebox, .rv-jr .rv-pick { grid-column: 2 / -1; }
|
| 180 |
+
.rv-jhdr span:nth-child(n + 4) { display: none; }
|
| 181 |
}
|
| 182 |
@media (prefers-reduced-motion: reduce) {
|
| 183 |
.rv-text .rv-draft, .rv-status::before { animation: none; }
|
| 184 |
+
.rv-server .rv-fillbar { transition: none; }
|
| 185 |
}
|
race_view.js
CHANGED
|
@@ -1,20 +1,42 @@
|
|
| 1 |
-
// RaceView: replays three readers at their measured pace, on the manuscript and as text
|
|
|
|
| 2 |
// Gradio runs this once with `element`, `props` and `watch` in scope; `props.value` is a Race payload (app.py).
|
| 3 |
// Every text is set with textContent: readings are model output and are never parsed as HTML.
|
| 4 |
|
| 5 |
const SVG = "http://www.w3.org/2000/svg";
|
| 6 |
const LANES = [
|
| 7 |
-
{ key: "trocr", name: "Swedish Lion TrOCR", how: "the region as its text lines, cut out beforehand; one
|
| 8 |
{ key: "plain", name: "MiniCPM-OCR reader alone", how: "the region as one image; one token per reader pass" },
|
| 9 |
{ key: "dspark", name: "Reader + DSpark draft", how: "the region as one image; up to 16 tokens per reader pass" },
|
| 10 |
];
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
const SPEEDS = [
|
| 12 |
-
{ label: "▶ Race them
|
| 13 |
{ label: "½ speed", speed: 0.5 },
|
| 14 |
{ label: "¼ speed", speed: 0.25 },
|
| 15 |
];
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
const still = window.matchMedia && window.matchMedia("(prefers-reduced-motion: reduce)").matches;
|
| 17 |
-
const state = { run: null,
|
| 18 |
const derived = new WeakMap(); // per lane payload: what drawing needs, worked out once
|
| 19 |
|
| 20 |
function make(tag, cls, text) {
|
|
@@ -46,6 +68,12 @@ function percent(rate) {
|
|
| 46 |
return rate === null || rate === undefined ? "" : ` · CER ${(100 * rate).toFixed(1)} %`;
|
| 47 |
}
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
// A reader lane's pieces as drawing needs them: when each came, the characters read by then, and its passes
|
| 50 |
// (pieces that became known together).
|
| 51 |
function readerFacts(lane) {
|
|
@@ -111,37 +139,28 @@ function inkLines(column, amount) {
|
|
| 111 |
});
|
| 112 |
}
|
| 113 |
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
const lanes = make("div", "rv-lanes");
|
| 128 |
for (const lane of LANES) {
|
| 129 |
-
const row = make("div", "rv-row");
|
| 130 |
-
row.dataset.lane = lane.key;
|
| 131 |
-
const name = make("div", "rv-name");
|
| 132 |
-
name.append(make("i", "rv-swatch"), make("span", "", lane.name));
|
| 133 |
-
const bar = make("div", "rv-bar");
|
| 134 |
-
bar.append(make("div", "rv-fillbar"));
|
| 135 |
-
row.append(name, bar, make("div", "rv-time", "waiting"));
|
| 136 |
-
track.append(row);
|
| 137 |
-
|
| 138 |
const column = make("section", "rv-lane");
|
| 139 |
column.dataset.lane = lane.key;
|
| 140 |
const head = make("header", "rv-head");
|
| 141 |
head.append(make("i", "rv-swatch"), make("b", "", lane.name));
|
| 142 |
const body = make("div", "rv-text");
|
| 143 |
body.dataset.n = "0";
|
| 144 |
-
column.append(head, make("div", "rv-how", lane.how), make("div", "rv-meta", "
|
| 145 |
if (lane.key !== "trocr") column.append(make("div", "rv-passes"));
|
| 146 |
column.append(body);
|
| 147 |
if (lane.key === "dspark") {
|
|
@@ -155,6 +174,25 @@ function ensure(race) {
|
|
| 155 |
}
|
| 156 |
lanes.append(column);
|
| 157 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 158 |
const controls = make("div", "rv-controls");
|
| 159 |
controls.hidden = true;
|
| 160 |
for (const choice of SPEEDS) {
|
|
@@ -163,22 +201,35 @@ function ensure(race) {
|
|
| 163 |
button.dataset.speed = String(choice.speed);
|
| 164 |
controls.append(button);
|
| 165 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
root.replaceChildren(
|
| 167 |
make("div", "rv-notice"),
|
| 168 |
make("div", "rv-status"),
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
make("
|
| 174 |
);
|
| 175 |
return root;
|
| 176 |
}
|
| 177 |
|
| 178 |
-
|
|
|
|
|
|
|
| 179 |
if (still) return Infinity;
|
| 180 |
-
|
| 181 |
-
return state.started[key] === undefined ? 0 : (now - state.started[key]) / 1000;
|
| 182 |
}
|
| 183 |
|
| 184 |
function countUntil(times, at, from) {
|
|
@@ -240,9 +291,9 @@ function drawReader(column, race, lane, at, finished, key) {
|
|
| 240 |
}
|
| 241 |
}
|
| 242 |
|
|
|
|
| 243 |
function drawTrocr(column, lane, at, finished) {
|
| 244 |
const body = column.querySelector(".rv-text");
|
| 245 |
-
const step = finished ? lane.steps.length : countUntil(lane.steps, at, Number(body.dataset.n || 0));
|
| 246 |
if (body.dataset.kind !== "trocr" || body.childElementCount !== lane.lines.length) {
|
| 247 |
body.replaceChildren(
|
| 248 |
...lane.lines.map((line, i) => {
|
|
@@ -253,77 +304,127 @@ function drawTrocr(column, lane, at, finished) {
|
|
| 253 |
);
|
| 254 |
body.dataset.kind = "trocr";
|
| 255 |
}
|
| 256 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 257 |
lane.lines.forEach((line, i) => {
|
| 258 |
-
body.children[i].textContent = line.text.slice(0, reached
|
| 259 |
});
|
| 260 |
-
|
| 261 |
-
inkLines(column, (i) => (lane.lines[i] ? reached(lane.lines[i]) / Math.max(1, lane.lines[i].text.length) : 0));
|
| 262 |
const meta = column.querySelector(".rv-meta");
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 266 |
}
|
| 267 |
|
| 268 |
-
function
|
| 269 |
-
const
|
| 270 |
-
|
|
|
|
|
|
|
|
|
|
| 271 |
const meter = make("span", "rv-meter");
|
| 272 |
const fill = make("i");
|
| 273 |
-
fill.style.width = `${Math.round(100 * score)}%`;
|
| 274 |
meter.append(fill);
|
| 275 |
-
|
|
|
|
|
|
|
| 276 |
make("span", "rv-k", name),
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
make("span", "rv-
|
| 281 |
);
|
| 282 |
-
|
| 283 |
-
return box;
|
| 284 |
}
|
| 285 |
|
| 286 |
function drawJev(root, race) {
|
| 287 |
const box = root.querySelector(".rv-jev");
|
| 288 |
const rows = race.jev || [];
|
| 289 |
-
|
| 290 |
-
|
| 291 |
-
return;
|
| 292 |
-
}
|
| 293 |
let list = box.querySelector(".rv-jlist");
|
| 294 |
if (!list) {
|
| 295 |
list = make("div", "rv-jlist");
|
| 296 |
-
const
|
| 297 |
-
|
| 298 |
-
"
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
list,
|
| 304 |
-
);
|
| 305 |
}
|
| 306 |
for (let i = list.childElementCount; i < rows.length; i += 1) {
|
| 307 |
const row = rows[i];
|
| 308 |
const item = make("div", "rv-jrow");
|
| 309 |
item.dataset.line = String(row.index);
|
|
|
|
| 310 |
const image = make("img", "rv-jimg");
|
| 311 |
image.src = row.image;
|
| 312 |
image.alt = `Line ${row.index + 1} of the region`;
|
| 313 |
image.loading = "lazy";
|
| 314 |
-
const
|
|
|
|
|
|
|
|
|
|
| 315 |
truth.append(make("span", "rv-k", "ground truth"), make("span", "rv-reading", row.truth));
|
| 316 |
-
const pickMinicpm = row.minicpm_score > row.trocr_score;
|
| 317 |
item.append(
|
| 318 |
-
|
| 319 |
truth,
|
| 320 |
-
|
| 321 |
-
|
| 322 |
);
|
| 323 |
list.append(item);
|
| 324 |
}
|
| 325 |
box.querySelector(".rv-jnote").textContent =
|
| 326 |
race.jev_note || `Jev has scored ${rows.length} of ${race.jev_total} lines…`;
|
|
|
|
|
|
|
| 327 |
}
|
| 328 |
|
| 329 |
function draw(now) {
|
|
@@ -331,37 +432,47 @@ function draw(now) {
|
|
| 331 |
const race = payload();
|
| 332 |
if (!race) return;
|
| 333 |
const root = ensure(race);
|
| 334 |
-
|
| 335 |
-
|
| 336 |
-
|
| 337 |
-
|
| 338 |
-
notice.textContent = race.notice || "";
|
| 339 |
-
notice.hidden = !race.notice;
|
| 340 |
-
const summary = root.querySelector(".rv-summary");
|
| 341 |
-
summary.textContent = race.summary || "";
|
| 342 |
-
summary.hidden = !race.summary;
|
| 343 |
-
root.querySelector(".rv-controls").hidden = !race.done || still;
|
| 344 |
|
| 345 |
-
const
|
|
|
|
|
|
|
|
|
|
|
|
|
| 346 |
let running = false;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 347 |
for (const lane of LANES) {
|
| 348 |
const data = race[lane.key];
|
| 349 |
-
const row = root.querySelector(`.rv-row[data-lane="${lane.key}"]`);
|
| 350 |
const column = root.querySelector(`.rv-lane[data-lane="${lane.key}"]`);
|
| 351 |
-
|
| 352 |
-
|
| 353 |
-
|
| 354 |
-
column.
|
| 355 |
continue;
|
| 356 |
}
|
| 357 |
-
const
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
row.querySelector(".rv-fillbar").style.width = `${(100 * at) / scale}%`;
|
| 361 |
-
row.querySelector(".rv-time").textContent = `${at.toFixed(2)} s`;
|
| 362 |
-
if (lane.key === "trocr") drawTrocr(column, data, at, finished);
|
| 363 |
-
else drawReader(column, race, data, at, finished, lane.key);
|
| 364 |
}
|
|
|
|
| 365 |
drawJev(root, race);
|
| 366 |
if (running) state.frame = requestAnimationFrame(draw);
|
| 367 |
}
|
|
@@ -369,21 +480,17 @@ function draw(now) {
|
|
| 369 |
function sync() {
|
| 370 |
const race = payload();
|
| 371 |
if (!race) return;
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
|
| 375 |
-
}
|
| 376 |
-
for (const lane of LANES) {
|
| 377 |
-
if (race[lane.key] && state.started[lane.key] === undefined) state.started[lane.key] = now;
|
| 378 |
-
}
|
| 379 |
if (state.frame) cancelAnimationFrame(state.frame);
|
| 380 |
-
draw(now);
|
| 381 |
}
|
| 382 |
|
| 383 |
element.addEventListener("click", (event) => {
|
| 384 |
const button = event.target.closest("[data-speed]");
|
| 385 |
if (!button) return;
|
| 386 |
-
Object.assign(state, {
|
| 387 |
if (state.frame) cancelAnimationFrame(state.frame);
|
| 388 |
state.frame = requestAnimationFrame(draw);
|
| 389 |
});
|
|
|
|
| 1 |
+
// RaceView: replays three readers from the same start at their measured pace, on the manuscript and as text; shows
|
| 2 |
+
// the same region's timings on our server; then lists Jev's scores line by line.
|
| 3 |
// Gradio runs this once with `element`, `props` and `watch` in scope; `props.value` is a Race payload (app.py).
|
| 4 |
// Every text is set with textContent: readings are model output and are never parsed as HTML.
|
| 5 |
|
| 6 |
const SVG = "http://www.w3.org/2000/svg";
|
| 7 |
const LANES = [
|
| 8 |
+
{ key: "trocr", name: "Swedish Lion TrOCR", how: "the region as its text lines, cut out beforehand; one line after the other" },
|
| 9 |
{ key: "plain", name: "MiniCPM-OCR reader alone", how: "the region as one image; one token per reader pass" },
|
| 10 |
{ key: "dspark", name: "Reader + DSpark draft", how: "the region as one image; up to 16 tokens per reader pass" },
|
| 11 |
];
|
| 12 |
+
// The stretches timed here: each lane's own, and TrOCR's lines once more as one batch.
|
| 13 |
+
const BARS = [
|
| 14 |
+
{ key: "trocr", lane: "trocr", name: "TrOCR · one line after the other", seconds: (race) => race.trocr && race.trocr.seconds },
|
| 15 |
+
{ key: "batch", lane: "trocr", name: "TrOCR · its lines as one batch", seconds: (race) => race.trocr && race.trocr.batch_seconds },
|
| 16 |
+
{ key: "plain", lane: "plain", name: "MiniCPM-OCR reader alone", seconds: (race) => race.plain && race.plain.seconds },
|
| 17 |
+
{ key: "dspark", lane: "dspark", name: "Reader + DSpark draft", seconds: (race) => race.dspark && race.dspark.seconds },
|
| 18 |
+
];
|
| 19 |
+
// The same stretches on our server (Race.server), the reader served by vLLM.
|
| 20 |
+
const SERVER = [
|
| 21 |
+
{ field: "trocr_single", lane: "trocr", name: "TrOCR · one line after the other" },
|
| 22 |
+
{ field: "trocr_batch", lane: "trocr", name: "TrOCR · its lines as one batch" },
|
| 23 |
+
{ field: "vllm", lane: "plain", name: "Reader alone · in vLLM" },
|
| 24 |
+
{ field: "vllm_dspark", lane: "dspark", name: "Reader + DSpark · in vLLM" },
|
| 25 |
+
];
|
| 26 |
const SPEEDS = [
|
| 27 |
+
{ label: "▶ Race them again", speed: 1, main: true },
|
| 28 |
{ label: "½ speed", speed: 0.5 },
|
| 29 |
{ label: "¼ speed", speed: 0.25 },
|
| 30 |
];
|
| 31 |
+
const JEV_INTRO =
|
| 32 |
+
"Each line is read on its own by TrOCR and by the MiniCPM-OCR reader. Marked characters differ from the ground " +
|
| 33 |
+
"truth (the line as a person transcribed it); ‸ stands for missing ones. Jev looks at the line and at a reading " +
|
| 34 |
+
"drawn as an image, and scores from 0 to 1 how sure it is that the reading is exactly what the line says. Where " +
|
| 35 |
+
"the two readings differ, the one Jev scores higher is its pick. Jev catches a reading that is plainly another " +
|
| 36 |
+
"text far better than one that is a letter or two off (see How it works), so a high score does not prove a " +
|
| 37 |
+
"reading right.";
|
| 38 |
const still = window.matchMedia && window.matchMedia("(prefers-reduced-motion: reduce)").matches;
|
| 39 |
+
const state = { run: null, raced: false, speed: 1, raceStart: 0, frame: 0 };
|
| 40 |
const derived = new WeakMap(); // per lane payload: what drawing needs, worked out once
|
| 41 |
|
| 42 |
function make(tag, cls, text) {
|
|
|
|
| 68 |
return rate === null || rate === undefined ? "" : ` · CER ${(100 * rate).toFixed(1)} %`;
|
| 69 |
}
|
| 70 |
|
| 71 |
+
function write(root, selector, text) {
|
| 72 |
+
const node = root.querySelector(selector);
|
| 73 |
+
node.textContent = text || "";
|
| 74 |
+
node.hidden = !text;
|
| 75 |
+
}
|
| 76 |
+
|
| 77 |
// A reader lane's pieces as drawing needs them: when each came, the characters read by then, and its passes
|
| 78 |
// (pieces that became known together).
|
| 79 |
function readerFacts(lane) {
|
|
|
|
| 139 |
});
|
| 140 |
}
|
| 141 |
|
| 142 |
+
function barRow(key, lane, name) {
|
| 143 |
+
const row = make("div", "rv-row");
|
| 144 |
+
row.dataset.bar = key;
|
| 145 |
+
row.dataset.lane = lane;
|
| 146 |
+
const label = make("div", "rv-name");
|
| 147 |
+
label.append(make("i", "rv-swatch"), make("span", "", name));
|
| 148 |
+
const bar = make("div", "rv-bar");
|
| 149 |
+
bar.append(make("div", "rv-fillbar"));
|
| 150 |
+
row.append(label, bar, make("div", "rv-time"));
|
| 151 |
+
return row;
|
| 152 |
+
}
|
| 153 |
+
|
| 154 |
+
function lanesOf(race) {
|
| 155 |
const lanes = make("div", "rv-lanes");
|
| 156 |
for (const lane of LANES) {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
const column = make("section", "rv-lane");
|
| 158 |
column.dataset.lane = lane.key;
|
| 159 |
const head = make("header", "rv-head");
|
| 160 |
head.append(make("i", "rv-swatch"), make("b", "", lane.name));
|
| 161 |
const body = make("div", "rv-text");
|
| 162 |
body.dataset.n = "0";
|
| 163 |
+
column.append(head, make("div", "rv-how", lane.how), make("div", "rv-meta", " "), page(race, lane.key));
|
| 164 |
if (lane.key !== "trocr") column.append(make("div", "rv-passes"));
|
| 165 |
column.append(body);
|
| 166 |
if (lane.key === "dspark") {
|
|
|
|
| 174 |
}
|
| 175 |
lanes.append(column);
|
| 176 |
}
|
| 177 |
+
return lanes;
|
| 178 |
+
}
|
| 179 |
+
|
| 180 |
+
// The skeleton, built again whenever Gradio re-renders the template or a new run starts.
|
| 181 |
+
function ensure(race) {
|
| 182 |
+
let root = element.querySelector(".rv-root");
|
| 183 |
+
if (!root) {
|
| 184 |
+
root = make("div", "rv-root");
|
| 185 |
+
element.appendChild(root);
|
| 186 |
+
}
|
| 187 |
+
root.dataset.theme = document.body.classList.contains("dark") ? "dark" : "light";
|
| 188 |
+
const built = root.dataset.run === race.run && root.dataset.image === (race.image || "");
|
| 189 |
+
if (built && root.querySelector(".rv-lanes")) return root;
|
| 190 |
+
root.dataset.run = race.run;
|
| 191 |
+
root.dataset.image = race.image || "";
|
| 192 |
+
|
| 193 |
+
const live = make("section", "rv-group rv-live");
|
| 194 |
+
const track = make("div", "rv-track");
|
| 195 |
+
for (const bar of BARS) track.append(barRow(bar.key, bar.lane, bar.name));
|
| 196 |
const controls = make("div", "rv-controls");
|
| 197 |
controls.hidden = true;
|
| 198 |
for (const choice of SPEEDS) {
|
|
|
|
| 201 |
button.dataset.speed = String(choice.speed);
|
| 202 |
controls.append(button);
|
| 203 |
}
|
| 204 |
+
const liveTitle = "Here, live: one stream each, in PyTorch on half a GPU";
|
| 205 |
+
live.append(make("h4", "rv-title", liveTitle), track, controls, make("p", "rv-summary"));
|
| 206 |
+
|
| 207 |
+
const server = make("section", "rv-group rv-server");
|
| 208 |
+
const serverTrack = make("div", "rv-track");
|
| 209 |
+
for (const bar of SERVER) serverTrack.append(barRow(bar.field, bar.lane, bar.name));
|
| 210 |
+
server.append(make("h4", "rv-title"), serverTrack, make("p", "rv-said"), make("p", "rv-note"));
|
| 211 |
+
|
| 212 |
+
const batched = make("section", "rv-group rv-batched");
|
| 213 |
+
const batchedTitle = "Everything batched, on our server: lines read a second";
|
| 214 |
+
batched.append(make("h4", "rv-title", batchedTitle), make("div", "rv-tiles"), make("p", "rv-note"));
|
| 215 |
+
|
| 216 |
root.replaceChildren(
|
| 217 |
make("div", "rv-notice"),
|
| 218 |
make("div", "rv-status"),
|
| 219 |
+
live,
|
| 220 |
+
lanesOf(race),
|
| 221 |
+
server,
|
| 222 |
+
batched,
|
| 223 |
+
make("section", "rv-group rv-jev"),
|
| 224 |
);
|
| 225 |
return root;
|
| 226 |
}
|
| 227 |
|
| 228 |
+
// Seconds into the replay: none before every reader is timed, all of them where motion is unwelcome.
|
| 229 |
+
function elapsed(now) {
|
| 230 |
+
if (!state.raced) return 0;
|
| 231 |
if (still) return Infinity;
|
| 232 |
+
return ((now - state.raceStart) / 1000) * state.speed;
|
|
|
|
| 233 |
}
|
| 234 |
|
| 235 |
function countUntil(times, at, from) {
|
|
|
|
| 291 |
}
|
| 292 |
}
|
| 293 |
|
| 294 |
+
// TrOCR reads its lines one after the other: each line's steps carry their own times.
|
| 295 |
function drawTrocr(column, lane, at, finished) {
|
| 296 |
const body = column.querySelector(".rv-text");
|
|
|
|
| 297 |
if (body.dataset.kind !== "trocr" || body.childElementCount !== lane.lines.length) {
|
| 298 |
body.replaceChildren(
|
| 299 |
...lane.lines.map((line, i) => {
|
|
|
|
| 304 |
);
|
| 305 |
body.dataset.kind = "trocr";
|
| 306 |
}
|
| 307 |
+
let current = lane.lines.length;
|
| 308 |
+
const reached = lane.lines.map((line, i) => {
|
| 309 |
+
if (finished) return line.text.length;
|
| 310 |
+
const steps = countUntil(line.steps, at, 0);
|
| 311 |
+
if (steps < line.steps.length && current === lane.lines.length) current = i;
|
| 312 |
+
return steps === 0 ? 0 : line.ends[steps - 1];
|
| 313 |
+
});
|
| 314 |
lane.lines.forEach((line, i) => {
|
| 315 |
+
body.children[i].textContent = line.text.slice(0, reached[i]) || " ";
|
| 316 |
});
|
| 317 |
+
inkLines(column, (i) => (lane.lines[i] ? reached[i] / Math.max(1, lane.lines[i].text.length) : 0));
|
|
|
|
| 318 |
const meta = column.querySelector(".rv-meta");
|
| 319 |
+
if (finished) {
|
| 320 |
+
const steps = lane.lines.reduce((sum, line) => sum + line.steps.length, 0);
|
| 321 |
+
meta.textContent = `${lane.lines.length} lines in ${steps} decoding steps${percent(lane.cer)}`;
|
| 322 |
+
} else {
|
| 323 |
+
meta.textContent = `line ${Math.min(current + 1, lane.lines.length)} of ${lane.lines.length}`;
|
| 324 |
+
}
|
| 325 |
+
}
|
| 326 |
+
|
| 327 |
+
function drawServer(root, race, scale) {
|
| 328 |
+
const box = root.querySelector(".rv-server");
|
| 329 |
+
const server = race.server;
|
| 330 |
+
box.hidden = !server;
|
| 331 |
+
if (server) {
|
| 332 |
+
box.querySelector(".rv-title").textContent =
|
| 333 |
+
`The same region on our server: a whole ${server.gpu}, the reader served by vLLM`;
|
| 334 |
+
for (const bar of SERVER) {
|
| 335 |
+
const row = box.querySelector(`[data-bar="${bar.field}"]`);
|
| 336 |
+
row.querySelector(".rv-fillbar").style.width = `${(100 * server[bar.field]) / scale}%`;
|
| 337 |
+
row.querySelector(".rv-time").textContent = `${server[bar.field].toFixed(2)} s`;
|
| 338 |
+
}
|
| 339 |
+
box.querySelector(".rv-said").textContent = race.server_said || "";
|
| 340 |
+
box.querySelector(".rv-note").textContent =
|
| 341 |
+
`Measured beforehand (${server.measured}): one request at a time, each image new to the server, the median ` +
|
| 342 |
+
`of ${server.runs} runs. This Space gets half of such a GPU and decodes in PyTorch, since ZeroGPU cannot ` +
|
| 343 |
+
"host vLLM. These bars and the live ones share one scale.";
|
| 344 |
+
}
|
| 345 |
+
const batched = root.querySelector(".rv-batched");
|
| 346 |
+
const rates = race.batched || [];
|
| 347 |
+
batched.hidden = !rates.length;
|
| 348 |
+
const tiles = batched.querySelector(".rv-tiles");
|
| 349 |
+
if (tiles.childElementCount !== rates.length) {
|
| 350 |
+
tiles.replaceChildren(
|
| 351 |
+
...rates.map((rate) => {
|
| 352 |
+
const tile = make("div", "rv-tile");
|
| 353 |
+
tile.dataset.lane = rate.lane;
|
| 354 |
+
const figure = make("div", "rv-rate", rate.per_second.toFixed(rate.per_second < 10 ? 1 : 0));
|
| 355 |
+
figure.append(make("span", "rv-unit", " lines a second"));
|
| 356 |
+
tile.append(make("div", "rv-tilename", rate.name), figure, make("div", "rv-how", rate.how));
|
| 357 |
+
return tile;
|
| 358 |
+
}),
|
| 359 |
+
);
|
| 360 |
+
}
|
| 361 |
+
batched.querySelector(".rv-note").textContent = race.batched_note || "";
|
| 362 |
}
|
| 363 |
|
| 364 |
+
function reading(name, key, data, picked) {
|
| 365 |
+
const row = make("div", picked ? "rv-jr rv-picked" : "rv-jr");
|
| 366 |
+
row.dataset.lane = key;
|
| 367 |
+
const text = make("span", "rv-reading");
|
| 368 |
+
for (const [piece, agrees] of data.marks) text.append(make("span", agrees ? "" : "rv-bad", piece));
|
| 369 |
+
if (!data.marks.length) text.textContent = "(empty)";
|
| 370 |
const meter = make("span", "rv-meter");
|
| 371 |
const fill = make("i");
|
| 372 |
+
fill.style.width = `${Math.round(100 * data.score)}%`;
|
| 373 |
meter.append(fill);
|
| 374 |
+
const score = make("span", "rv-scorebox");
|
| 375 |
+
score.append(meter, make("span", "rv-score", data.score.toFixed(3)));
|
| 376 |
+
row.append(
|
| 377 |
make("span", "rv-k", name),
|
| 378 |
+
text,
|
| 379 |
+
make("span", data.exact ? "rv-right" : "rv-wrong", data.exact ? "✓ right" : "✗ wrong"),
|
| 380 |
+
score,
|
| 381 |
+
make("span", "rv-pick", picked ? "Jev's pick" : ""),
|
| 382 |
);
|
| 383 |
+
return row;
|
|
|
|
| 384 |
}
|
| 385 |
|
| 386 |
function drawJev(root, race) {
|
| 387 |
const box = root.querySelector(".rv-jev");
|
| 388 |
const rows = race.jev || [];
|
| 389 |
+
box.hidden = !race.jev_total;
|
| 390 |
+
if (!race.jev_total) return;
|
|
|
|
|
|
|
| 391 |
let list = box.querySelector(".rv-jlist");
|
| 392 |
if (!list) {
|
| 393 |
list = make("div", "rv-jlist");
|
| 394 |
+
const header = make("div", "rv-jr rv-jhdr");
|
| 395 |
+
for (const name of ["Reader", "Its reading of the line", "Right?", "Jev's score, 0 to 1", ""]) {
|
| 396 |
+
header.append(make("span", "", name));
|
| 397 |
+
}
|
| 398 |
+
const fold = make("details", "rv-fold");
|
| 399 |
+
fold.append(make("summary", "rv-foldhead"), make("p", "rv-jintro", JEV_INTRO), header, list);
|
| 400 |
+
box.replaceChildren(make("h4", "rv-title", "Jev, the reader's proof-reader"), make("p", "rv-jnote"), fold);
|
|
|
|
|
|
|
| 401 |
}
|
| 402 |
for (let i = list.childElementCount; i < rows.length; i += 1) {
|
| 403 |
const row = rows[i];
|
| 404 |
const item = make("div", "rv-jrow");
|
| 405 |
item.dataset.line = String(row.index);
|
| 406 |
+
const top = make("div", "rv-jtop");
|
| 407 |
const image = make("img", "rv-jimg");
|
| 408 |
image.src = row.image;
|
| 409 |
image.alt = `Line ${row.index + 1} of the region`;
|
| 410 |
image.loading = "lazy";
|
| 411 |
+
const verdict = make("span", "rv-verdict", row.verdict);
|
| 412 |
+
verdict.dataset.kind = row.verdict.startsWith("Jev picks the right") ? "good" : row.verdict === "both right" ? "" : "bad";
|
| 413 |
+
top.append(image, verdict);
|
| 414 |
+
const truth = make("div", "rv-jr rv-truth");
|
| 415 |
truth.append(make("span", "rv-k", "ground truth"), make("span", "rv-reading", row.truth));
|
|
|
|
| 416 |
item.append(
|
| 417 |
+
top,
|
| 418 |
truth,
|
| 419 |
+
reading("TrOCR", "trocr", row.trocr, row.pick === "trocr"),
|
| 420 |
+
reading("MiniCPM-OCR", "plain", row.minicpm, row.pick === "minicpm"),
|
| 421 |
);
|
| 422 |
list.append(item);
|
| 423 |
}
|
| 424 |
box.querySelector(".rv-jnote").textContent =
|
| 425 |
race.jev_note || `Jev has scored ${rows.length} of ${race.jev_total} lines…`;
|
| 426 |
+
box.querySelector(".rv-foldhead").textContent =
|
| 427 |
+
`Line by line: both readings and Jev's scores (${rows.length} of ${race.jev_total} lines)`;
|
| 428 |
}
|
| 429 |
|
| 430 |
function draw(now) {
|
|
|
|
| 432 |
const race = payload();
|
| 433 |
if (!race) return;
|
| 434 |
const root = ensure(race);
|
| 435 |
+
write(root, ".rv-status", race.status);
|
| 436 |
+
write(root, ".rv-notice", race.notice);
|
| 437 |
+
write(root, ".rv-summary", race.summary);
|
| 438 |
+
root.querySelector(".rv-controls").hidden = !race.ready || still;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 439 |
|
| 440 |
+
const server = race.server;
|
| 441 |
+
const known = BARS.map((bar) => bar.seconds(race) || 0);
|
| 442 |
+
if (server) known.push(...SERVER.map((bar) => server[bar.field]));
|
| 443 |
+
const scale = Math.max(0.001, ...known);
|
| 444 |
+
const at = elapsed(now);
|
| 445 |
let running = false;
|
| 446 |
+
for (const bar of BARS) {
|
| 447 |
+
const row = root.querySelector(`.rv-live [data-bar="${bar.key}"]`);
|
| 448 |
+
const total = bar.seconds(race);
|
| 449 |
+
const fill = row.querySelector(".rv-fillbar");
|
| 450 |
+
const time = row.querySelector(".rv-time");
|
| 451 |
+
if (!total || !state.raced) {
|
| 452 |
+
const skipped = !total && (race.done || (bar.lane === "trocr" && race.plain));
|
| 453 |
+
fill.style.width = "0";
|
| 454 |
+
time.textContent = total ? "timed" : skipped ? "not run" : "waiting";
|
| 455 |
+
continue;
|
| 456 |
+
}
|
| 457 |
+
const shown = Math.min(at, total);
|
| 458 |
+
running = running || shown < total;
|
| 459 |
+
fill.style.width = `${(100 * shown) / scale}%`;
|
| 460 |
+
time.textContent = `${shown.toFixed(2)} s`;
|
| 461 |
+
}
|
| 462 |
for (const lane of LANES) {
|
| 463 |
const data = race[lane.key];
|
|
|
|
| 464 |
const column = root.querySelector(`.rv-lane[data-lane="${lane.key}"]`);
|
| 465 |
+
column.classList.toggle("rv-skipped", !data && Boolean(race.done || (lane.key === "trocr" && race.plain)));
|
| 466 |
+
if (!data) continue;
|
| 467 |
+
if (!state.raced) {
|
| 468 |
+
column.querySelector(".rv-meta").textContent = "timed; the replay starts when all are";
|
| 469 |
continue;
|
| 470 |
}
|
| 471 |
+
const shown = Math.min(at, data.seconds);
|
| 472 |
+
if (lane.key === "trocr") drawTrocr(column, data, shown, shown >= data.seconds);
|
| 473 |
+
else drawReader(column, race, data, shown, shown >= data.seconds, lane.key);
|
|
|
|
|
|
|
|
|
|
|
|
|
| 474 |
}
|
| 475 |
+
drawServer(root, race, scale);
|
| 476 |
drawJev(root, race);
|
| 477 |
if (running) state.frame = requestAnimationFrame(draw);
|
| 478 |
}
|
|
|
|
| 480 |
function sync() {
|
| 481 |
const race = payload();
|
| 482 |
if (!race) return;
|
| 483 |
+
if (race.run !== state.run) Object.assign(state, { run: race.run, raced: false, speed: 1 });
|
| 484 |
+
// every reader is timed: they start together
|
| 485 |
+
if (race.ready && !state.raced) Object.assign(state, { raced: true, speed: 1, raceStart: performance.now() });
|
|
|
|
|
|
|
|
|
|
|
|
|
| 486 |
if (state.frame) cancelAnimationFrame(state.frame);
|
| 487 |
+
draw(performance.now());
|
| 488 |
}
|
| 489 |
|
| 490 |
element.addEventListener("click", (event) => {
|
| 491 |
const button = event.target.closest("[data-speed]");
|
| 492 |
if (!button) return;
|
| 493 |
+
Object.assign(state, { raced: true, speed: Number(button.dataset.speed), raceStart: performance.now() });
|
| 494 |
if (state.frame) cancelAnimationFrame(state.frame);
|
| 495 |
state.frame = requestAnimationFrame(draw);
|
| 496 |
});
|
race_view.py
CHANGED
|
@@ -1,5 +1,6 @@
|
|
| 1 |
-
"""RaceView, a custom Gradio component: three readers' readings replayed at their measured pace,
|
| 2 |
-
line by line. Its value is a `Race` payload (app.py); the
|
|
|
|
| 3 |
|
| 4 |
from __future__ import annotations
|
| 5 |
|
|
@@ -10,15 +11,16 @@ import gradio as gr
|
|
| 10 |
|
| 11 |
_HERE = Path(__file__).parent
|
| 12 |
HINT = (
|
| 13 |
-
"Pick a region and press Read: the three readers run one after the other, and
|
| 14 |
-
"the pace it was measured."
|
| 15 |
)
|
| 16 |
TEMPLATE = f'<div class="rv-root"><p class="rv-hint">{HINT}</p></div>'
|
| 17 |
|
| 18 |
|
| 19 |
class RaceView(gr.HTML):
|
| 20 |
-
"""Replays a `Race`: a bar and a text lane per reader, growing as the reader produced its
|
| 21 |
-
line with Jev's scores. Readings are written as text,
|
|
|
|
| 22 |
|
| 23 |
def __init__(self, value: dict[str, Any] | None = None, **kwargs: Any) -> None:
|
| 24 |
super().__init__(
|
|
|
|
| 1 |
+
"""RaceView, a custom Gradio component: three readers' readings replayed from the same start at their measured pace,
|
| 2 |
+
the same region's timings on our server, then Jev's scores line by line. Its value is a `Race` payload (app.py); the
|
| 3 |
+
drawing is in race_view.js and race_view.css."""
|
| 4 |
|
| 5 |
from __future__ import annotations
|
| 6 |
|
|
|
|
| 11 |
|
| 12 |
_HERE = Path(__file__).parent
|
| 13 |
HINT = (
|
| 14 |
+
"Pick a region and press Read: the three readers run one after the other, and once all are timed their "
|
| 15 |
+
"readings are replayed here from the same start, each at the pace it was measured."
|
| 16 |
)
|
| 17 |
TEMPLATE = f'<div class="rv-root"><p class="rv-hint">{HINT}</p></div>'
|
| 18 |
|
| 19 |
|
| 20 |
class RaceView(gr.HTML):
|
| 21 |
+
"""Replays a `Race`: a bar per timed stretch and a text lane per reader, growing as the reader produced its
|
| 22 |
+
tokens; the same stretches on our server; and one row per line with Jev's scores. Readings are written as text,
|
| 23 |
+
never as HTML."""
|
| 24 |
|
| 25 |
def __init__(self, value: dict[str, Any] | None = None, **kwargs: Any) -> None:
|
| 26 |
super().__init__(
|
reference.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"gpu": "RTX PRO 6000 Blackwell",
|
| 3 |
+
"measured": "2026-11-11",
|
| 4 |
+
"runs": 5,
|
| 5 |
+
"regions": {
|
| 6 |
+
"8d24e7df7dea_2": {
|
| 7 |
+
"lines": 10,
|
| 8 |
+
"tokens": 124,
|
| 9 |
+
"trocr_single": 1.661,
|
| 10 |
+
"trocr_batch": 0.837,
|
| 11 |
+
"vllm": 0.497,
|
| 12 |
+
"vllm_dspark": 0.321
|
| 13 |
+
},
|
| 14 |
+
"19a1b2195bab_right_0": {
|
| 15 |
+
"lines": 8,
|
| 16 |
+
"tokens": 76,
|
| 17 |
+
"trocr_single": 1.246,
|
| 18 |
+
"trocr_batch": 0.603,
|
| 19 |
+
"vllm": 0.379,
|
| 20 |
+
"vllm_dspark": 0.266
|
| 21 |
+
},
|
| 22 |
+
"eed569a6699c_left_3": {
|
| 23 |
+
"lines": 9,
|
| 24 |
+
"tokens": 96,
|
| 25 |
+
"trocr_single": 1.859,
|
| 26 |
+
"trocr_batch": 0.752,
|
| 27 |
+
"vllm": 0.525,
|
| 28 |
+
"vllm_dspark": 0.345
|
| 29 |
+
},
|
| 30 |
+
"f970d28ac08b_left_0": {
|
| 31 |
+
"lines": 11,
|
| 32 |
+
"tokens": 129,
|
| 33 |
+
"trocr_single": 1.474,
|
| 34 |
+
"trocr_batch": 0.668,
|
| 35 |
+
"vllm": 0.569,
|
| 36 |
+
"vllm_dspark": 0.335
|
| 37 |
+
},
|
| 38 |
+
"38c4ce6331ad_left_1": {
|
| 39 |
+
"lines": 17,
|
| 40 |
+
"tokens": 209,
|
| 41 |
+
"trocr_single": 3.922,
|
| 42 |
+
"trocr_batch": 1.543,
|
| 43 |
+
"vllm": 0.688,
|
| 44 |
+
"vllm_dspark": 0.334
|
| 45 |
+
},
|
| 46 |
+
"35ffe6b8b600_left_0": {
|
| 47 |
+
"lines": 18,
|
| 48 |
+
"tokens": 198,
|
| 49 |
+
"trocr_single": 3.412,
|
| 50 |
+
"trocr_batch": 1.323,
|
| 51 |
+
"vllm": 0.729,
|
| 52 |
+
"vllm_dspark": 0.45
|
| 53 |
+
}
|
| 54 |
+
}
|
| 55 |
+
}
|
reference.py
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The Space's reference timings (reference.json): measure_reference.py writes them and app.py shows them beside its
|
| 2 |
+
live race."""
|
| 3 |
+
|
| 4 |
+
from __future__ import annotations
|
| 5 |
+
|
| 6 |
+
from pydantic import BaseModel
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
class RegionTimes(BaseModel):
|
| 10 |
+
"""One region's median seconds, each from the encoded images to the reading: TrOCR on its lines one after the
|
| 11 |
+
other and as one batch, and the reader in vLLM alone and with its DSpark draft."""
|
| 12 |
+
|
| 13 |
+
lines: int
|
| 14 |
+
tokens: int
|
| 15 |
+
trocr_single: float
|
| 16 |
+
trocr_batch: float
|
| 17 |
+
vllm: float
|
| 18 |
+
vllm_dspark: float
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
class Reference(BaseModel):
|
| 22 |
+
"""The timings of every example region, with the GPU and the day they were measured on."""
|
| 23 |
+
|
| 24 |
+
gpu: str
|
| 25 |
+
measured: str
|
| 26 |
+
runs: int
|
| 27 |
+
regions: dict[str, RegionTimes]
|