Spaces:
Running on Zero
Download HOW_IT_WORKS.md from Gabriel/diffu-htr-speed: direct link, hf CLI and curl.
- Browser
- Download file 8.83 kB
-
https://huggingface.co/spaces/Gabriel/diffu-htr-speed/resolve/main/HOW_IT_WORKS.md
- Command line
-
hf download hf://spaces/Gabriel/diffu-htr-speed/HOW_IT_WORKS.md
-
curl -L -o HOW_IT_WORKS.md https://huggingface.co/spaces/Gabriel/diffu-htr-speed/resolve/main/HOW_IT_WORKS.md
A newer version of the Gradio SDK is available: 6.32.0
What is timed
All three read the same region, one after the other on the same GPU (ZeroGPU: half an RTX PRO 6000 Blackwell), with one stream each and nothing batched. Each is timed from the images to the text, preprocessing included, after a warm-up that is not timed:
- Swedish Lion TrOCR (Riksarkivet's Swedish Lion Libre 2026, large, character-based) cannot read a region. It gets the region as its text lines, cut out beforehand along the ground truth's line outlines, and reads them one after the other, greedily, in float32 as Riksarkivet runs it. Finding the lines is a model of its own in practice; that step is not timed here.
- MiniCPM-OCR reader alone reads the region as one image, one token per pass, greedily.
- Reader + DSpark draft: the draft proposes 15 tokens at a time; the reader checks them in one pass and keeps the ones it agrees with, plus one of its own. Greedy decoding, so the text is the reader's own.
Comparing them fairly
A batch lets a GPU decode many sequences side by side. TrOCR can batch a region's lines. The reader reads a region as one sequence, which has nothing to batch; what it batches is whole regions, in vLLM, which this Space cannot run. Racing a batch of 8 to 18 lines against one sequence would compare two different things, so there are three views, each with one rule for everyone:
- Nothing batched (the race): TrOCR reads its lines one after the other, the reader one region.
- TrOCR's lines as one batch is timed too and drawn as the second bar, so nothing is hidden: on these regions it is about as fast as the reader alone in this Space's PyTorch, and 2 to 4 times slower than the reader with its draft.
- Everything batched (the three rates under the race): on our server, TrOCR in batches of 32 lines against the reader with 64 requests at a time in vLLM, which reads about ten times as many lines a second.
What you see
Each reader notes when every token became known. Once all three are timed, the page replays their readings from the same start, each at its measured pace; the buttons replay it, also at half and quarter speed. Nothing is sent to the page inside a timed stretch, so the display slows no reader down.
- On each manuscript a highlighter follows the reader: TrOCR's goes down the lines one by one, a character per decoding step; the reader's follows its tokens through the region.
- The strip above a reader's text has one block per reader pass, as wide as the tokens that pass produced: many thin ones alone, a few wide ones with the draft.
- In the DSpark lane, shaded text was proposed by the draft and kept; underlined text is the reader's own token, one per pass. Long shaded runs are where the time is saved.
- Point at a line (on a manuscript, in TrOCR's text or in Jev's list) to see it lit up everywhere.
Why the draft makes it faster
Decoding one token at a time runs the whole reader for each token. A check of 16 tokens costs about the same pass, and the reader keeps about 8 of them on average (the acceptance length). The draft is small (5 layers) and reads the reader's own hidden states, so its guesses are cheap and usually right. It helps most when one region is read at a time and its text is long; reading many requests at once keeps the GPU busy anyway.
How it runs here
vLLM runs the draft natively, but vLLM cannot run on ZeroGPU, so this Space decodes in PyTorch (dspark_hf.py).
Profiled with torch.profiler, eager decoding was launch-bound: about 1,500 small kernels a pass, the GPU busy 12 % of
the time. So:
- Each reader pass is replayed from a CUDA graph (transformers' StaticCache, the attention mask built on the GPU), for the reader alone and with the draft alike, which removes most of that launch overhead.
- The draft's whole proposal, its Markov sampling included, is one graph.
- One reader pass per draft block. The reader is hybrid: its linear-attention layers keep a recurrent state that cannot be cut back after rejected guesses. The checking pass keeps each such layer's inputs for the block, and the state is rebuilt over the accepted tokens only (a few kernels instead of a second pass).
The same region on our server, and vLLM
The outlined bars under the race were measured beforehand on our server (measure_reference.py, reference.json):
one whole RTX PRO 6000 Blackwell with nothing else running on it, TrOCR in transformers as here, the reader served by
vLLM alone and with its draft, one request at a time, the median of 5 runs. vLLM keeps the processed form of an image
it has seen, which saves 0.05 to 0.1 s a region, so every request sent the region with one pixel changed by one
level: new to the server, as a region is in production. Over the six example regions:
| a region read by | seconds |
|---|---|
| TrOCR, one line after the other | 1.78 β 3.52 |
| TrOCR, its lines as one batch | 0.72 β 1.86 |
| the reader in vLLM | 0.50 β 0.74 |
| the reader with its draft in vLLM | 0.32 β 0.50 |
The reader in vLLM is 1.4 to 2.6 times as fast as TrOCR's one batch, and 2.3 to 3.7 times with its draft. On six
other regions and the same whole GPU, this Space's PyTorch decoding was as fast as vLLM with the draft and 1.2 to 1.8
times slower alone; the live bars are slower again because ZeroGPU gives half a GPU. The reader alone writes the same
tokens as transformers' own generate.
What vLLM adds is reading many requests at once. On the first 1,000 test lines and 200 dev regions, one RTX PRO 6000 Blackwell:
| one at a time | many at a time | |
|---|---|---|
| lines: Swedish Lion TrOCR | 4.3 lines/s (batch 1) | 6.5 lines/s (batch 32) |
| lines: reader alone β with DSpark | 13.3 β 18.2 lines/s (1.37Γ) | 67 β 77 lines/s (64 at a time) |
| regions: reader alone β with DSpark | 1.95 β 4.08 regions/s (2.09Γ) | 11.9 β 12.7 regions/s (64 at a time) |
The 64-at-a-time rows are from a run where every image was new to the servers (two servers on one GPU, asked in turn); with each server on a GPU of its own they were 75 β 82 lines/s and 12.4 β 13.2 regions/s. Acceptance length 8.72; the character error rate is the same with and without the draft. OpenBMB's deployment guide also serves the reader and a DSpark draft with SGLang (through a pull request still open there in October 2026); we have measured vLLM only.
Jev
Jev compares a line's image with a candidate reading drawn as an image (@RENDERER@) and returns a score from 0 to 1.
Here each line is read on its own by TrOCR and by the MiniCPM reader, and Jev scores both; keeping, line by line, the
reading Jev scores higher is one way to combine two readers. A reading's errors are the characters it adds, drops or
gets wrong against the ground truth. The fold under the race is titled with what Jev did: on how many of the lines
where one reading has fewer errors it picked that one. Inside, the character error rate over the region of each
reader, of Jev's picks, and of the best possible picks (the reading with fewer errors on every line, the most any
chooser could get); then one row per line: each reading with its wrong characters marked, its errors and Jev's score
in percent, Jev's pick shaded and marked β when it has fewer errors than the other reading, β when it has more. Click
a row to see the line.
Jev catches gross mismatches well and near-misses badly. On 2,000 test pairs it tells the correct text from rule-made wrong candidates (dropped or swapped words, another line's text) with an AUROC of 0.988, but from the reader's own wrong readings, a letter or two off, with only 0.624: those were 1.2 % of its training pairs. So a high score here does not prove a reading right.
Limits
- The examples are drawn at random (seeded, one each from six collections) from GT-Lejonet's test pages, which no model here trained on. Our dev volumes would not do: they are cut out of the training pages, which TrOCR trained on (6 epochs) and our reader did not. TrOCR chose its checkpoint on the test pages, which flatters it a little. Other hands and periods read worse.
- TrOCR gets each line cut along the ground truth's outline. In practice a line detector finds the lines, and its mistakes would count against TrOCR; here they cannot.
- Near-greedy decoding can repeat a phrase to the token limit on dense pages (a loop); such a reading is wrong from where the loop starts.
- Timings vary with the GPU slice and the queue. A fresh GPU worker captures its CUDA graphs first (a few seconds, not counted in the timings shown).
- With the draft the reader's text can differ from its text alone by a token where two tokens were nearly tied: a check of 16 tokens rounds differently from a pass of one. vLLM shows the same between 1 and 64 requests at a time.