Hugging Face's logo

ameraino11
/
baberu-ocr-webgpu

Image-to-Text
ONNX
Japanese
English
Chinese
baberu_ocr
ocr
manga
webgpu
on-device
quantized
Model card Files
xet
 Community
baberu-ocr-webgpu / README.md
ameraino11's picture
ameraino11
Optimize Baberu WebGPU decoder memory execution
cdf34e1 verified
preview code
|
Raw
History Blame Contribute Delete
7.04 kB
metadata 
license: apache-2.0
language:
  - ja
  - en
  - zh
tags:
  - ocr
  - manga
  - onnx
  - webgpu
  - on-device
  - quantized
base_model: genshiai-daichi/baberu-ocr
pipeline_tag: image-to-text

Baberu OCR experimental full-WebGPU ONNX bundles

Personal model hosting/download mirror. These files are hosted so I can download and test my two experimental browser models. This is not an official Baberu release, not production ready, and not a recommendation for public deployment without independent accuracy, browser, memory, and security tests.

This repository contains two complete client-only OCR bundles derived from Baberu OCR, together with the ONNX export, validation, benchmark, and browser WebGPU source. No inference server is required after the browser downloads the files.

Latest recommended models

The latest decoder uses one complete six-layer QDQ graph for both prefill and cached token steps. It accepts an INT32 token ID, gathers the selected INT8 embedding row first, and only then runs DequantizeLinear. This exact rewrite avoids materializing the full [14630,512] FP32 embedding table (28.6 MiB).

Folder Vision encoder Latest decoder Latest ONNX download
variants/webgpu-121/ INT4 vision_int4.onnx, 52.29 MB decoder_unified_gather_qdq_int8.onnx, 34.65 MB 86.94 MB
variants/webgpu-242/ FP16 vision_fp16.onnx, 172.92 MB decoder_unified_gather_qdq_int8.onnx, 34.65 MB 207.57 MB

The smaller downloads come from removing the duplicate decoder weights that were embedded separately in the old prefill and step files. They are not reduced models: all six layers, hidden size 512, intermediate size 1,536, eight attention heads, two KV heads, and the full 14,630-token vocabulary remain. The decoder retains all 33,069,056 quantized weight elements.

Each variant also retains the old split prefill/step graphs for compatibility and decoder_unified_qdq_int8.onnx as a one-hot unified baseline. Hugging Face deduplicates identical decoder blobs stored under both variant folders.

WebGPU execution path

The vision encoder resizes the complete RGB text crop to 224 x 224, applies ImageNet normalization, and returns vision_embeds with shape [1,256,512]. The unified decoder consumes those embeddings and keeps all six key/six value cache pairs GPU-resident during generation.

The vision encoder, decoder compute, and KV tensors execute with ONNX Runtime WebGPU. Only the FP32 logits return to JavaScript for the published greedy token selection policy. A GPU ArgMax graph and fused GroupQueryAttention graph were tested locally but removed because both were slower on the tested runtime.

The QDQ decoder stores INT8 weights and executes DequantizeLinear followed by FP32 MatMul on WebGPU. The token embedding is the exception: its INT8 Gather runs before dequantization. It contains neither DynamicQuantizeLinear nor MatMulInteger, which would leave the intended browser WebGPU path.

13 difficult showcase-image check

These public Hayai OCR showcase crops are useful conversion checks, but they are a small selected set rather than a neutral accuracy corpus. CER uses NFKC normalization and removes whitespace.

Variant Provider nCER Exact Median/crop
Upstream native 121 ONNX Runtime CPU 13.87% 5/13 197 ms
Upstream native 242 ONNX Runtime CPU 12.61% 5/13 137 ms
Latest webgpu-121 Gather ONNX Runtime WebGPU 14.29% 4/13 about 300 ms
Latest webgpu-242 Gather ONNX Runtime WebGPU 11.76% 5/13 190 ms in the latest warm-cache check

All 13 outputs from the optimized 121 Gather graph were identical to the previous unified one-hot graph. The 242 Gather graph also retained the earlier 242 WebGPU result of 11.76% nCER and 5/13 exact. Latency depends heavily on GPU, shader cache, browser, and system load; these numbers are not portable.

The Gather-before-dequantize decoder is also bit-exact against the previous token-Gather decoder on ONNX Runtime CPU at prefill, cache length 257, and the maximum tested cache length 383. Three full WebGPU runs per vision tier retained all 13 previous output strings.

In one same-process 121 comparison, the optimized graph sampled an 833 MB shared GPU-process peak and 577 MB renderer peak, versus 866 MB and 613 MB for the unified one-hot baseline. Those values must not be added as model-owned RAM: the maxima were not necessarily simultaneous, the GPU process is shared, and Apple silicon uses unified memory.

Full outputs are in benchmarks/. Source images are not redistributed; download_hayai13.py downloads the public examples from a pinned upstream revision.

Accuracy status

Full converted-model accuracy remains unknown. The upstream 121 and 242 releases report Japanese nCER 0.0893 and 0.0867, but those values must not be assigned to these conversions. The exact 2,000-crop reproduction remains blocked on gated Manga109-v2026 data and the unpublished exact crop manifest and evaluation scripts.

The 13-image results and synthetic FP32/QDQ top-token comparisons are port validation, not evidence that these files are production ready.

Run the hosted models

Clone the repository with Git LFS/Xet support, then:

npm install
sh run-browser-server.sh

Open the optimized 121-capability bundle:

http://127.0.0.1:8765/webgpu-e2e.html?layout=hf&bundle=compact-unified-gather

Or the optimized 242-capability bundle:

http://127.0.0.1:8765/webgpu-e2e.html?layout=hf&bundle=balanced-unified-gather

Choose a tightly cropped text image. Add &suite=hayai13 after running python3 download_hayai13.py to reproduce the 13-image checks. Both test paths explicitly release their ONNX sessions and destroy the dedicated GPU device when complete.

Included port source

  • exact FP32 decoder and vision exporters;
  • complete unified and token-ID Gather decoder exporters;
  • per-output-channel INT8-QDQ converter and CPU parity checks;
  • browser WebGPU end-to-end and decoder smoke harnesses;
  • exact Gather-before-dequantize and experimental execution optimizers;
  • staged vision-worker and decoder-only cold/warm memory harnesses;
  • native CPU and MangaOCR comparison harnesses;
  • isolated Python/npm dependencies and local server;
  • benchmark helpers and generated validation reports.

Detailed local experiment notes are in docs/experiment-results.md.

Provenance and license

  • Base model: genshiai-daichi/baberu-ocr
  • Base revision: d9cc13153e9a1cd8fdfa3b7b1cc329da2020aeae
  • ONNX opset: 17
  • Decoder conversion: symmetric per-output-channel INT8 weight-only QDQ
  • Browser runtime tested: ONNX Runtime Web 1.27, WebGPU provider
  • License: Apache-2.0; see LICENSE and NOTICE

The vision ONNX files are unmodified upstream copies. The QDQ decoder graphs and browser port are an independent experiment, not an official Baberu release.