| --- |
| license: mit |
| base_model: baidu/Unlimited-OCR |
| tags: |
| - onnx |
| - webgpu |
| - ocr |
| - vision |
| - document-parsing |
| - onnxruntime-web |
| - browser |
| library_name: onnxruntime |
| --- |
| |
| # Unlimited-OCR DeepEncoder — ONNX (browser-ready vision stack) |
|
|
| The **complete vision encoder of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR)** |
| (the DeepSeek-OCR family DeepEncoder: **SAM ViT-B → CLIP-L fusion → linear projector**) exported as a |
| single ONNX graph, verified numerically identical to the PyTorch reference |
| (**torch-vs-onnxruntime cosine 1.0000000**, maxAbsDiff 5e-5). |
|
|
| Paired with the language-model GGUF, this runs the full Unlimited-OCR pipeline **entirely in a |
| browser tab** — image in, det-boxed markdown out, no server. Reference implementation: |
| [NakliTechie/gemma4-webgpu](https://github.com/NakliTechie/gemma4-webgpu) (hand-written WGSL |
| WebGPU engine with DeepSeek-V2-MoE decoder support; see `ocr-demo.html`). Measured on Apple |
| Metal-3: vision 1.5 s (onnxruntime-web WebGPU EP) + decode at ~125 tok/s → **single page OCR'd |
| in < 3 s in-tab**. |
|
|
| ## Files |
|
|
| | file | what | |
| |---|---| |
| | `deepencoder_fp32.onnx` | SAM→CLIP→projector, opset 18, static `[1,3,1024,1024]` → `[1,256,1280]` (fp32, 1.6 GB) | |
| | `deepencoder_extras.npz` | `image_newline`, `view_seperator` embeddings (`[1280]` f32 each) — spliced host-side, not part of the graph | |
| | `deepencoder_ref_in.npy` / `deepencoder_ref_out.npy` | parity fixtures (seed-42 input + reference output) — verify your runtime reproduces cosine ≈ 1.0 | |
| | `control_doc.png` | the ground-truth test document used in the e2e verification | |
|
|
| ## I/O contract |
|
|
| - **Input** `pixel_values` `[1,3,1024,1024]` f32 — RGB, normalized `(x/255 − 0.5)/0.5` (mean/std 0.5, per the upstream processor config). |
| - **Output** `vision_embeds` `[1,256,1280]` f32 — 256 vision tokens (16×16 grid) already projected to the decoder's hidden size. |
|
|
| ### Splicing into the decoder sequence (single 1024² global view) |
|
|
| ``` |
| [BOS(0)] |
| + 16 rows × ( 16 patch embeds [row-major] + image_newline ) |
| + view_seperator → 273 embedding rows total |
| + prompt token ids |
| ``` |
|
|
| The 273 rows replace `<image>` placeholder tokens (id 128815) — feed them as `inputs_embeds`. |
|
|
| ### ⚠ The prompt matters more than you think |
|
|
| Use this model family's canonical prompt: **`<image>document parsing.`** |
| DeepSeek-OCR-v1 phrasings fail on Unlimited-OCR — `<image>\nFree OCR.` produces an **immediate EOS |
| even in the bf16 reference**, and `<|grounding|>Convert the document to markdown.` makes it recite |
| instruction boilerplate. (Verified against the full-precision HF stack; see the engine repo's |
| `reference/pytorch/hf_image_control.py`.) |
|
|
| ## The rest of the pipeline |
|
|
| - **Decoder GGUF** (DeepSeek-V2 MoE, 64×550M, 12 layers — *no MLA despite the family name*: |
| `use_mla: false`, plain Llama MHA): community K-quants at |
| [sahilchachra/Unlimited-OCR-GGUF](https://huggingface.co/sahilchachra/Unlimited-OCR-GGUF) |
| (Q4_K_M 1.95 GB works with the engine's in-shader q4k/q8 storage). |
| - **Browser engine**: [NakliTechie/gemma4-webgpu](https://github.com/NakliTechie/gemma4-webgpu) — |
| WGSL kernels incl. on-GPU top-6 MoE routing and batched expert GEMVs; crossLabDiff-verified |
| against the HF bf16 reference (per-layer sweep, argmax match). |
|
|
| ## Provenance & license |
|
|
| Weights are a mechanical export of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) |
| (MIT, © 2026 Baidu — notice retained per license). Export script: |
| `reference/pytorch/export_deepencoder_onnx.py` in the engine repo. fp32; an fp16 (~800 MB) pass is |
| planned — mind LayerNorm precision if you convert yourself. |
|
|
