File size: 3,663 Bytes
2afd903 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
license: mit
base_model: baidu/Unlimited-OCR
tags:
- onnx
- webgpu
- ocr
- vision
- document-parsing
- onnxruntime-web
- browser
library_name: onnxruntime
---
# Unlimited-OCR DeepEncoder — ONNX (browser-ready vision stack)
The **complete vision encoder of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR)**
(the DeepSeek-OCR family DeepEncoder: **SAM ViT-B → CLIP-L fusion → linear projector**) exported as a
single ONNX graph, verified numerically identical to the PyTorch reference
(**torch-vs-onnxruntime cosine 1.0000000**, maxAbsDiff 5e-5).
Paired with the language-model GGUF, this runs the full Unlimited-OCR pipeline **entirely in a
browser tab** — image in, det-boxed markdown out, no server. Reference implementation:
[NakliTechie/gemma4-webgpu](https://github.com/NakliTechie/gemma4-webgpu) (hand-written WGSL
WebGPU engine with DeepSeek-V2-MoE decoder support; see `ocr-demo.html`). Measured on Apple
Metal-3: vision 1.5 s (onnxruntime-web WebGPU EP) + decode at ~125 tok/s → **single page OCR'd
in < 3 s in-tab**.
## Files
| file | what |
|---|---|
| `deepencoder_fp32.onnx` | SAM→CLIP→projector, opset 18, static `[1,3,1024,1024]` → `[1,256,1280]` (fp32, 1.6 GB) |
| `deepencoder_extras.npz` | `image_newline`, `view_seperator` embeddings (`[1280]` f32 each) — spliced host-side, not part of the graph |
| `deepencoder_ref_in.npy` / `deepencoder_ref_out.npy` | parity fixtures (seed-42 input + reference output) — verify your runtime reproduces cosine ≈ 1.0 |
| `control_doc.png` | the ground-truth test document used in the e2e verification |
## I/O contract
- **Input** `pixel_values` `[1,3,1024,1024]` f32 — RGB, normalized `(x/255 − 0.5)/0.5` (mean/std 0.5, per the upstream processor config).
- **Output** `vision_embeds` `[1,256,1280]` f32 — 256 vision tokens (16×16 grid) already projected to the decoder's hidden size.
### Splicing into the decoder sequence (single 1024² global view)
```
[BOS(0)]
+ 16 rows × ( 16 patch embeds [row-major] + image_newline )
+ view_seperator → 273 embedding rows total
+ prompt token ids
```
The 273 rows replace `<image>` placeholder tokens (id 128815) — feed them as `inputs_embeds`.
### ⚠ The prompt matters more than you think
Use this model family's canonical prompt: **`<image>document parsing.`**
DeepSeek-OCR-v1 phrasings fail on Unlimited-OCR — `<image>\nFree OCR.` produces an **immediate EOS
even in the bf16 reference**, and `<|grounding|>Convert the document to markdown.` makes it recite
instruction boilerplate. (Verified against the full-precision HF stack; see the engine repo's
`reference/pytorch/hf_image_control.py`.)
## The rest of the pipeline
- **Decoder GGUF** (DeepSeek-V2 MoE, 64×550M, 12 layers — *no MLA despite the family name*:
`use_mla: false`, plain Llama MHA): community K-quants at
[sahilchachra/Unlimited-OCR-GGUF](https://huggingface.co/sahilchachra/Unlimited-OCR-GGUF)
(Q4_K_M 1.95 GB works with the engine's in-shader q4k/q8 storage).
- **Browser engine**: [NakliTechie/gemma4-webgpu](https://github.com/NakliTechie/gemma4-webgpu) —
WGSL kernels incl. on-GPU top-6 MoE routing and batched expert GEMVs; crossLabDiff-verified
against the HF bf16 reference (per-layer sweep, argmax match).
## Provenance & license
Weights are a mechanical export of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR)
(MIT, © 2026 Baidu — notice retained per license). Export script:
`reference/pytorch/export_deepencoder_onnx.py` in the engine repo. fp32; an fp16 (~800 MB) pass is
planned — mind LayerNorm precision if you convert yourself.
|