README.md

---
license: mit
base_model: baidu/Unlimited-OCR
tags:
  - onnx
  - webgpu
  - ocr
  - vision
  - document-parsing
  - onnxruntime-web
  - browser
library_name: onnxruntime
---

# Unlimited-OCR DeepEncoder — ONNX (browser-ready vision stack)

The **complete vision encoder of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR)**
(the DeepSeek-OCR family DeepEncoder: **SAM ViT-B → CLIP-L fusion → linear projector**) exported as a
single ONNX graph, verified numerically identical to the PyTorch reference
(**torch-vs-onnxruntime cosine 1.0000000**, maxAbsDiff 5e-5).

Paired with the language-model GGUF, this runs the full Unlimited-OCR pipeline **entirely in a
browser tab** — image in, det-boxed markdown out, no server. Reference implementation:
[NakliTechie/gemma4-webgpu](https://github.com/NakliTechie/gemma4-webgpu) (hand-written WGSL
WebGPU engine with DeepSeek-V2-MoE decoder support; see `ocr-demo.html`). Measured on Apple
Metal-3: vision 1.5 s (onnxruntime-web WebGPU EP) + decode at ~125 tok/s → **single page OCR'd
in < 3 s in-tab**.

## Files

| file | what |
|---|---|
| `deepencoder_fp32.onnx` | SAM→CLIP→projector, opset 18, static `[1,3,1024,1024]` → `[1,256,1280]` (fp32, 1.6 GB) |
| `deepencoder_extras.npz` | `image_newline`, `view_seperator` embeddings (`[1280]` f32 each) — spliced host-side, not part of the graph |
| `deepencoder_ref_in.npy` / `deepencoder_ref_out.npy` | parity fixtures (seed-42 input + reference output) — verify your runtime reproduces cosine ≈ 1.0 |
| `control_doc.png` | the ground-truth test document used in the e2e verification |

## I/O contract

- **Input** `pixel_values` `[1,3,1024,1024]` f32 — RGB, normalized `(x/255 − 0.5)/0.5` (mean/std 0.5, per the upstream processor config).
- **Output** `vision_embeds` `[1,256,1280]` f32 — 256 vision tokens (16×16 grid) already projected to the decoder's hidden size.

### Splicing into the decoder sequence (single 1024² global view)

```
[BOS(0)]
+ 16 rows × ( 16 patch embeds [row-major] + image_newline )
+ view_seperator                       → 273 embedding rows total
+ prompt token ids
```

The 273 rows replace `<image>` placeholder tokens (id 128815) — feed them as `inputs_embeds`.

### ⚠ The prompt matters more than you think

Use this model family's canonical prompt: **`<image>document parsing.`**
DeepSeek-OCR-v1 phrasings fail on Unlimited-OCR — `<image>\nFree OCR.` produces an **immediate EOS
even in the bf16 reference**, and `<|grounding|>Convert the document to markdown.` makes it recite
instruction boilerplate. (Verified against the full-precision HF stack; see the engine repo's
`reference/pytorch/hf_image_control.py`.)

## The rest of the pipeline

- **Decoder GGUF** (DeepSeek-V2 MoE, 64×550M, 12 layers — *no MLA despite the family name*:
  `use_mla: false`, plain Llama MHA): community K-quants at
  [sahilchachra/Unlimited-OCR-GGUF](https://huggingface.co/sahilchachra/Unlimited-OCR-GGUF)
  (Q4_K_M 1.95 GB works with the engine's in-shader q4k/q8 storage).
- **Browser engine**: [NakliTechie/gemma4-webgpu](https://github.com/NakliTechie/gemma4-webgpu) —
  WGSL kernels incl. on-GPU top-6 MoE routing and batched expert GEMVs; crossLabDiff-verified
  against the HF bf16 reference (per-layer sweep, argmax match).

## Provenance & license

Weights are a mechanical export of [baidu/Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR)
(MIT, © 2026 Baidu — notice retained per license). Export script:
`reference/pytorch/export_deepencoder_onnx.py` in the engine repo. fp32; an fp16 (~800 MB) pass is
planned — mind LayerNorm precision if you convert yourself.