[Proposal / Benchmark] ~3.7x Faster Stage-2 Inference using vLLM Backend (Validated across 5,000+ PDFs)
Hi Bodhan-AI Team,
First of all, congratulations on the release of IndicOCR. It has demonstrated outstanding recognition quality on complex Devanagari documents, table layouts, and multilingual text.
While processing high-volume document pipelines on NVIDIA Ada GPUs (RTX PRO 6000), I observed that Stage 2 (IndicBlockOCR) accounts for over 95% of per-page latency when running the default Hugging Face transformers backend (~10–12 seconds/page).
To optimize this for production scale, I implemented and deployed a drop-in vLLM recognizer backend. I wanted to share my empirical benchmarks and open-source implementation with the community:
GitHub Repository: https://github.com/SarwadnyaMahajan/indic-ocr-vllm
Benchmark and Production Validation
1. Controlled Character-Level Evaluation (Scientific Baseline)
To strictly measure Character Error Rate (CER) and verify that vLLM does not degrade Devanagari conjuncts or punctuation, I ran both backends side-by-side on an identical 15-page / 111-crop evaluation set (scanned Marathi, English, tables, and slides across 5 domains):
| Language | Crops | Identical Crops | Reference Chars | Differing Chars | CER | Speedup |
|---|---|---|---|---|---|---|
| Marathi (Devanagari) | 68 | 67 / 68 | 17,265 | 1 | 0.0058% | 3.79x |
| English | 43 | 40 / 43 | 14,235 | 20 | 0.14% | 3.60x |
| Total | 111 | 107 / 111 | 31,500 | 21 | 0.067% | 3.67x |
- Accuracy Parity: Across 17,265 Marathi characters, there was only 1 character difference (U+0903 Visarga vs U+003A ASCII colon), with all actual vocabulary, letters, and numbers matching bit-for-bit.
- Latency: End-to-end runtime dropped from 120.8s (Hugging Face) to 32.9s (vLLM) — a 3.67x overall speedup.
2. Large-Scale Production Run
Following the controlled benchmark, I deployed the vLLM engine into production. It has now successfully processed over 3,000+ complex Marathi and English PDFs (including multi-hundred page documents) with consistent stability and zero memory leaks.
Key Technical Insights for Qwen3_5ForConditionalGeneration on vLLM
- CUDA Graphs (enforce_eager=False):
- Eager mode yielded only a 2.25x speedup.
- Enabling CUDA graph capture unlocked the full 3.7x–3.9x speedup.
- Mamba Cache Block Allocation (max_num_seqs):
- Because this architecture is a GDN/Mamba hybrid, every decode sequence consumes a Mamba cache block. vLLM’s default max_num_seqs=1024 exceeds available cache blocks at conservative memory fractions. Explicitly setting max_num_seqs=64 allows seamless CUDA graph capture and matches standard crop batch sizes.
- SentencePiece Metaspace Cleanup:
- vLLM detokenizes internally; replacing chr(0x2581) with " " before stripping preserves exact whitespace equivalence with Hugging Face.
- Sampler Optimization:
- Setting VLLM_USE_FLASHINFER_SAMPLER=0 ensures smooth execution on machines without nvcc compiler toolchains during greedy decoding.
The standalone drop-in recognizer, complete usage examples, and installation instructions are available here:
https://github.com/SarwadnyaMahajan/indic-ocr-vllm
Would the maintainers be interested in a PR integrating an optional --backend {hf, vllm} into the official IndicOCR inference scripts?
Happy to coordinate and prepare the PR according to your contribution guidelines.