Copy from bodhan-ai/indic-doc-parser

.gitattributes

@@ -33,3 +33,26 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+weights/ocr/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/layout-bengali.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-english-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-english.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-hindi-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-hindi.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-santali-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-santali.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-telugu.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-bengali-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-urdu-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-urdu.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-hw-english-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-hw-english.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-hw-hindi-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-hw-hindi.png filter=lfs diff=lfs merge=lfs -text
+assets/example-ramanujan-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/example-ramanujan.png filter=lfs diff=lfs merge=lfs -text
+assets/layout-telugu-thumb.png filter=lfs diff=lfs merge=lfs -text
+assets/diagram.png filter=lfs diff=lfs merge=lfs -text
+assets/cand-hindi-maths-g10-6pr6eq-7dcd5d95-p20.png filter=lfs diff=lfs merge=lfs -text
+assets/gallery-1-english-math-ramanujan.png filter=lfs diff=lfs merge=lfs -text
+assets/gallery-2-telugu-novel.png filter=lfs diff=lfs merge=lfs -text

ARCHITECTURE.md

@@ -0,0 +1,95 @@
+# Architecture
+
+Two models with a JSON file between them.
+
+```text
+page image ──> IndicDocLayout ──> layout JSON ──> IndicBlockOCR ──> page JSON + markdown
+```
+
+`IndicDocLayout` finds the blocks and orders them. `IndicBlockOCR` transcribes one block crop
+per request. Everything between the two models (cropping, prompt selection, reassembly) is this
+package.
+
+## The call path
+
+One page, top to bottom, with the file that owns each step.
+
+```text
+IndicDocParser.parse(image)                              idp_offline.py
+
+  IndicDocLayout.detect(image)                           idp_offline.py
+    _open                     image -> PIL RGB           idp_offline.py
+    IndicDocLayoutBackend.detect                         idp_layout.py
+      infer                   -> [y0,x0,y1,x1] at 0-1000 idp_model_infer.py
+      convert + clamp_to_page -> pixel [x0,y0,x1,y1]     idp_layout.py, blocks.py
+      clean_layout            drop duplicate boxes       idp_blocks.py
+      _densify                order -> gap-free 0..n-1   idp_layout.py
+    = PageResult, every block has text None              idp_types.py
+
+  IndicBlockOCR.run(image, layout)                       idp_offline.py
+    _as_page                  dict -> PageResult, validated
+    is_transcribed            drop OCR_SKIP_LABELS       idp_contract.py
+    resolve_nested_equations  drop nested equations      idp_blocks.py
+    build_requests            -> [CropRequest]           idp_recognizer.py
+      crop_for + area_clamp   block -> image             idp_crops.py
+      prompt_for(block.type)  type  -> prompt            idp_contract.py
+    backend.transcribe        [CropRequest] -> [str]     a recognizer backend
+    match by order            texts -> blocks            idp_offline.py
+    reconstruct               blocks -> markdown         idp_reconstruct.py
+  = PageResult, every block has text
+```
+
+Blocks that were skipped are not deleted. They come back with `text: ""`.
+
+## Files by role
+
+**Contract and data.** The vocabulary everything else shares.
+
+| file | holds |
+| --- | --- |
+| `idp_contract.py` | prompts, the label to type map, which labels are skipped |
+| `idp_types.py` | `Block`, `PageResult`, and every tunable config |
+
+**The layout model.** Torch lives here and nowhere else.
+
+| file | holds |
+| --- | --- |
+| `idp_model_infer.py` | preprocessing and decode for one page |
+| `idp_model_labels.py` | the 37 classes, pure Python |
+| `idp_model_ppdoc.py` | the PP-DocLayoutV3 subclass |
+| `idp_model_order_loss.py` | reading-order decode, training loss |
+
+**Geometry and crops.** No model, no network.
+
+| file | holds |
+| --- | --- |
+| `idp_blocks.py` | box math, `clean_layout`, nested-equation dedup |
+| `idp_crops.py` | `crop_for` and `area_clamp` |
+
+**Backends.** Two Protocols, five implementations, one per deployment.
+
+| class | file | used by |
+| --- | --- | --- |
+| `IndicDocLayoutBackend` | `idp_layout.py` | the real detector |
+| `JsonLayoutBackend` | `idp_layout.py` | replaying a layout file, no torch |
+| `HfRecognizer` | `idp_recognizer.py` | plain transformers, the Hub package |
+
+
+**Orchestration.**
+
+| file | holds |
+| --- | --- |
+| `idp_offline.py` | `IndicDocLayout`, `IndicBlockOCR`, `IndicDocParser` |
+| `idp_recognizer.py` | `CropRequest`, `build_requests`, the recognizer Protocol |
+| `idp_reconstruct.py` | blocks to markdown, math and hyphen repair |
+
+## Invariants
+
+1. **Two box conventions.** The model emits `[y0, x0, y1, x1]` normalised to 0-1000. The
+   pipeline uses pixel `[x0, y0, x1, y1]`. 
+2. **`order` must be gap-free and 0-based.** Transcriptions are matched back to blocks by
+   position, so a gap or a duplicate moves text onto the wrong block.
+3. **The crop clamp is on area, not on a side.** Pinning a side explodes elongated crops. See
+   `CropConfig` in `idp_types.py`.
+4. **`type` selects the prompt.** A wrong `type` changes what the model was asked to do, not
+   just how a block is labelled. Unrecognised labels are rejected for this reason.

README.md

@@ -0,0 +1,359 @@
+---
+language:
+- en
+- as
+- bn
+- brx
+- doi
+- gu
+- hi
+- kn
+- ks
+- kok
+- mai
+- ml
+- mni
+- mr
+- ne
+- or
+- pa
+- sa
+- sat
+- sd
+- ta
+- te
+- ur
+pipeline_tag: image-to-text
+tags:
+- ocr
+- document-parsing
+- layout-analysis
+- reading-order
+- indic
+- vision-language-model
+- qwen
+- rt-detr
+---
+
+
+<div align="center">
+
+[![Pipeline](https://img.shields.io/badge/Pipeline-Layout%20%2B%20OCR-F97316?style=flat)](#the-two-models)
+[![Layout](https://img.shields.io/badge/IndicDocLayout-33M-F97316?style=flat)](#the-two-models)
+[![Recognizer](https://img.shields.io/badge/IndicBlockOCR-0.8B-F97316?style=flat)](#the-two-models)
+[![Languages](https://img.shields.io/badge/Languages-23-F97316?style=flat)](#supported-languages)
+[![License](https://img.shields.io/badge/License-Apache--2.0-F97316?style=flat)](#license)
+
+</div>
+
+**Document parsing for English and 22 Indian languages, printed and handwritten.** A page image
+in; reading-ordered Markdown out, with math as LaTeX and tables as HTML or Markdown, plus
+per-block JSON.
+
+<div align="center">
+
+<img src="assets/diagram.png" alt="IndicDocParser: page image to layout detection with reading order, then block-level OCR, then Markdown" width="100%">
+
+</div>
+
+IndicDocParser reads a document page and returns its text in reading order. It is a modular,
+two-stage parser: **IndicDocLayout** detects the blocks on the page and orders them, and
+**IndicBlockOCR** transcribes the textual blocks. The two stages communicate through a structured
+JSON file, so either stage can be used independently or replaced with another implementation.
+
+[`ARCHITECTURE.md`](ARCHITECTURE.md) traces one page through the whole call path, names what
+each module does, and lists the invariants that break the output silently when violated.
+
+---
+
+<h2 id="examples" style="color:#F97316;">Examples</h2>
+
+Detected blocks with their reading order on the left, the transcription on the right.
+
+<div align="center">
+  <img src="assets/gallery-1-english-math-ramanujan.png" alt="A page from Ramanujan's notebooks: text and display equations detected in reading order, with the transcription rendering the mathematics as LaTeX" width="100%">
+  <p><b>Example #1. English page with dense mathematics.</b></p>
+</div>
+
+<div align="center">
+  <img src="assets/gallery-2-telugu-novel.png" alt="A printed Telugu novel page: paragraph blocks and a page number detected and numbered in reading order, with the Telugu transcription beside it" width="100%">
+  <p><b>Example #2. Printed Telugu page.</b></p>
+</div>
+
+<div align="center">
+  <img src="assets/cand-hindi-maths-g10-6pr6eq-7dcd5d95-p20.png" alt="A handwritten Hindi maths exercise on ruled paper: alternating Equation and Paragraph blocks detected in reading order, with the transcription rendering the algebra as LaTeX" width="100%">
+  <p><b>Example #3. Handwritten Hindi maths.</b></p>
+</div>
+
+---
+
+<h2 id="model-summary" style="color:#F97316;">Model Summary</h2>
+
+| | IndicDocLayout | IndicBlockOCR |
+| --- | --- | --- |
+| **Role** | Layout detection + reading order | Block-level text recognition |
+| **Architecture** | PP-DocLayoutV3 / RT-DETR | Qwen3.5-0.8B |
+| **Parameters** | 33 M | 0.8 B |
+| **Precision** | fp32 | bf16 |
+| **In this repo** | `weights/layout` (133 MB) | `weights/ocr` (1.7 GB) |
+| **Output** | Layout JSON | Markdown + block JSON |
+
+
+IndicBlockOCR uses the **Sarvam-30B tokenizer**, with a vocabulary designed to cover Indian
+scripts. IndicDocLayout is a fine-tune of PP-DocLayoutV3/RT-DETR, trained with a 37-class
+taxonomy designed for education-domain documents.
+
+IndicDocLayout predicts a labelled bounding box for each detected layout element.
+The 37 supported labels are:
+
+> Advertisement, Answer, Author, Chapter-end-section, Chapter-title, Chart, Code, Contact-info, Dateline, Diagram, Equation, Expression, Flag, Folio, Footer, Footnote, Header, Image, Image-caption, Index, Infobox, List, MCQ, Page-number, Paragraph, Placeholder-text, Question, Reference, Section-title, Solved-example, Sub-section-title, Sub-sub-section-title, Table, Table-caption, Table-of-contents, Title, Website-link
+
+---
+
+<h2 id="supported-languages" style="color:#F97316;">Supported languages</h2>
+
+**Printed** page recognition is supported across English and the 22 constitutionally recognised Indian languages: Assamese, Bengali, Bodo, Dogri,
+Gujarati, Hindi, Kannada, Kashmiri, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali,
+Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu, Urdu.
+
+**Handwriting** recognition currently supports English and 12 Indian languages: Hindi, Bengali,
+Telugu, Marathi, Tamil, Gujarati, Kannada, Malayalam, Odia, Punjabi, Assamese, and Urdu.
+
+Handwriting quality is still a work in progress, particularly across different writing styles. We are working on improving recognition and extending support to additional languages.
+
+---
+
+<h2 id="usage" style="color:#F97316;">Usage</h2>
+
+<h3 style="color:#F97316;">Installation</h3>
+
+The repo ships an installer that reads your driver and picks matching CUDA wheels. If you work in a
+virtual environment, please activate it first, as the installer installs into whichever
+Python is active.
+
+```bash
+IDP=$(python -c "from huggingface_hub import snapshot_download as d; print(d('bodhan-ai/indic-doc-parser'))")
+cd "$IDP" && ./install.sh
+```
+
+It will use `uv` if that is available, and `pip` otherwise. Where running a shell script is not
+convenient, [TROUBLESHOOTING.md](TROUBLESHOOTING.md) lists the two commands it runs.
+
+<h3 style="color:#F97316;">Basic inference</h3>
+
+```python
+import sys
+from huggingface_hub import snapshot_download
+
+repo = snapshot_download("bodhan-ai/indic-doc-parser")
+sys.path.insert(0, repo)                    # the code ships in the repo
+from indic_doc_parser import IndicDocParser
+
+parser = IndicDocParser.from_pretrained(repo)
+
+page = parser.parse("page.png")
+print(page["markdown"])                     # reading-ordered Markdown
+```
+
+`page` also carries the per-block detail, which you can save as follows:
+
+```python
+import json
+
+with open("page.json", "w", encoding="utf-8") as f:
+    json.dump(page, f, ensure_ascii=False, indent=2)
+```
+
+<h3 style="color:#F97316;">Running one stage at a time</h3>
+
+To run the two stages separately:
+
+```python
+from indic_doc_parser import IndicDocLayout, IndicBlockOCR
+
+layout = IndicDocLayout(f"{repo}/weights/layout").detect("page.png")
+page = IndicBlockOCR(f"{repo}/weights/ocr").run("page.png", layout)
+```
+
+`run()` takes a layout object, a dict, or the path to a layout JSON file.
+
+---
+
+<h2 id="output" style="color:#F97316;">Output</h2>
+
+`parser.parse("page.png")` returns the page metadata and its blocks in reading order:
+
+```json
+{
+  "image": "sample1.png",
+  "width": 800,
+  "height": 1273,
+  "blocks": [
+    {"order": 0, "label": "Header", "type": "PageHeader",
+     "bbox_xyxy": [345.6, 51.7, 437.1, 114.7], "conf": 0.6, "text": ""},
+    {"order": 1, "label": "Page-number", "type": "PageNumber",
+     "bbox_xyxy": [367.8, 78.9, 413.2, 107.4], "conf": 0.747, "text": "229"},
+    {"order": 2, "label": "Paragraph", "type": "Text",
+     "bbox_xyxy": [77.9, 121.3, 711.8, 199.4], "conf": 0.863,
+     "text": "Thus we see that, if we can prove that twice the L.H.S. of (30) ..."}
+  ]
+}
+```
+
+| field | meaning |
+| --- | --- |
+| `order` | reading-order rank, 0-based and gap-free |
+| `label` | the raw IndicDocLayout class (37-class taxonomy) |
+| `type` | coarse pipeline category: `Text`, `Table`, `Equation`, `Title`, ... |
+| `bbox_xyxy` | pixel box `[x0, y0, x1, y1]` |
+| `conf` | detection confidence |
+| `text` | transcription; `""` for blocks not sent to the recognizer |
+
+**Note:** Figures, charts, advertisements, running headers, and footers are not sent through the recognizer
+by default. They remain in the JSON with `text: ""`, so you can see what was detected and where.
+Page numbers and other margin text such as folios are transcribed.
+
+<h3 style="color:#F97316;">Schemas</h3>
+
+Machine-readable JSON Schema for each envelope, in [`schemas/`](schemas):
+
+| file | describes |
+| --- | --- |
+| `layout_output.schema.json` | The layout file: what **IndicDocLayout** writes and **IndicBlockOCR** reads. Blocks and reading order, before any text is read, so there is **no** `text` key at all. |
+| `parse_output.schema.json` | The parsed page shown above. Every block now has `text`; `""` means the block was detected but deliberately not sent to the recognizer. |
+
+A layout from your own detector must use a `label` from the 37-class taxonomy, or declare `type`
+explicitly. An unrecognised label is rejected rather than silently read as prose.
+
+<h3 style="color:#F97316;">Table format</h3>
+
+Tables come back as HTML by default. Choose the format when you construct the parser:
+
+```python
+parser = IndicDocParser.from_pretrained(repo)                            # HTML (default)
+parser = IndicDocParser.from_pretrained(repo, table_format="markdown")   # Markdown
+```
+
+---
+
+<h2 id="performance" style="color:#F97316;">Performance</h2>
+
+<h3 style="color:#F97316;">OmniDocBench 1.6 (english subset)</h3>
+
+| OmniDocBench 1.6 (english subset) | Overall↑ | TextEdit↓ | FormulaCDM↑ | TableTEDS↑ | TableTEDS-S↑ | Read OrderEdit↓ |
+| --- | :---: | :---: | :---: | :---: | :---: | :---: |
+| PaddleOCRVL-1.6 | 96.36 | 0.03 | 98.55 | 93.37 | 96.33 | 0.09 |
+| Chandra OCR 2 | 93.11 | 0.04 | 96.93 | 86.07 | 90.34 | 0.09 |
+| **IndicOCR (ours)** | **92.76** | **0.04** | **97.53** | **85.10** | **90.58** | **0.11** |
+| GPT-5.6-sol | 92.46 | 0.04 | 95.42 | 85.87 | 90.98 | 0.10 |
+| Gemini 3.1 Pro | 91.15 | 0.06 | 95.53 | 83.46 | 88.77 | 0.13 |
+| Surya OCR 2 (model) | 91.13 | 0.04 | 95.67 | 81.61 | 86.37 | 0.10 |
+| Sarvam Vision | 90.08 | 0.04 | 97.62 | 76.82 | 82.01 | 0.10 |
+| Gemma 31B | 86.71 | 0.09 | 89.48 | 79.79 | 85.19 | 0.19 |
+| Nemotron Parse 2 | 79.12 | 0.159 | 78.94 | 74.32 | 81.09 | 0.29 |
+
+<h3 style="color:#F97316;">olmOCR-Bench (<a href="https://huggingface.co/datasets/sarvamai/olmOCR-Bench-English" style="color:#F97316;">english subset</a>)</h3>
+
+| OlmoOCRBench ([english subset](https://huggingface.co/datasets/sarvamai/olmOCR-Bench-English)) | Overall↑ | arxiv_math↑ | baseline↑ | headers_footers↑ | long_tiny_text↑ | multi_column↑ | old_scans↑ | old_scans_math↑ | table_tests↑ |
+| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| Chandra OCR 2 | 85.9 | 86.7 | 99.8 | 91.5 | 93.7 | 84.7 | 51 | 88.2 | 92.2 |
+| Sarvam Vision | 84.3 | 86.5 | 99.6 | 96.3 | 91 | 82.2 | 49.8 | 81 | 88.3 |
+| Gemini 3.1 Pro | 82.6 | 90.5 | 99 | 82.9 | 88.5 | 81.6 | 47 | 84.3 | 87.3 |
+| **IndicOCR (ours)** | **82.2** | **83.2** | **99.4** | **92.9** | **89.8** | **76** | **48.3** | **77.7** | **90** |
+| Surya OCR 2 (model) | 81.4 | 82.5 | 99.8 | 92.9 | 79.9 | 85.1 | 42.8 | 84.3 | 84.2 |
+| Gemma 31B | 80.4 | 79 | 99.4 | 92.9 | 89.8 | 80.5 | 45.8 | 73.8 | 82.2 |
+| PaddleOCRVL-1.6 | 78.7 | 85.1 | 98.4 | 96.2 | 75.3 | 83.9 | 39 | 68.3 | 83 |
+| GPT | 78 | 79.3 | 93.9 | 95.4 | 87.8 | 77.4 | 43.7 | 64.6 | 82.2 |
+| Nemotron Parse 2 | 68.2 | 64 | 96.7 | 90 | 79.6 | 72.8 | 31.9 | 28.6 | 81.8 |
+
+<h3 style="color:#F97316;">IndicOCR-PR: printed accuracy by language (higher is better)</h3>
+
+Word-level accuracy, reported as 100 x (1 - WER).
+
+| Language | Sarvam Vision | **IndicOCR (ours)** | Gemini 3.1 Pro | SuryaOCR | Gemma 31B | Chandra OCR 2 |
+| --- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Overall** | 86.6 | **86.2** | 80.4 | 67.9 | 66.3 | 64.2 |
+| Assamese | 89.5 | 90.2 | 90.7 | 86.4 | 70.6 | 73.5 |
+| Bodo | 91.0 | 86.5 | 91.0 | 55.6 | 68.1 | 46.6 |
+| Bengali | 91.6 | 91.4 | 92.5 | 81.1 | 83.9 | 79.2 |
+| Dogri | 85.8 | 81.7 | 83.7 | 60.5 | 64.4 | 55.8 |
+| English | 96.6 | 97.0 | 97.7 | 93.8 | 97.2 | 91.3 |
+| Gujarati | 91.6 | 91.7 | 92.8 | 79.6 | 81.6 | 73.0 |
+| Hindi | 95.7 | 96.0 | 96.3 | 90.3 | 93.7 | 89.3 |
+| Konkani | 93.6 | 93.7 | 93.5 | 90.5 | 76.9 | 85.5 |
+| Kannada | 88.8 | 88.0 | 89.8 | 75.7 | 68.3 | 69.6 |
+| Kashmiri | 43.3 | 52.2 | 38.1 | 23.4 | 19.9 | 17.6 |
+| Malayalam | 90.6 | 89.9 | 90.6 | 76.5 | 72.0 | 68.3 |
+| Manipuri | 81.9 | 83.8 | 0.8 | 0.1 | 0.1 | 0.0 |
+| Marathi | 93.9 | 93.5 | 94.5 | 84.3 | 89.1 | 83.1 |
+| Maithili | 86.7 | 83.0 | 86.7 | 67.6 | 76.3 | 66.1 |
+| Nepali | 92.5 | 91.5 | 93.7 | 87.6 | 87.2 | 82.1 |
+| Odia | 77.5 | 75.7 | 84.8 | 64.5 | 38.7 | 62.6 |
+| Punjabi | 92.2 | 93.2 | 93.5 | 86.3 | 75.1 | 84.1 |
+| Sanskrit | 82.0 | 76.2 | 83.7 | 57.8 | 60.8 | 55.8 |
+| Sindhi | 89.2 | 87.1 | 86.3 | 80.5 | 74.5 | 71.4 |
+| Santhali | 71.9 | 74.7 | 0.2 | 0.1 | 0.2 | 0.0 |
+| Tamil | 94.2 | 91.3 | 94.4 | 79.9 | 83.3 | 79.0 |
+| Telugu | 84.3 | 82.3 | 85.5 | 63.1 | 66.6 | 59.6 |
+| Urdu | 87.1 | 85.9 | 88.0 | 76.4 | 76.6 | 74.4 |
+
+<h3 style="color:#F97316;">IndicOCR-HW: handwriting accuracy by language (higher is better)</h3>
+
+Word-level accuracy, reported as 100 x (1 - WER).
+
+| Language | Gemini 3.1 Pro | **IndicOCR (ours)** | Sarvam Vision | Gemma 31B | Chandra OCR 2 | SuryaOCR |
+| --- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Overall** | 72.0 | **66.7** | 55.4 | 33.9 | 24.7 | 23.0 |
+| Assamese | 71.6 | 66.1 | 47.8 | 24.1 | 8.9 | 17.8 |
+| Bengali | 74.8 | 71.3 | 58.3 | 35.1 | 6.6 | 10.0 |
+| English | 84.4 | 80.7 | 77.7 | 78.5 | 78.2 | 72.7 |
+| Gujarati | 60.0 | 55.9 | 39.2 | 23.7 | 11.8 | 11.5 |
+| Hindi | 83.1 | 77.6 | 72.3 | 70.7 | 54.6 | 42.7 |
+| Kannada | 73.8 | 69.6 | 57.7 | 17.2 | 11.5 | 13.2 |
+| Malayalam | 63.9 | 60.5 | 45.6 | 16.0 | 15.7 | 11.8 |
+| Marathi | 79.0 | 70.2 | 61.8 | 56.5 | 35.4 | 28.8 |
+| Odia | 66.7 | 68.2 | 40.6 | 15.5 | 19.4 | 19.9 |
+| Punjabi | 70.1 | 69.4 | 54.8 | 11.7 | 11.5 | 15.7 |
+| Tamil | 80.5 | 76.8 | 60.5 | 33.4 | 18.8 | 16.8 |
+| Telugu | 72.0 | 53.5 | 59.1 | 32.0 | 20.8 | 14.6 |
+| Urdu | 54.4 | 46.4 | 44.4 | 25.6 | 27.6 | 22.6 |
+
+---
+
+<h2 id="limitations" style="color:#F97316;">Limitations</h2>
+
+Reading order remains a challenge for **complex, multi-column layouts**. Handwriting recognition
+is also still being improved, particularly across different writing styles and writing
+characteristics.
+
+We are also extending handwriting support to additional Indic languages.
+
+
+---
+
+<h2 id="hardware" style="color:#F97316;">Hardware</h2>
+
+Latency and throughput numbers to follow.
+
+---
+
+<h2 id="license" style="color:#F97316;">License</h2>
+
+
+Released under [Bodhan Open License 1.0]().
+
+The release incorporates components distributed under Apache 2.0, including PP-DocLayoutV3,
+Qwen3.5, and the Sarvam-30B tokenizer. See the repository license and the corresponding upstream
+licenses for the applicable terms and attribution requirements.
+---
+
+<h2 id="citation" style="color:#F97316;">Citation</h2>
+
+```bibtex
+@misc{indicdocparser2026,
+  title  = {IndicDocParser: Multilingual Document Parsing for English and 22 Indian Languages},
+  author = {Bodhan.AI},
+  year   = {2026},
+  url    = {https://huggingface.co/bodhan-ai/indic-doc-parser}
+}
+```

TROUBLESHOOTING.md

@@ -0,0 +1,41 @@
+# IndicDocParser: troubleshooting
+
+Install and runtime problems, and what to do about them. The model card is
+[README.md](README.md).
+
+Nearly every install problem is the torch/torchvision/CUDA triangle.
+
+**Installing without the script**, on Colab, Windows, or anywhere a shell script is awkward. These
+are the two commands `install.sh` runs:
+
+```bash
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124   # your CUDA line
+pip install -r requirements.txt
+```
+
+Keep `torch` and `torchvision` in that one command. `torchvision` pins an exact `torch`, so
+installing them separately lets pip pick a mismatched pair. A plain `pip install torch` also takes
+the newest wheel, currently CUDA 13, which on an older driver initialises to CPU with nothing but a
+warning. Use the index for *your* CUDA: `cu121`, `cu124`, `cu128`, `cu130`, or `cpu`.
+
+**`ImportError` after `ImportError`, or complaints that torchvision is too old.** `torchvision`
+pins an exact `torch`, so if the two were installed separately pip may have paired them wrongly.
+Reinstall both in one command, from one index:
+
+```bash
+pip install --force-reinstall torch torchvision --index-url https://download.pytorch.org/whl/cu124
+```
+
+**`CUDA initialization: The NVIDIA driver on your system is too old`, then everything runs on CPU.**
+Plain `pip install torch` fetched a wheel built for a newer CUDA than your driver. Check with
+`nvidia-smi`, then reinstall from the matching index. Verify with:
+
+```python
+import torch; print(torch.__version__, torch.cuda.is_available())
+```
+
+**`Qwen3VLVideoProcessor requires the Torchvision library`.** The recognizer needs `torchvision`;
+the layout stage does not. Install it *together with* torch, from the same index.
+
+**`RTDetrHungarianMatcher requires the scipy library`.** You are on an old revision of this repo.
+Update to the current revision, which no longer builds the matcher during inference.

idp_blocks.py

@@ -0,0 +1,139 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/engine/blocks.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""Layout cleanup: geometry, duplicate suppression, nested-equation resolution.
+
+Pure geometry on Blocks -- no PIL, no torch, no page image -- so the rules that decide what gets
+transcribed stay testable with none of the GPU stack installed.
+"""
+
+from __future__ import annotations
+
+from idp_types import Block, DedupConfig
+from idp_contract import HEAD_FOOT, MARGINALIA, is_transcribed
+
+TEXTLIKE = frozenset({"Text", "Title", "SectionHeader", "Caption", "Footnote"})
+
+
+def area(bbox) -> float:
+    return max(0.0, bbox[2] - bbox[0]) * max(0.0, bbox[3] - bbox[1])
+
+
+def contained_frac(small, big) -> float:
+    """Fraction of ``small``'s area lying inside ``big``."""
+    ix0, iy0 = max(small[0], big[0]), max(small[1], big[1])
+    ix1, iy1 = min(small[2], big[2]), min(small[3], big[3])
+    inter = max(0.0, ix1 - ix0) * max(0.0, iy1 - iy0)
+    a = area(small)
+    return inter / a if a > 0 else 0.0
+
+
+def clamp_to_page(bbox, width: int, height: int) -> list[float]:
+    return [
+        max(0.0, bbox[0]),
+        max(0.0, bbox[1]),
+        min(float(width), bbox[2]),
+        min(float(height), bbox[3]),
+    ]
+
+
+def clean_layout(blocks: list[Block], cfg: DedupConfig | None = None) -> list[Block]:
+    """Drop duplicate and spurious boxes. Three rules; survivors keep input order.
+
+    1. Nested duplicates -- a box ``cfg.contain`` inside a larger box of the same group goes.
+    2. One header, one footer -- only the largest of each survives.
+    3. Empty frames -- a header/footer wrapping nothing is dropped.
+    """
+    cfg = cfg or DedupConfig()
+    n = len(blocks)
+    box = lambda i: blocks[i].bbox_xyxy  # noqa: E731
+    drop: set[int] = set()
+
+    # Header/Footer are in neither group: rules 2 and 3 govern them entirely.
+    groups = (
+        lambda label: label not in MARGINALIA,
+        lambda label: label in MARGINALIA and label not in HEAD_FOOT,
+    )
+    for in_group in groups:
+        idxs = sorted(
+            (i for i in range(n) if in_group(blocks[i].label)),
+            key=lambda i: area(box(i)),
+            reverse=True,
+        )
+        kept: list[int] = []
+        for i in idxs:
+            has_text = is_transcribed(blocks[i].label)
+            # A block is never absorbed into a container that will not itself be transcribed:
+            # otherwise a caption inside a Diagram is dropped for a container that is then
+            # skipped, losing the text. One-directional -- a non-transcribed box may go anywhere.
+            if any(
+                contained_frac(box(i), box(j)) >= cfg.contain
+                and (is_transcribed(blocks[j].label) or not has_text)
+                for j in kept
+            ):
+                drop.add(i)
+            else:
+                kept.append(i)
+
+    for label in HEAD_FOOT:
+        group = [i for i in range(n) if blocks[i].label == label and i not in drop]
+        if not group:
+            continue
+        biggest = max(group, key=lambda i: area(box(i)))
+        drop.update(i for i in group if i != biggest)
+        # Measured over range(n), NOT the survivors: a header whose only occupant was already
+        # removed as a nested duplicate is still occupied, and deleting it loses a real header.
+        wraps = any(
+            blocks[k].label != label and contained_frac(box(k), box(biggest)) >= cfg.wrap
+            for k in range(n)
+        )
+        if not wraps:
+            drop.add(biggest)
+
+    return [blocks[i] for i in range(n) if i not in drop]
+
+
+def _nested_in(inner, outer, thresh: float) -> bool:
+    # The strictness check stops two boxes over the same region each claiming to contain the
+    # other, which would drop both.
+    inner_area = area(inner)
+    if inner_area <= 0:
+        return False
+    return contained_frac(inner, outer) >= thresh and inner_area < 0.95 * area(outer)
+
+
+def _absorbs_equation(container: Block, mode: str) -> bool:
+    if container.type == "Equation":
+        # A display array is one outer box plus a box per row; collapsing keeps it one block.
+        return mode != "text_only"
+    if container.type in TEXTLIKE:
+        # The text prompt already renders inline math as $...$, so a separate equation box
+        # would emit it a second time as display math.
+        return mode != "eq_only"
+    return False
+
+
+def resolve_nested_equations(blocks: list[Block], cfg: DedupConfig | None = None) -> list[Block]:
+    """Drop equation boxes nested in a container that already covers them.
+
+    Runs before cropping -- it needs only boxes and types, so resolving first avoids building
+    crops that are immediately discarded.
+    """
+    cfg = cfg or DedupConfig()
+    if not cfg.nest:
+        return list(blocks)
+
+    drop: set[int] = set()
+    for i, inner in enumerate(blocks):
+        if inner.type != "Equation":
+            continue
+        for j, container in enumerate(blocks):
+            if (
+                i != j
+                and _nested_in(inner.bbox_xyxy, container.bbox_xyxy, cfg.nested)
+                and _absorbs_equation(container, cfg.mode)
+            ):
+                drop.add(i)
+                break
+    return [b for k, b in enumerate(blocks) if k not in drop]

idp_contract.py

@@ -0,0 +1,128 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/templates/contract.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""IndicDocParser's contract: prompts, block taxonomy, output schema.
+
+Two vocabularies meet here and mixing them fails silently. **Labels** are what IndicDocLayout
+emits (``ocr.layout.labels.CLASSES``); MARGINALIA and HEAD_FOOT match them case-SENSITIVELY.
+**Types** are the coarse categories a label maps to, and select the prompt. OCR_SKIP_LABELS is
+the exception: matched case-insensitively.
+
+stdlib-only -- importable with no GPU stack and no PIL.
+"""
+
+from __future__ import annotations
+
+from enum import StrEnum
+
+
+class TableFormat(StrEnum):
+    """HTML is the default: colspan/rowspan and in-cell breaks have no GFM spelling, so a
+    merged-cell table rendered as markdown silently loses its structure."""
+
+    HTML = "html"
+    MARKDOWN = "markdown"
+
+
+TEXT_PROMPT = (
+    "Transcribe the text in this image. Write any mathematical expressions in LaTeX, "
+    "using $...$ for inline math and $$...$$ for display equations."
+)
+EQUATION_PROMPT = "Output only the LaTeX for this equation image."
+TABLE_PROMPTS = {
+    TableFormat.HTML: (
+        "Convert this table image to HTML. Preserve the structure exactly, using colspan and "
+        "rowspan for merged cells and <br/> for line breaks within a cell. "
+        "Output only the HTML table."
+    ),
+    TableFormat.MARKDOWN: (
+        "Convert this table image to a GitHub-flavored markdown table. Output only the table."
+    ),
+}
+
+
+def prompt_for(block_type: str, table_format: TableFormat = TableFormat.HTML) -> str:
+    if block_type == "Table":
+        return TABLE_PROMPTS[TableFormat(table_format)]
+    if block_type == "Equation":
+        return EQUATION_PROMPT
+    return TEXT_PROMPT
+
+
+# Labels absent from this map fall through to "Text" by design -- Question, Paragraph, Answer,
+# List, MCQ, Code, Reference and the rest all carry prose.
+LABEL_TO_TYPE = {
+    "table": "Table",
+    "table-caption": "Caption",
+    "equation": "Equation",
+    "expression": "Equation",
+    "diagram": "Figure",
+    "chart": "Figure",
+    "image": "Picture",
+    "image-caption": "Caption",
+    "title": "Title",
+    "chapter-title": "Title",
+    "section-title": "SectionHeader",
+    "sub-section-title": "SectionHeader",
+    "sub-sub-section-title": "SectionHeader",
+    "header": "PageHeader",
+    "footer": "PageFooter",
+    "page-number": "PageNumber",
+    "folio": "PageNumber",
+    "footnote": "Footnote",
+}
+
+
+def map_label(label) -> str:
+    """Label -> pipeline type. Unknown labels are ``Text``."""
+    return LABEL_TO_TYPE.get(str(label).strip().lower(), "Text")
+
+
+# Exactly the types map_label can produce, less DROP_TYPES. test_contract asserts this, so a
+# dead or undocumented type cannot creep in.
+KEPT_BLOCK_TYPES = (
+    "Text",
+    "Title",
+    "SectionHeader",
+    "Table",
+    "Equation",
+    "Caption",
+    "Footnote",
+    "PageHeader",
+    "PageFooter",
+    "PageNumber",
+)
+
+# Never cropped, never reconstructed: pictorial regions have no text and invite hallucination.
+# Deliberately narrow -- page numbers and margin text DO reach the recognizer.
+DROP_TYPES = frozenset({"Figure", "Picture"})
+
+# Cleaned as their own group so a page-spanning paragraph cannot swallow a page number.
+MARGINALIA = frozenset({"Header", "Footer", "Page-number", "Folio"})
+HEAD_FOOT = ("Header", "Footer")
+
+# Never sent to the recognizer, but NOT deleted: these keep their place in the output with
+# text "", so a consumer can still see what was detected and where.
+OCR_SKIP_LABELS = frozenset({"header", "footer", "diagram", "image", "chart", "advertisement"})
+
+
+def is_transcribed(label) -> bool:
+    return str(label).strip().lower() not in OCR_SKIP_LABELS
+
+
+OUTPUT_BLOCK_SCHEMA = {
+    "order": "int    -- reading-order rank (0 = first)",
+    "label": "str    -- raw IndicDocLayout label",
+    "type": "str    -- pipeline type (see KEPT_BLOCK_TYPES)",
+    "bbox_xyxy": "[float x4] -- pixel [x0, y0, x1, y1], clamped to the page",
+    "conf": "float  -- detection confidence",
+    "text": "str    -- transcription; '' when not sent to the recognizer",
+}
+
+OUTPUT_PAGE_SCHEMA = {
+    "image": "str   -- source image filename",
+    "width": "int   -- page width in pixels",
+    "height": "int   -- page height in pixels",
+    "blocks": "[BLOCK] -- in reading order",
+}

idp_crops.py

@@ -0,0 +1,65 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/engine/crops.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""Turning boxes into the images the recognizer sees.
+
+Split from ``engine.blocks`` because this is the only part that needs PIL -- keeping the
+geometry PIL-free is what lets the cleanup rules be tested without an image library.
+"""
+
+from __future__ import annotations
+
+import math
+from typing import TYPE_CHECKING
+
+from idp_types import Block, CropConfig
+from idp_contract import DROP_TYPES, is_transcribed
+
+if TYPE_CHECKING:  # pragma: no cover
+    from PIL.Image import Image
+
+
+def area_clamp(image: Image, cfg: CropConfig | None = None) -> Image:
+    """Scale a crop so its area lands in ``[cfg.min_px, cfg.max_px]``, preserving aspect.
+
+    Area rather than a side: pinning a side exploded elongated crops (a 122:1 rule line became
+    ~32k image tokens and wedged the engine).
+    """
+    from PIL import Image as PILImage
+
+    cfg = cfg or CropConfig()
+    width, height = image.size
+    pixels = width * height
+    if pixels <= 0:
+        return image
+
+    if pixels < cfg.min_px:
+        scale = math.sqrt(cfg.min_px / pixels)
+    elif pixels > cfg.max_px:
+        scale = math.sqrt(cfg.max_px / pixels)
+    else:
+        return image
+
+    return image.resize(
+        (max(1, round(width * scale)), max(1, round(height * scale))), PILImage.LANCZOS
+    )
+
+
+def crop_for(block: Block, page: Image, cfg: CropConfig | None = None) -> Image | None:
+    """The image for one block, or None if it should not be transcribed."""
+    if block.type in DROP_TYPES or not is_transcribed(block.label):
+        return None
+
+    cfg = cfg or CropConfig()
+    width, height = page.size
+    x0, y0, x1, y1 = (round(v) for v in block.bbox_xyxy)
+    if cfg.pad_px:  # recover glyph edges a tight box clips
+        x0, y0 = x0 - cfg.pad_px, y0 - cfg.pad_px
+        x1, y1 = x1 + cfg.pad_px, y1 + cfg.pad_px
+    x0, y0 = max(0, x0), max(0, y0)
+    x1, y1 = min(width, x1), min(height, y1)
+
+    if x1 <= x0 or y1 <= y0:  # rounding can collapse a thin rule to zero width
+        return None
+    return page.crop((x0, y0, x1, y1)).convert("RGB")

idp_layout.py

@@ -0,0 +1,124 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/engine/layout.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""Layout backends: page image -> cleaned, reading-ordered blocks.
+
+The two stages hand off a plain JSON layout, so stage 2 does not care where the layout came
+from. :class:`LayoutBackend` makes that an interface rather than a claim.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING, Protocol, runtime_checkable
+
+from idp_blocks import clamp_to_page, clean_layout
+from idp_types import Block, DedupConfig, LayoutConfig, PageResult
+from idp_contract import map_label
+
+if TYPE_CHECKING:  # pragma: no cover
+    from PIL.Image import Image
+
+
+@runtime_checkable
+class LayoutBackend(Protocol):
+    """``detect`` must return blocks already cleaned and densely ordered: ``order`` a gap-free
+    0-based rank. Stage 2 matches transcriptions back by ``order``, so gaps mis-assign text."""
+
+    def detect(self, image: Image) -> list[Block]: ...
+
+    def close(self) -> None: ...
+
+
+def _densify(blocks: list[Block]) -> list[Block]:
+    """Sort by the detector's reading order and renumber to a gap-free 0-based rank."""
+    ordered = sorted(blocks, key=lambda b: b.order)
+    for rank, block in enumerate(ordered):
+        block.order = rank
+    return ordered
+
+
+class IndicDocLayoutBackend:
+    """Our finetuned PP-DocLayoutV3 with an integrated reading-order head. Torch only --
+    constructing this does not load vLLM, which is what lets stage 1 run alone."""
+
+    def __init__(
+        self,
+        ckpt: str | None = None,
+        config: LayoutConfig | None = None,
+        dedup: DedupConfig | None = None,
+    ) -> None:
+        from idp_model_infer import get_model
+
+        self.config = config or LayoutConfig()
+        self.dedup = dedup or DedupConfig()
+        if ckpt is None:
+            raise ValueError(
+                "no layout weights given -- "
+                "IndicDocParser.from_pretrained(snapshot_download(REPO))"
+            )
+        self.ckpt = ckpt
+        self.model = get_model(self.ckpt, device=self.config.device)
+
+    def detect(self, image: Image) -> list[Block]:
+        from idp_model_infer import infer
+
+        width, height = image.size
+        detections = infer(
+            self.model,
+            image,
+            conf=self.config.conf,
+            img_size=self.config.img_size,
+            device=self.config.device,
+        )
+
+        # The model emits [y0, x0, y1, x1] normalised to 0-1000; the pipeline works in pixel
+        # [x0, y0, x1, y1]. Axis swap and rescale happen here, once.
+        blocks = []
+        for det in detections:
+            y0, x0, y1, x1 = det["bbox"]
+            bbox = [x0 / 1000 * width, y0 / 1000 * height, x1 / 1000 * width, y1 / 1000 * height]
+            label = str(det["label"])
+            blocks.append(
+                Block(
+                    order=det["reading_order"],
+                    label=label,
+                    type=map_label(label),
+                    bbox_xyxy=[round(v, 1) for v in clamp_to_page(bbox, width, height)],
+                    conf=round(float(det.get("score", 1.0)), 3),
+                )
+            )
+
+        return _densify(clean_layout(blocks, self.dedup))
+
+    def close(self) -> None:
+        self.model = None
+
+
+class JsonLayoutBackend:
+    """Replay a layout produced elsewhere -- by stage 1, by hand, or by another detector.
+
+    Assumed already clean, so no cleanup runs; blocks are only renumbered, which makes a
+    hand-edited file usable without fixing ranks. Needs no torch.
+    """
+
+    def __init__(self, layout: str | dict | PageResult, *, strict: bool = True) -> None:
+        if isinstance(layout, PageResult):
+            self.page = layout
+        else:
+            if isinstance(layout, str):
+                with open(layout, encoding="utf-8") as fh:
+                    layout = json.load(fh)
+            # Validates by default: this is the door a foreign layout comes through, and an
+            # unrecognised label would otherwise become Text without a word. `strict=False`
+            # replays a file written before validation existed.
+            self.page = PageResult.from_record(layout, strict=strict)
+
+    def detect(self, image: Image) -> list[Block]:
+        """Copies, so renumbering cannot write back into the stored layout. ``image`` is
+        accepted for interface parity and not read."""
+        return _densify([b.copy() for b in self.page.blocks])
+
+    def close(self) -> None:
+        return None

idp_model_infer.py

@@ -0,0 +1,93 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/layout/infer.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""Single-image inference for IndicDocLayout -> boxes, labels and reading order.
+
+Emits ``[{bbox: [y0, x0, y1, x1] normalised to 0-1000, label, reading_order, score}]``. That is
+the layout viewer's schema, kept verbatim so predictions render with the same box and
+reading-order visualisation as the dataset viewer; ``engine.layout`` converts it to pixel
+``[x0, y0, x1, y1]``.
+
+The decode mirrors the training-time evaluation exactly: sigmoid-max detections, and reading
+order from a voting sort over the pairwise ``order_logits`` restricted to the kept boxes.
+"""
+
+import numpy as np
+import torch
+
+from idp_model_labels import ID2LABEL
+from idp_model_order_loss import decode_order, pairwise_scores
+
+_CACHE = {}
+
+
+def cxcywh_to_xyxy(b):
+    """Centre-form boxes -> corner form, preserving the input tensor's dtype and device."""
+    c = b.clone()
+    c[..., 0], c[..., 1] = b[..., 0] - b[..., 2] / 2, b[..., 1] - b[..., 3] / 2
+    c[..., 2], c[..., 3] = b[..., 0] + b[..., 2] / 2, b[..., 1] + b[..., 3] / 2
+    return c
+
+
+def get_model(ckpt, device="cuda"):
+    """Load (and cache) an IndicDocLayout checkpoint.
+
+    The checkpoint is saved as ``PPDocLayoutV3Trainable`` -- our subclass -- so that class has to
+    be importable here even though nothing is trained at inference time.
+    """
+    key = (ckpt, device)
+    if key not in _CACHE:
+        from idp_model_ppdoc import PPDocLayoutV3Trainable
+
+        _CACHE[key] = PPDocLayoutV3Trainable.from_pretrained(ckpt).to(device).eval()
+    return _CACHE[key]
+
+
+@torch.no_grad()
+def infer(model, pil_img, conf=0.5, img_size=1024, device="cuda"):
+    """Detect blocks on one page image."""
+    im = pil_img.convert("RGB").resize((img_size, img_size))
+    # np.array (not asarray): a PIL buffer is read-only, and torch warns on every page about
+    # wrapping a non-writable array.
+    x = torch.from_numpy(np.array(im)).permute(2, 0, 1).float().div(255.0)  # [3,S,S] in [0,1]
+    out = model(pixel_values=x[None].to(device))
+    scores, labels = out.logits.sigmoid().max(-1)  # [1,N]
+    boxes = cxcywh_to_xyxy(out.pred_boxes)[0]  # [N,4] normalised x0,y0,x1,y1
+
+    if getattr(out, "order_logits", None) is not None:
+        order_scores = out.order_logits[0]
+    else:
+        nq = model.config.num_queries
+        order_scores = pairwise_scores(out.last_hidden_state[:, -nq:], model.ro_q, model.ro_k)[0]
+
+    keep = (scores[0] > conf).nonzero().squeeze(-1)
+    if keep.numel() == 0:
+        return []
+    kept_boxes, kept_scores, kept_labels = boxes[keep], scores[0][keep], labels[0][keep]
+
+    # Reading order is decoded over the KEPT sub-block only: ranking against suppressed queries
+    # would leave gaps in the sequence.
+    sub = order_scores[keep][:, keep].cpu().float()
+    sequence = decode_order(sub).tolist()  # positions, first -> last
+    rank = [0] * len(sequence)
+    for r, position in enumerate(sequence):
+        rank[position] = r + 1
+
+    content = []
+    for j in range(keep.numel()):
+        x0, y0, x1, y1 = kept_boxes[j].tolist()
+        content.append(
+            {
+                "bbox": [  # viewer schema: [y0, x0, y1, x1], 0-1000
+                    round(y0 * 1000, 1),
+                    round(x0 * 1000, 1),
+                    round(y1 * 1000, 1),
+                    round(x1 * 1000, 1),
+                ],
+                "label": ID2LABEL[int(kept_labels[j])],
+                "reading_order": rank[j],
+                "score": round(float(kept_scores[j]), 3),
+            }
+        )
+    return content

idp_model_labels.py

@@ -0,0 +1,159 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/layout/labels.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""IndicDocLayout's 37 education-domain layout classes = SUPERSET of the printed + handwritten
+taxonomies.
+Order is fixed; do not reorder and only APPEND (ids are baked into checkpoints). ids 0-24 are the
+original 25; 25-26 are the printed-only additions (Chapter-title, Chapter-end-section); 27-36 are
+the indicdlp-v2 magazine/newspaper additions. Class weights are sqrt-inverse block frequency
+(capped [0.5, 5]); optional, used only if enabled.
+
+These are the raw *labels*. The map onto the coarse pipeline *types* that select prompts lives in
+``ocr.templates.contract`` -- see its module docstring for why the two vocabularies differ.
+"""
+
+CLASSES = [
+    "Question",
+    "Paragraph",
+    "Answer",
+    "List",
+    "Title",
+    "Section-title",
+    "Equation",
+    "Table",
+    "Diagram",
+    "Image",
+    "MCQ",
+    "Infobox",
+    "Sub-section-title",
+    "Expression",
+    "Image-caption",
+    "Placeholder-text",
+    "Chart",
+    "Solved-example",
+    "Footnote",
+    "Table-caption",
+    "Sub-sub-section-title",
+    "Footer",
+    "Header",
+    "Code",
+    "Page-number",
+    "Chapter-title",
+    "Chapter-end-section",  # 25-26: printed-only additions (superset)
+    # 27-36: indicdlp-v2 additions (magazine/newspaper furniture kept as their own classes;
+    # only populated by the indicdlp-printed source -> no cross-dataset partial-annotation conflict).
+    "Folio",
+    "Reference",
+    "Table-of-contents",
+    "Index",
+    "Advertisement",
+    "Author",
+    "Dateline",
+    "Contact-info",
+    "Website-link",
+    "Flag",
+]
+
+LABEL2ID = {c: i for i, c in enumerate(CLASSES)}
+ID2LABEL = {i: c for i, c in enumerate(CLASSES)}
+NUM_CLASSES = len(CLASSES)
+
+# sqrt-inverse-frequency weights (measured on 155,922 blocks); rare classes upweighted.
+CLASS_WEIGHTS = {
+    "Question": 0.50,
+    "Paragraph": 0.50,
+    "Equation": 0.50,
+    "Answer": 0.50,
+    "List": 0.50,
+    "Section-title": 0.50,
+    "Title": 0.50,
+    "Image": 0.50,
+    "Table": 0.52,
+    "Diagram": 0.53,
+    "MCQ": 0.74,
+    "Expression": 1.00,
+    "Image-caption": 1.11,
+    "Infobox": 1.43,
+    "Solved-example": 1.43,
+    "Placeholder-text": 1.48,
+    "Sub-section-title": 1.64,
+    "Chart": 2.11,
+    "Footnote": 4.63,
+    "Table-caption": 5.0,
+    "Sub-sub-section-title": 5.0,
+    "Footer": 5.0,
+    "Header": 5.0,
+    "Code": 5.0,
+    "Page-number": 5.0,
+    "Chapter-title": 0.74,
+    "Chapter-end-section": 4.5,  # by frequency in label_frequency.csv
+}
+
+# --- Shared label parser (used by BOTH the training blob cache AND the val/test disk path,
+#     so train/val/test are always consistent). Content blocks + metadata Header/Footer region
+#     boxes (header reads first, footer last; placeholder stubs dropped; hw dedup vs content). ---
+MIN_HF_AREA = 1e-4  # drop placeholder header/footer stubs (e.g. bbox [0,0,5,10])
+
+
+def _to_cxcywh(bb):
+    """[y0,x0,y1,x1] in 0-1000 -> clamped [cx,cy,w,h] normalized, or None if degenerate."""
+    try:
+        y0, x0, y1, x1 = [float(c) / 1000 for c in bb]
+    except (TypeError, ValueError):
+        return None  # malformed bbox (e.g. nested list) -> skip this box
+    if x1 - x0 <= 1e-3 or y1 - y0 <= 1e-3:
+        return None
+    cx, cy, w, h = (x0 + x1) / 2, (y0 + y1) / 2, x1 - x0, y1 - y0
+    cx, cy = min(max(cx, 1e-4), 1 - 1e-4), min(max(cy, 1e-4), 1 - 1e-4)
+    w, h = min(w, 2 * min(cx, 1 - cx)), min(h, 2 * min(cy, 1 - cy))
+    return [cx, cy, w, h]
+
+
+def _iou(a, b):
+    iw = max(0, min(a[0] + a[2] / 2, b[0] + b[2] / 2) - max(a[0] - a[2] / 2, b[0] - b[2] / 2))
+    ih = max(0, min(a[1] + a[3] / 2, b[1] + b[3] / 2) - max(a[1] - a[3] / 2, b[1] - b[3] / 2))
+    inter = iw * ih
+    ua = a[2] * a[3] + b[2] * b[3] - inter
+    return inter / ua if ua > 0 else 0.0
+
+
+def labels_from_doc(d, drop_hf_strips=False):
+    """Parse a page JSON dict -> (boxes[cxcywh], cls[ids], order). THE canonical parser.
+    drop_hf_strips: skip metadata Header/Footer boxes that are full-width page-edge STRIPS (the
+    hw-vs/hw-dps placeholders like [0,0,100,1000]) — they're not real text regions and teach the
+    model to hallucinate top/bottom bars. Real (non-strip) header/footer regions are kept."""
+    cls, boxes, order = [], [], []
+    for b in d.get("content", []):
+        bb, lab = b.get("bbox"), b.get("label")
+        if not (isinstance(bb, list) and len(bb) == 4) or lab not in LABEL2ID:
+            continue
+        box = _to_cxcywh(bb)
+        if box is None:
+            continue
+        cls.append(LABEL2ID[lab])
+        boxes.append(box)
+        order.append(b.get("reading_order", len(order) + 1))
+    meta = d.get("metadata", {}) or {}
+    ords = order or [0]
+    for field, lab, ro in (
+        ("header", "Header", min(ords) - 1),
+        ("footer", "Footer", max(ords) + 1),
+    ):
+        bb = (meta.get(field) or {}).get("bbox")
+        if not (isinstance(bb, list) and len(bb) == 4):
+            continue
+        box = _to_cxcywh(bb)
+        if box is None or box[2] * box[3] < MIN_HF_AREA:
+            continue
+        if drop_hf_strips and box[2] > 0.9:  # full-width strip flush to a page edge = placeholder
+            y0, y1 = box[1] - box[3] / 2, box[1] + box[3] / 2
+            if (field == "header" and y0 < 0.02) or (field == "footer" and y1 > 0.98):
+                continue
+        lid = LABEL2ID[lab]
+        if any(cls[i] == lid and _iou(boxes[i], box) > 0.5 for i in range(len(boxes))):
+            continue
+        cls.append(lid)
+        boxes.append(box)
+        order.append(ro)
+    return boxes, cls, order

idp_model_order_loss.py

@@ -0,0 +1,47 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/layout/order_loss.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""Locality-weighted Generalized Cross-Entropy on the antisymmetric pairwise
+precedence scores. Validated in learnings (recovers arbitrary order, tau=1.0).
+"""
+
+import math
+
+import torch
+
+
+def decode_order(S):
+    """Reading order from a pairwise score matrix, matching PP-DocLayoutV3's _get_order_seqs.
+    P(i before j) = sigmoid(S[i,j]) if i<j else 1-sigmoid(S[j,i]); votes[j]=#elements before j;
+    argsort ascending -> reading sequence (first..last). Correct for BOTH the triangular ppdoc
+    GlobalPointer AND an antisymmetric head (they coincide under this formula)."""
+    sc = torch.sigmoid(S)
+    votes = sc.triu(1).sum(0) + (1.0 - sc.t()).tril(-1).sum(0)
+    return torch.argsort(votes)
+
+
+def pairwise_scores(Q, ro_q, ro_k):
+    """Q:[B,N,d] -> antisymmetric S:[B,N,N], S_ij>0 => query i precedes j."""
+    A, K = ro_q(Q), ro_k(Q)  # [B,N,r]
+    M = A @ K.transpose(-1, -2)  # M_ij = q_i^T Wq^T Wk q_j
+    return (M - M.transpose(-1, -2)) / math.sqrt(A.shape[-1])
+
+
+def locality_gce(S, order, q=0.7, tau=3.0, eps=1e-6):
+    """S:[m,m] scores over matched queries, order:[m] their GT reading_order values."""
+    m = order.numel()
+    if m < 2:
+        return S.sum() * 0.0  # nothing to order; keep graph
+    P = (order.unsqueeze(1) < order.unsqueeze(0)).float()  # P_ab=1 if a precedes b
+    p = torch.sigmoid(S)
+    pc = torch.where(P > 0.5, p, 1 - p).clamp(eps, 1)
+    gce = (1 - pc.pow(q)) / q
+    pos = order.argsort().argsort().float()  # dense rank positions
+    dist = (pos.unsqueeze(1) - pos.unsqueeze(0)).abs()
+    W = torch.exp(-dist / tau)
+    # only the strict UPPER triangle carries real scores (ppdoc GlobalPointer masks a>=b to -1e4);
+    # training the masked lower triangle penalizes unfixable entries. Upper-tri also suffices for
+    # an antisymmetric head. P[i,j] for i<j = (order_i < order_j).
+    mask = torch.triu(torch.ones(m, m, dtype=torch.bool, device=S.device), diagonal=1)
+    return (gce * W)[mask].sum() / W[mask].sum().clamp_min(eps)

idp_model_ppdoc.py

@@ -0,0 +1,129 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/layout/modeling_ppdoc.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""IndicDocLayout: trainable PP-DocLayoutV3 (document-pretrained strong init).
+
+HF ships PPDocLayoutV3ForObjectDetection inference-only (forward raises on labels).
+This subclass unblocks training: it calls the inner model with labels (which builds the
+contrastive-denoising groups), reuses the base RT-DETR detection loss on its outputs, and
+adds our locality-weighted GCE order loss on its (pretrained) order_logits.
+Backbone + decoder + order/mask heads start from the document-pretrained checkpoint;
+only the class heads are re-init'd for our 37 education classes.
+"""
+
+from dataclasses import dataclass
+
+import torch
+import torch.nn as nn
+from transformers import PPDocLayoutV3Config, PPDocLayoutV3ForObjectDetection
+from transformers.loss.loss_rt_detr import RTDetrHungarianMatcher
+from transformers.utils import ModelOutput
+
+from idp_model_order_loss import locality_gce
+
+
+@dataclass
+class PPDocOutput(ModelOutput):
+    loss: torch.FloatTensor | None = None
+    logits: torch.FloatTensor | None = None
+    pred_boxes: torch.FloatTensor | None = None
+    order_logits: torch.FloatTensor | None = None
+    last_hidden_state: torch.FloatTensor | None = None
+
+
+class PPDocLayoutV3Trainable(PPDocLayoutV3ForObjectDetection):
+    def __init__(self, config):
+        super().__init__(config)
+        self.lambda_order = getattr(config, "lambda_order", 5.0)
+        # Built on first use, not here: the matcher is training-only, and constructing it calls
+        # requires_backends(["scipy"]). Eagerly, that makes scipy a hard dependency of merely
+        # LOADING the detector -- so inference-only installs fail on import with a library they
+        # will never call. It holds no parameters, so this does not change the state dict.
+        self._matcher = None
+        self.loss_type = "RTDetrForObjectDetection"  # base RT-DETR loss over its outputs
+
+    @classmethod
+    def build(cls, ckpt, num_labels, id2label, label2id, lambda_order=5.0):
+        config = PPDocLayoutV3Config.from_pretrained(
+            ckpt, num_labels=num_labels, id2label=id2label, label2id=label2id
+        )
+        config.lambda_order = lambda_order
+        config.loss_type = "RTDetrForObjectDetection"
+        # PP-DocLayoutV3's denoising path is buggy (embed size num_labels but pads with
+        # num_labels -> index error); it was never run since HF blocks training. Disable it
+        # (optional convergence aid). Re-enable later by resizing denoising_class_embed to +1.
+        config.num_denoising = 0
+        # RT-DETR loss/matcher fields the base config lacks
+        defaults = {
+            "use_focal_loss": True,
+            "auxiliary_loss": True,
+            "weight_loss_vfl": 1.0,
+            "weight_loss_bbox": 5.0,
+            "weight_loss_giou": 2.0,
+            "matcher_class_cost": 2.0,
+            "matcher_bbox_cost": 5.0,
+            "matcher_giou_cost": 2.0,
+            "matcher_alpha": 0.25,
+            "matcher_gamma": 2.0,
+            "focal_loss_alpha": 0.25,
+            "focal_loss_gamma": 2.0,
+            "eos_coefficient": 1e-4,
+        }
+        for k, v in defaults.items():
+            if not hasattr(config, k):
+                setattr(config, k, v)
+        model = cls.from_pretrained(ckpt, config=config, ignore_mismatched_sizes=True)
+        # re-init class heads (paddle doc classes -> our education classes); keep everything else
+        for m in model.modules():
+            if (isinstance(m, nn.Linear) and m.out_features == num_labels) or (
+                isinstance(m, nn.Embedding) and m.num_embeddings == num_labels + 1
+            ):
+                m.reset_parameters()
+        return model
+
+    def _order_loss(self, order_logits, logits, pred_boxes, labels):
+        if self._matcher is None:
+            self._matcher = RTDetrHungarianMatcher(self.config)  # needs scipy; training only
+        idx = self._matcher({"logits": logits, "pred_boxes": pred_boxes}, labels)
+        tot, n = 0.0, 0
+        for b, (src, tgt) in enumerate(idx):
+            if src.numel() < 2:
+                continue
+            order = labels[b]["reading_order"][tgt]
+            S = order_logits[b][src][:, src]
+            tot = tot + locality_gce(S, order)
+            n += 1
+        return tot / max(n, 1) if n else order_logits.sum() * 0.0
+
+    def forward(self, pixel_values, pixel_mask=None, labels=None, **kwargs):
+        outputs = self.model(pixel_values=pixel_values, pixel_mask=pixel_mask, labels=labels)
+        dn = outputs.denoising_meta_values if self.training else None
+        outputs_class = outputs.intermediate_logits
+        outputs_coord = outputs.intermediate_reference_points
+        logits, pred_boxes = outputs_class[:, -1], outputs_coord[:, -1]
+        order_logits = outputs.out_order_logits[:, -1]  # [B, num_queries, num_queries]
+        loss = None
+        if labels is not None:
+            loss, _, _ = self.loss_function(
+                logits,
+                labels,
+                self.device,
+                pred_boxes,
+                self.config,
+                outputs_class,
+                outputs_coord,
+                enc_topk_logits=outputs.enc_topk_logits,
+                enc_topk_bboxes=outputs.enc_topk_bboxes,
+                denoising_meta_values=dn,
+            )
+            loss = loss + self.lambda_order * self._order_loss(
+                order_logits, logits, pred_boxes, labels
+            )
+        return PPDocOutput(
+            loss=loss,
+            logits=logits,
+            pred_boxes=pred_boxes,
+            order_logits=order_logits,
+            last_hidden_state=outputs.last_hidden_state,
+        )

idp_offline.py

@@ -0,0 +1,177 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/engine/offline.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""The two stages, and the pipeline that runs both.
+
+    IndicDocLayout   page image   -> PageResult (blocks, no text)
+    IndicBlockOCR    image+layout -> PageResult (blocks with text, plus markdown)
+    IndicDocParser   page image   -> both
+
+Heavy imports live inside methods, so importing this module stays free.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+
+from idp_blocks import resolve_nested_equations
+from idp_reconstruct import reconstruct
+from idp_types import (
+    CropConfig,
+    DedupConfig,
+    LayoutConfig,
+    PageResult,
+    RecognizerConfig,
+)
+from idp_contract import is_transcribed
+
+
+def _open(image_path: str):
+    from PIL import Image
+
+    # Large scans and newspapers legitimately exceed PIL's decompression-bomb guard; layout
+    # resizes to img_size and crops are area-clamped, so compute stays bounded regardless.
+    Image.MAX_IMAGE_PIXELS = None
+    return Image.open(image_path).convert("RGB")
+
+
+def _as_page(layout: PageResult | dict | str) -> PageResult:
+    if isinstance(layout, PageResult):
+        return layout
+    if isinstance(layout, str):
+        with open(layout, encoding="utf-8") as fh:
+            layout = json.load(fh)
+    return PageResult.from_record(layout)
+
+
+class IndicDocLayout:
+    """Stage 1 -- layout and reading order. Loads torch only, never vLLM."""
+
+    def __init__(
+        self,
+        ckpt: str | None = None,
+        config: LayoutConfig | None = None,
+        dedup: DedupConfig | None = None,
+        backend=None,
+    ) -> None:
+        if backend is None:
+            from idp_layout import IndicDocLayoutBackend
+
+            backend = IndicDocLayoutBackend(ckpt, config, dedup)
+        self.backend = backend
+
+    def detect(self, image_path: str) -> PageResult:
+        image = _open(image_path)
+        return PageResult(
+            image=os.path.basename(image_path),
+            width=image.width,
+            height=image.height,
+            blocks=self.backend.detect(image),
+        )
+
+    def close(self) -> None:
+        self.backend.close()
+
+    def __enter__(self) -> IndicDocLayout:
+        return self
+
+    def __exit__(self, *exc_info) -> None:
+        self.close()
+
+
+class IndicBlockOCR:
+    """Stage 2 -- per-block transcription against a layout, which may be your own."""
+
+    def __init__(
+        self,
+        ckpt: str | None = None,
+        config: RecognizerConfig | None = None,
+        dedup: DedupConfig | None = None,
+        crop: CropConfig | None = None,
+        backend=None,
+    ) -> None:
+        self.config = config or RecognizerConfig()
+        self.dedup = dedup or DedupConfig()
+        self.crop = crop or CropConfig()
+        if backend is None:
+            from idp_recognizer import HfRecognizer
+
+            backend = HfRecognizer(ckpt, self.config)
+        self.backend = backend
+
+    def run(self, image_path: str, layout: PageResult | dict | str) -> PageResult:
+        """Every block of the layout comes back, in its original order. Blocks that were not
+        transcribed carry ``text: ""`` rather than being dropped."""
+        from idp_recognizer import build_requests
+
+        image = _open(image_path)
+        page = _as_page(layout)
+        blocks = [b.copy() for b in page.blocks]
+
+        eligible = resolve_nested_equations(
+            [b for b in blocks if is_transcribed(b.label)], self.dedup
+        )
+        requests, orders = build_requests(eligible, image, self.crop, self.config.table_format)
+        texts = self.backend.transcribe(requests)
+        if len(texts) != len(orders):
+            raise RuntimeError(
+                f"recognizer returned {len(texts)} transcriptions for {len(orders)} crops; "
+                "results would be misaligned"
+            )
+
+        by_order = dict(zip(orders, texts, strict=True))
+        for block in blocks:
+            block.text = (by_order.get(block.order) or "").strip()
+
+        return PageResult(
+            image=page.image,
+            width=page.width,
+            height=page.height,
+            blocks=blocks,
+            markdown=reconstruct(blocks),
+        )
+
+    def close(self) -> None:
+        self.backend.close()
+
+    def __enter__(self) -> IndicBlockOCR:
+        return self
+
+    def __exit__(self, *exc_info) -> None:
+        self.close()
+
+
+class IndicDocParser:
+    """Both stages in one process."""
+
+    def __init__(
+        self,
+        layout_ckpt: str | None = None,
+        recognizer_ckpt: str | None = None,
+        layout_config: LayoutConfig | None = None,
+        recognizer_config: RecognizerConfig | None = None,
+        dedup: DedupConfig | None = None,
+        crop: CropConfig | None = None,
+    ) -> None:
+        # vLLM FIRST. Its EngineCore forks/spawns at construction and must initialise CUDA
+        # before the torch layout model touches the device; reversed, the child cannot re-init.
+        self.ocr = IndicBlockOCR(recognizer_ckpt, recognizer_config, dedup, crop)
+        self.layout = IndicDocLayout(layout_ckpt, layout_config, dedup)
+
+    def detect(self, image_path: str) -> PageResult:
+        return self.layout.detect(image_path)
+
+    def parse(self, image_path: str) -> PageResult:
+        return self.ocr.run(image_path, self.layout.detect(image_path))
+
+    def close(self) -> None:
+        self.ocr.close()
+        self.layout.close()
+
+    def __enter__(self) -> IndicDocParser:
+        return self
+
+    def __exit__(self, *exc_info) -> None:
+        self.close()

idp_recognizer.py

@@ -0,0 +1,146 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/engine/recognizer.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""IndicBlockOCR: crops in, transcriptions out.
+
+Heavy imports live inside methods, so importing this module stays free -- asserted by
+tests/ocr/test_ocr_lazy_import.py.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, NamedTuple, Protocol, runtime_checkable
+
+from idp_types import CropConfig, RecognizerConfig
+
+if TYPE_CHECKING:  # pragma: no cover
+    from PIL.Image import Image
+
+
+class CropRequest(NamedTuple):
+    image: Image
+    prompt: str
+
+
+@runtime_checkable
+class RecognizerBackend(Protocol):
+    """``transcribe`` returns one string per request, in the same order."""
+
+    def transcribe(self, requests: list[CropRequest]) -> list[str]: ...
+
+    def close(self) -> None: ...
+
+
+def build_requests(blocks, page, crop_cfg: CropConfig, table_format) -> tuple[list, list]:
+    """Crop each block and pair it with its prompt.
+
+    Returns ``(requests, orders)`` -- the reading-order rank of each request, so transcriptions
+    can be matched back. Blocks that yield no crop are simply absent from both.
+    """
+    from idp_crops import area_clamp, crop_for
+    from idp_contract import prompt_for
+
+    requests, orders = [], []
+    for block in blocks:
+        crop = crop_for(block, page, crop_cfg)
+        if crop is None:
+            continue
+        requests.append(
+            CropRequest(area_clamp(crop, crop_cfg), prompt_for(block.type, table_format))
+        )
+        orders.append(block.order)
+    return requests, orders
+
+
+class HfRecognizer:
+    """Reference recognizer on plain ``transformers`` -- no vLLM.
+
+    Exists so IndicDocParser can run anywhere ``transformers`` runs, including straight from the
+    Hub with ``trust_remote_code=True``. It is the *quickstart* path, not the working one:
+    without continuous batching it is orders of magnitude slower per block than
+    :class:`VllmRecognizer`, so use it to try a page, not to parse a corpus.
+
+    Output also diverges slightly from the vLLM path. Both decode greedily, but different kernels
+    give different logits, and a near-tie flips the argmax -- so do not expect byte-identical
+    transcriptions between the two backends.
+    """
+
+    def __init__(
+        self,
+        ckpt: str | None = None,
+        config: RecognizerConfig | None = None,
+        device: str = "auto",
+        attn_implementation: str = "sdpa",
+        batch_size: int = 8,
+    ) -> None:
+        import torch
+        from transformers import AutoModelForImageTextToText, AutoProcessor
+
+
+        self._torch = torch
+        self.config = config or RecognizerConfig()
+        # RecognizerConfig.batch_size sizes a vLLM chunk (~2048). Generating that many at once
+        # here would simply OOM; HF batches are bounded by memory, not by scheduler behaviour.
+        self.batch_size = batch_size
+        if ckpt is None:
+            raise ValueError(
+                "no recognizer weights given -- "
+                "IndicDocParser.from_pretrained(snapshot_download(REPO))"
+            )
+        self.ckpt = ckpt
+
+        self.processor = AutoProcessor.from_pretrained(self.ckpt)
+        tokenizer = self.processor.tokenizer
+        # Left padding so every sequence in a batch ends flush against the generation boundary.
+        tokenizer.padding_side = "left"
+
+        self.model = AutoModelForImageTextToText.from_pretrained(
+            self.ckpt,
+            dtype=getattr(torch, self.config.dtype),
+            device_map=device,
+            attn_implementation=attn_implementation,
+        )
+        self.model.eval()
+
+        # The checkpoint's generation_config carries eos_token_id 248044, which is an ordinary
+        # word piece, not a turn terminator. Left alone, generate() never stops and every block
+        # runs to max_new_tokens, repeating itself. vLLM does not hit this because it takes the
+        # tokenizer's EOS. Trust the tokenizer here too.
+        self.eos_token_id = tokenizer.eos_token_id
+        self.pad_token_id = tokenizer.pad_token_id or tokenizer.eos_token_id
+
+    def _prompt(self, text: str) -> str:
+        return self.processor.apply_chat_template(
+            [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": text}]}],
+            add_generation_prompt=True,
+            tokenize=False,
+        )
+
+    def transcribe(self, requests: list[CropRequest]) -> list[str]:
+        texts: list[str] = []
+        for i in range(0, len(requests), self.batch_size):
+            chunk = requests[i : i + self.batch_size]
+            inputs = self.processor(
+                text=[self._prompt(r.prompt) for r in chunk],
+                images=[r.image for r in chunk],
+                padding=True,
+                return_tensors="pt",
+            ).to(self.model.device)
+            prompt_len = inputs["input_ids"].shape[-1]
+
+            with self._torch.inference_mode():
+                out = self.model.generate(
+                    **inputs,
+                    max_new_tokens=self.config.max_tokens,
+                    do_sample=False,
+                    use_cache=True,
+                    eos_token_id=self.eos_token_id,
+                    pad_token_id=self.pad_token_id,
+                )
+            texts.extend(self.processor.batch_decode(out[:, prompt_len:], skip_special_tokens=True))
+        return [t.strip() for t in texts]
+
+    def close(self) -> None:
+        self.model = None
+        self.processor = None

idp_reconstruct.py

@@ -0,0 +1,112 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/engine/reconstruct.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""Assemble transcribed blocks into the page's markdown.
+
+stdlib-only -- importable with no GPU stack and no PIL.
+"""
+
+from __future__ import annotations
+
+import re
+
+from idp_types import Block
+from idp_contract import DROP_TYPES
+
+# Dashes as codepoints: several of the seven are indistinguishable in a source file.
+# 002D HYPHEN-MINUS | 2010-2014 HYPHEN..EM DASH | 2212 MINUS SIGN
+_HYPHEN_BREAK = re.compile("(\\w)[-\\u2010-\\u2014\\u2212]\\n[ \\t]*(\\w)")
+_MATH_DELIMITERS = ("$", "\\[", "\\(")
+
+# A math span: $$...$$ (display) or $...$ (inline).
+_MATH_SPAN = re.compile(r"\$\$(.+?)\$\$|(?<!\$)\$(?!\$)([^$\n]+?)\$(?!\$)", re.S)
+
+# Runs of the scripts IndicBlockOCR transcribes: Arabic (Urdu, Kashmiri, Sindhi),
+# Devanagari..Malayalam, and Ol Chiki (Santali). Intervening spaces and the ZW(N)J joiners that
+# Indic shaping relies on are kept inside the run so one run does not fragment into many.
+# The run must both start and end on a script character, so a trailing space stays outside the
+# \text{} and keeps separating it from what follows.
+_NON_LATIN_RUN = re.compile(
+    "[\\u0600-\\u06ff\\u0900-\\u0d7f\\u1c50-\\u1c7f]"
+    "(?:[\\u0600-\\u06ff\\u0900-\\u0d7f\\u1c50-\\u1c7f \\u200c\\u200d]*"
+    "[\\u0600-\\u06ff\\u0900-\\u0d7f\\u1c50-\\u1c7f])?"
+)
+_TEXT_CMD = re.compile(r"\\text\{[^{}]*\}")
+
+
+def dehyphenate(text: str) -> str:
+    """Rejoin words split by a hyphen at a line break.
+
+    Iterates to a fixpoint: re.sub matches non-overlappingly, so in "a-\\nb-\\nc" the first pass
+    consumes the "b" the second break needs.
+    """
+    previous = None
+    while previous != text:
+        previous = text
+        text = _HYPHEN_BREAK.sub(r"\1\2", text)
+    return text
+
+
+def _repair_expression(tex: str, display: bool) -> str:
+    """Make one transcribed expression valid LaTeX.
+
+    Two repairs, both for things the recognizer emits that no LaTeX engine accepts:
+
+    * **Indic script in math mode.** ``$$প্রোটন = 9$$`` is invalid -- math mode has no glyphs for
+      those codepoints, so KaTeX, MathJax and a real TeX run all fail on it. Each run is wrapped
+      in ``\\text{}``, which is what the recognizer itself does when it gets it right. Runs
+      already inside ``\\text{}`` are left alone.
+    * **Bare newlines in display math**, which are a syntax error; ``\\\\`` is the row separator.
+    """
+    protected: list[str] = []
+
+    def stash(match: re.Match) -> str:
+        protected.append(match.group(0))
+        return f"\x00{len(protected) - 1}\x00"
+
+    tex = _TEXT_CMD.sub(stash, tex)
+    tex = _NON_LATIN_RUN.sub(lambda m: f"\\text{{{m.group(0)}}}", tex)
+    tex = re.sub(r"\x00(\d+)\x00", lambda m: protected[int(m.group(1))], tex)
+
+    if display:
+        tex = re.sub(r"\s*\n\s*", r" \\\\ ", tex.strip())
+    return tex
+
+
+def repair_math(text: str) -> str:
+    """Repair every math span in a markdown string. Prose outside ``$`` is untouched."""
+
+    def fix(match: re.Match) -> str:
+        display = match.group(1) is not None
+        inner = _repair_expression(match.group(1) or match.group(2), display)
+        return f"$${inner}$$" if display else f"${inner}$"
+
+    return _MATH_SPAN.sub(fix, text)
+
+
+def reconstruct(blocks: list[Block], repair: bool = True) -> str:
+    """Reading-ordered markdown. Blocks with no text contribute nothing but are not removed --
+    they still appear in the JSON with text "" .
+
+    ``repair=False`` emits the recognizer's math verbatim, including expressions no LaTeX engine
+    can render. Only useful for comparing byte-for-byte against output produced before the repair
+    existed.
+    """
+    kept = sorted((b for b in blocks if b.type not in DROP_TYPES), key=lambda b: b.order)
+
+    parts = []
+    for block in kept:
+        text = (block.text or "").strip()
+        if not text:
+            continue
+        # Bare LaTeX would render as literal source, so wrap it -- but only when the recognizer
+        # supplied no delimiters at all. Checking just the first character is not enough: an
+        # Equation block often comes back as a prose prefix followed by already-delimited math
+        # ("বা, $\\frac{a}{b}$"), and wrapping that produces "$$...$$$", which nothing can parse.
+        if block.type == "Equation" and "$" not in text and not text.startswith(_MATH_DELIMITERS):
+            text = f"$${text}$$"
+        parts.append(text)
+
+    markdown = dehyphenate("\n\n".join(parts))
+    return repair_math(markdown) if repair else markdown

idp_types.py

@@ -0,0 +1,246 @@
+# GENERATED by hub/build_hub_package.py from src/bodhan_genai/ocr/engine/types.py -- do not edit.
+# Vendored so this repo is self-contained: `pip install transformers torch pillow` is the
+# whole install. See indic_doc_parser.py for usage.
+
+"""Plain-data types and configuration.
+
+Every tunable that used to be an environment variable read at import time lives here instead.
+That fixes a real defect: callers corrected the old module-level defaults by setting os.environ
+*before* importing the engine, two shipped callers set different values, and importing it
+directly gave a third pipeline. These defaults ARE the canonical recipe.
+
+stdlib-only -- importable with no GPU stack and no PIL.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+from dataclasses import dataclass, field
+
+from idp_contract import TableFormat
+
+
+class LayoutSchemaError(ValueError):
+    """An incoming layout record cannot be interpreted unambiguously.
+
+    Raised rather than warned because the alternative is silent and wrong. A label the taxonomy
+    does not contain falls through to ``Text``, so a table would be sent the prose prompt and
+    come back as flattened text with nothing in the output saying so. Our own detector only ever
+    emits ``layout.labels.CLASSES``, so an unrecognised label always means either a third-party
+    taxonomy -- which should declare ``type`` -- or a typo. Both are worth stopping for.
+
+    ``map_label`` itself stays total: 19 of the 37 classes are deliberately absent from
+    ``LABEL_TO_TYPE`` and correctly mean ``Text``, so leniency belongs there. Validation belongs
+    here, at the boundary where foreign JSON enters.
+    """
+
+
+def _unknown(value, candidates, kind: str) -> str:
+    """``unknown label 'Tabel'; did you mean 'Table'?`` -- the suggestion is most of the value."""
+    import difflib
+
+    match = difflib.get_close_matches(str(value), list(candidates), n=1, cutoff=0.6)
+    hint = f"; did you mean {match[0]!r}?" if match else ""
+    return f"unknown {kind} {value!r}{hint}"
+
+
+@dataclass
+class Block:
+    """One detected region. ``text`` is None before OCR, and "" for blocks deliberately not
+    transcribed (kept in place rather than deleted)."""
+
+    order: int
+    label: str
+    type: str
+    bbox_xyxy: list[float]
+    conf: float
+    text: str | None = None
+
+    def as_record(self) -> dict:
+        # Key order is load-bearing: json.dump writes insertion order and the regression gate
+        # compares byte for byte. Do not reorder.
+        record: dict = {
+            "order": self.order,
+            "label": self.label,
+            "type": self.type,
+            "bbox_xyxy": [round(float(v), 1) for v in self.bbox_xyxy],
+            "conf": round(float(self.conf), 3),
+        }
+        if self.text is not None:
+            record["text"] = self.text
+        return record
+
+    @classmethod
+    def problems(cls, record: dict) -> list[str]:
+        """Everything wrong with one incoming block record, as readable strings.
+
+        Returns rather than raises so a whole page can be reported at once. Fixing a 60-block
+        layout one exception at a time is the difference between one edit and sixty.
+        """
+        from idp_model_labels import CLASSES
+        from idp_contract import DROP_TYPES, KEPT_BLOCK_TYPES
+
+        found: list[str] = []
+        for key in ("order", "bbox_xyxy"):
+            if key not in record:
+                found.append(f"missing required key {key!r}")
+
+        bbox = record.get("bbox_xyxy")
+        if bbox is not None:
+            try:
+                if len([float(v) for v in bbox]) != 4:
+                    found.append(f"bbox_xyxy must be 4 numbers [x0, y0, x1, y1], got {bbox!r}")
+            except (TypeError, ValueError):
+                found.append(f"bbox_xyxy must be 4 numbers [x0, y0, x1, y1], got {bbox!r}")
+
+        # An explicit `type` is the documented escape hatch for a foreign taxonomy, so the label
+        # is not checked when one is given -- but the type itself is. Left unchecked, `Tabel`
+        # is accepted verbatim and never matches `Table`, which is the same silent failure one
+        # level up.
+        declared = record.get("type")
+        if declared:
+            valid = tuple(KEPT_BLOCK_TYPES) + tuple(sorted(DROP_TYPES))
+            if str(declared) not in valid:
+                found.append(_unknown(declared, valid, "type"))
+        else:
+            label = str(record.get("label", "")).strip()
+            if label.lower() not in {c.strip().lower() for c in CLASSES}:
+                found.append(
+                    _unknown(label, CLASSES, "label")
+                    + ' -- use a spelling from layout.labels.CLASSES, or declare "type"'
+                    " explicitly if your detector has its own taxonomy"
+                )
+        return found
+
+    @classmethod
+    def from_record(cls, record: dict, *, strict: bool = True) -> Block:
+        # `type` is derived when absent, so a third-party layout carrying only labels works.
+        from idp_contract import map_label
+
+        if strict:
+            found = cls.problems(record)
+            if found:
+                raise LayoutSchemaError("; ".join(found))
+
+        label = record.get("label", "")
+        return cls(
+            order=int(record["order"]),
+            label=str(label),
+            type=str(record.get("type") or map_label(label)),
+            bbox_xyxy=[float(v) for v in record["bbox_xyxy"]],
+            conf=float(record.get("conf", 1.0)),
+            text=record.get("text"),
+        )
+
+    def copy(self) -> Block:
+        return dataclasses.replace(self, bbox_xyxy=list(self.bbox_xyxy))
+
+
+@dataclass
+class PageResult:
+    image: str
+    width: int
+    height: int
+    blocks: list[Block] = field(default_factory=list)
+    markdown: str | None = None
+
+    def as_record(self) -> dict:
+        return {
+            "image": self.image,
+            "width": self.width,
+            "height": self.height,
+            "blocks": [b.as_record() for b in self.blocks],
+        }
+
+    @classmethod
+    def from_record(cls, record: dict, *, strict: bool = True) -> PageResult:
+        """``strict=False`` replays a layout without validating it -- for reading back files
+        written before validation existed, not for new integrations."""
+        blocks = list(record.get("blocks", []))
+
+        if strict:
+            errors = [
+                f"  block[{i}] (order={b.get('order', '?')!r}): {problem}"
+                for i, b in enumerate(blocks)
+                for problem in Block.problems(b)
+            ]
+            if errors:
+                raise LayoutSchemaError(
+                    f"{len(errors)} problem(s) in layout for "
+                    f"{record.get('image', '<unknown>')!r}:\n" + "\n".join(errors)
+                )
+
+        return cls(
+            image=str(record["image"]),
+            width=int(record["width"]),
+            height=int(record["height"]),
+            # Already validated above; re-checking each block would repeat the work.
+            blocks=[Block.from_record(b, strict=False) for b in blocks],
+        )
+
+
+@dataclass(frozen=True)
+class LayoutConfig:
+    conf: float = 0.5  # below ~0.4, stains and page borders start scoring as blocks
+    img_size: int = 1024
+    device: str = "cuda"
+
+
+@dataclass(frozen=True)
+class CropConfig:
+    """The clamp is on area, not on a side: pinning a side exploded elongated crops (a 122:1
+    rule line became ~32k image tokens and wedged the engine)."""
+
+    min_px_side: int = 256  # largest single measured win; 0 disables upscaling
+    max_px_side: int = 1536  # token ceiling
+    pad_px: int = 0
+
+    @property
+    def min_px(self) -> int:
+        return self.min_px_side**2
+
+    @property
+    def max_px(self) -> int:
+        return self.max_px_side**2
+
+
+#: both -- text-like OR a larger Equation | text_only -- text-like only | eq_only -- Equation only
+DEDUP_MODES = ("both", "text_only", "eq_only")
+
+
+@dataclass(frozen=True)
+class DedupConfig:
+    """IndicDocLayout over-produces equation boxes nested inside the paragraphs and display
+    arrays that already contain them; transcribing both emits the same math twice."""
+
+    nest: bool = True
+    mode: str = "both"
+    contain: float = 0.90  # duplicate threshold in clean_layout
+    wrap: float = 0.5  # a header counts as occupied at this containment
+    nested: float = 0.70  # nested-equation threshold
+
+    def __post_init__(self) -> None:
+        if self.mode not in DEDUP_MODES:
+            raise ValueError(f"DedupConfig.mode must be one of {DEDUP_MODES}, got {self.mode!r}")
+
+
+@dataclass(frozen=True)
+class RecognizerConfig:
+    max_model_len: int = 8192
+    max_tokens: int = 2048
+    temperature: float = 0.0  # greedy: the only setting reproducible run to run
+    gpu_memory_utilization: float = 0.80
+    dtype: str = "bfloat16"
+    # One giant batch over tens of thousands of multi-modal requests wedges the vLLM V1
+    # scheduler at 100% util with no progress; ~2k chunks run clean.
+    batch_size: int = 2048
+    enforce_eager: bool = True  # skips ~4 min of torch.compile on a 0.8B model
+    table_format: TableFormat = TableFormat.HTML
+
+    def merged(self, **overrides) -> RecognizerConfig:
+        """New config with non-None overrides applied; unknown keys raise."""
+        known = {f.name for f in dataclasses.fields(self)}
+        unknown = sorted(set(overrides) - known)
+        if unknown:
+            raise TypeError(f"Unknown RecognizerConfig field(s): {', '.join(unknown)}")
+        return dataclasses.replace(self, **{k: v for k, v in overrides.items() if v is not None})

indic_doc_parser.py

@@ -0,0 +1,147 @@
+"""IndicDocParser -- page image in, reading-ordered Markdown and per-block JSON out.
+
+    import sys
+    from huggingface_hub import snapshot_download
+
+    repo = snapshot_download("bodhan-ai/indic-doc-parser")
+    sys.path.insert(0, repo)                       # the code ships in the repo
+    from indic_doc_parser import IndicDocParser
+
+    parser = IndicDocParser.from_pretrained(repo)
+    print(parser("page.png"))                      # markdown
+
+``sys.path.insert`` is needed because ``snapshot_download`` returns a cache directory, which is
+not importable on its own. With the path added, this is ordinary Python -- no
+``trust_remote_code``, and nothing of ours to install.
+
+The two stages are also public, and each runs on its own:
+
+    IndicDocLayout(f"{repo}/weights/layout").detect("page.png")   # blocks and reading order
+    IndicBlockOCR(f"{repo}/weights/ocr").run("page.png", layout)  # layout may be your own
+
+``layout`` there can be a PageResult, a dict, or the path to a layout JSON file, so the stages
+compose across processes and you can correct a layout before transcribing it.
+"""
+
+from __future__ import annotations
+
+import importlib
+from pathlib import Path
+
+
+def _preflight() -> None:
+    """One clear error instead of a cascade of ImportErrors from inside transformers.
+
+    Every failure here is one someone has actually hit: a torch built for a CUDA line the driver
+    cannot run (silently CPU), a torchvision that does not match its torch, or a transformers
+    older than PPDocLayoutV3.
+    """
+    fix = "./install.sh  (or see the Installation section of the model card)"
+    for name, floor in (("torch", (2, 4)), ("transformers", (5, 7))):
+        try:
+            mod = importlib.import_module(name)
+        except ImportError:
+            raise ImportError(f"IndicDocParser needs {name}. Run: {fix}") from None
+        got = tuple(int(p) for p in mod.__version__.split(".")[:2] if p.isdigit())
+        if got < floor:
+            raise ImportError(
+                f"IndicDocParser needs {name}>={'.'.join(map(str, floor))}, "
+                f"found {mod.__version__}. Run: {fix}"
+            )
+    # torchvision is stage 2 only, so its absence is not fatal here -- the layout stage runs
+    # without it. It is checked where the recognizer is built.
+
+
+_preflight()
+
+# ruff: noqa: E402  -- the preflight has to run before transformers is imported, which is the
+# whole point of it; these imports pull transformers in transitively.
+from idp_offline import IndicBlockOCR, IndicDocLayout
+from idp_types import CropConfig, DedupConfig, LayoutConfig, RecognizerConfig, TableFormat
+
+#: The two stages are public: either runs on its own, and IndicBlockOCR accepts a layout you
+#: produced or corrected yourself.
+__all__ = ["IndicBlockOCR", "IndicDocLayout", "IndicDocParser"]
+
+
+class IndicDocParser:
+    """Both stages. Construct with :meth:`from_pretrained`, then call it on a page image."""
+
+    def __init__(
+        self,
+        path: str | Path,
+        device: str = "cuda",
+        table_format: str = "html",
+        dedup_mode: str = "both",
+        contain: float = 0.90,
+        min_px_side: int = 256,
+        max_new_tokens: int = 2048,
+    ) -> None:
+        self.path = Path(path)
+        self._device = device
+        self._layout_cfg = LayoutConfig(device=device)
+        self._dedup = DedupConfig(mode=dedup_mode, contain=contain)
+        self._crop = CropConfig(min_px_side=min_px_side)
+        self._rec_cfg = RecognizerConfig(
+            max_tokens=max_new_tokens, table_format=TableFormat(table_format)
+        )
+        # Both stages are built on first use, so detect() never loads the 1.7 GB recognizer.
+        self._layout = None
+        self._ocr = None
+
+    @classmethod
+    def from_pretrained(cls, path: str | Path | None = None, **kwargs) -> IndicDocParser:
+        """Load from a downloaded snapshot. Defaults to the directory this file lives in, which
+        is the snapshot itself -- so ``IndicDocParser.from_pretrained()`` also works."""
+        return cls(Path(path) if path else Path(__file__).resolve().parent, **kwargs)
+
+    # -- stages ------------------------------------------------------------ #
+
+    @property
+    def layout(self):
+        if self._layout is None:
+            self._layout = IndicDocLayout(
+                ckpt=str(self.path / "weights" / "layout"),
+                config=self._layout_cfg,
+                dedup=self._dedup,
+            )
+        return self._layout
+
+    @property
+    def recognizer(self):
+        if self._ocr is None:
+            if importlib.util.find_spec("torchvision") is None:
+                raise ImportError(
+                    "The recognizer needs torchvision (its image processor uses it); the layout "
+                    "stage does not, so detect() still works. Install it together with torch, "
+                    "from the same index -- torchvision pins an exact torch version. See "
+                    "./install.sh"
+                )
+            from idp_recognizer import HfRecognizer
+
+            self._ocr = IndicBlockOCR(
+                backend=HfRecognizer(
+                    ckpt=str(self.path / "weights" / "ocr"),
+                    config=self._rec_cfg,
+                    device=self._device,
+                ),
+                config=self._rec_cfg,
+                dedup=self._dedup,
+                crop=self._crop,
+            )
+        return self._ocr
+
+    # -- public API -------------------------------------------------------- #
+
+    def detect(self, image_path: str) -> dict:
+        """Stage 1 only -- blocks, labels, reading order. Loads no recognizer."""
+        return self.layout.detect(image_path).as_record()
+
+    def parse(self, image_path: str) -> dict:
+        """Both stages -> ``{image, width, height, blocks, markdown}``."""
+        page = self.recognizer.run(image_path, self.layout.detect(image_path))
+        return {**page.as_record(), "markdown": page.markdown}
+
+    def __call__(self, image_path: str) -> str:
+        """The markdown for a page. ``parse()`` if you also want the per-block JSON."""
+        return self.parse(image_path)["markdown"]

install.sh

@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# Install IndicDocParser's dependencies into the active Python environment.
+#
+#     ./install.sh                # pick the CUDA wheels from your driver
+#     IDP_CUDA=cu121 ./install.sh # force a CUDA line
+#     IDP_CUDA=cpu   ./install.sh # CPU only -- fine for layout, slow for the recognizer
+#
+# The one thing pip cannot work out for itself is which CUDA build you need. `pip install torch`
+# takes the newest wheel, which on an older driver initialises to CPU with nothing but a warning --
+# you get a working install that silently never touches the GPU. And torchvision pins an exact
+# torch, so resolving the two separately is how you end up with a mismatched pair and a run of
+# import errors. This installs both together, from one index.
+set -euo pipefail
+
+cd "$(dirname "$0")"
+PY=${PYTHON:-python3}
+
+# `uv venv` creates environments without pip, so `$PY -m pip` is not a safe assumption. Prefer
+# whatever the environment actually has; uv is also just faster.
+if command -v uv >/dev/null 2>&1; then
+    pip_install() { uv pip install --python "$PY" "$@"; }
+elif "$PY" -m pip --version >/dev/null 2>&1; then
+    pip_install() { "$PY" -m pip install "$@"; }
+elif "$PY" -m ensurepip --upgrade >/dev/null 2>&1; then
+    pip_install() { "$PY" -m pip install "$@"; }
+else
+    echo "no installer found for $PY -- install pip (python -m ensurepip) or uv, then re-run" >&2
+    exit 1
+fi
+
+detect_cuda() {
+    command -v nvidia-smi >/dev/null 2>&1 || { echo cpu; return; }
+    local v
+    v=$(nvidia-smi 2>/dev/null | sed -n 's/.*CUDA Version: \([0-9]*\)\.\([0-9]*\).*/\1/p' | head -1)
+    case "$v" in
+        # CUDA 12.x drivers run any cu12x wheel (minor version compatibility), so one build covers
+        # the whole line. A 13.x wheel on a 12.x driver does NOT work -- that is the trap.
+        12) echo cu124 ;;
+        13|1[4-9]) echo cu130 ;;
+        "") echo cpu ;;
+        *)  echo cu124 ;;
+    esac
+}
+
+CUDA=${IDP_CUDA:-$(detect_cuda)}
+INDEX=https://download.pytorch.org/whl/$CUDA
+
+echo "==> target: $CUDA"
+[ "$CUDA" = cpu ] && echo "    (no GPU detected -- layout runs fine on CPU, the recognizer is slow)"
+
+echo "==> torch + torchvision, together, from $INDEX"
+pip_install torch torchvision --index-url "$INDEX"
+
+echo "==> everything else"
+pip_install -r requirements.txt
+
+echo "==> checking"
+$PY - <<'PY'
+import torch, torchvision, transformers
+print(f"    torch        {torch.__version__}")
+print(f"    torchvision  {torchvision.__version__}")
+print(f"    transformers {transformers.__version__}")
+print(f"    cuda         {'yes, ' + torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'NO -- running on CPU'}")
+import sys; sys.path.insert(0, ".")
+from indic_doc_parser import IndicDocParser  # noqa: F401
+print("    import       ok")
+PY
+echo "==> done"

requirements.txt

@@ -0,0 +1,19 @@
+# The model code ships in this repo; these are its only third-party dependencies.
+#
+# Install torch and torchvision TOGETHER, from the index for your CUDA version -- torchvision pins
+# an exact torch, so resolving them separately is how you end up with a mismatched pair:
+#
+#     pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
+#     pip install -r requirements.txt
+#
+# ./install.sh does this for you.
+
+torch>=2.4              # transformers 5.7 floor
+torchvision             # stage 2 only; MUST match torch (see above)
+transformers>=5.7       # PPDocLayoutV3
+accelerate>=1.1
+numpy>=1.17
+pillow>=10.0.1
+huggingface_hub>=1.0
+
+# scipy is NOT needed for inference. Finetuning the detector needs it (the Hungarian matcher).

schemas/layout_output.schema.json

@@ -0,0 +1,148 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://huggingface.co/bodhan-ai/indic-doc-parser/raw/main/schemas/layout_output.schema.json",
+  "title": "Layout JSON (stage-1 output, stage-2 input)",
+  "description": "Blocks carry no 'text' key. Absence means not transcribed yet, which is distinct from text: \"\" in a parsed page.",
+  "type": "object",
+  "required": [
+    "image",
+    "width",
+    "height",
+    "blocks"
+  ],
+  "properties": {
+    "image": {
+      "type": "string"
+    },
+    "width": {
+      "type": "integer",
+      "exclusiveMinimum": 0
+    },
+    "height": {
+      "type": "integer",
+      "exclusiveMinimum": 0
+    },
+    "blocks": {
+      "type": "array",
+      "items": {
+        "$ref": "#/$defs/block"
+      }
+    }
+  },
+  "$defs": {
+    "block": {
+      "type": "object",
+      "required": [
+        "order",
+        "bbox_xyxy"
+      ],
+      "properties": {
+        "order": {
+          "type": "integer",
+          "minimum": 0,
+          "description": "Reading-order rank. Gap-free and 0-based across the page."
+        },
+        "label": {
+          "type": "string",
+          "description": "IndicDocLayout class. Restricted to the taxonomy unless 'type' is given."
+        },
+        "type": {
+          "type": "string",
+          "enum": [
+            "Text",
+            "Title",
+            "SectionHeader",
+            "Table",
+            "Equation",
+            "Caption",
+            "Footnote",
+            "PageHeader",
+            "PageFooter",
+            "PageNumber",
+            "Figure",
+            "Picture"
+          ],
+          "description": "Pipeline category; selects the prompt. Derived from 'label' when omitted."
+        },
+        "bbox_xyxy": {
+          "type": "array",
+          "items": {
+            "type": "number"
+          },
+          "minItems": 4,
+          "maxItems": 4,
+          "description": "Pixel [x0, y0, x1, y1], clamped to the page."
+        },
+        "conf": {
+          "type": "number",
+          "minimum": 0,
+          "maximum": 1
+        }
+      },
+      "allOf": [
+        {
+          "if": {
+            "not": {
+              "required": [
+                "type"
+              ]
+            }
+          },
+          "then": {
+            "required": [
+              "label"
+            ],
+            "properties": {
+              "label": {
+                "enum": [
+                  "Question",
+                  "Paragraph",
+                  "Answer",
+                  "List",
+                  "Title",
+                  "Section-title",
+                  "Equation",
+                  "Table",
+                  "Diagram",
+                  "Image",
+                  "MCQ",
+                  "Infobox",
+                  "Sub-section-title",
+                  "Expression",
+                  "Image-caption",
+                  "Placeholder-text",
+                  "Chart",
+                  "Solved-example",
+                  "Footnote",
+                  "Table-caption",
+                  "Sub-sub-section-title",
+                  "Footer",
+                  "Header",
+                  "Code",
+                  "Page-number",
+                  "Chapter-title",
+                  "Chapter-end-section",
+                  "Folio",
+                  "Reference",
+                  "Table-of-contents",
+                  "Index",
+                  "Advertisement",
+                  "Author",
+                  "Dateline",
+                  "Contact-info",
+                  "Website-link",
+                  "Flag"
+                ]
+              }
+            }
+          }
+        }
+      ],
+      "not": {
+        "required": [
+          "text"
+        ]
+      }
+    }
+  }
+}

schemas/parse_output.schema.json

@@ -0,0 +1,148 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://huggingface.co/bodhan-ai/indic-doc-parser/raw/main/schemas/parse_output.schema.json",
+  "title": "Parsed page JSON (final output)",
+  "description": "Every block carries 'text'.",
+  "type": "object",
+  "required": [
+    "image",
+    "width",
+    "height",
+    "blocks"
+  ],
+  "properties": {
+    "image": {
+      "type": "string"
+    },
+    "width": {
+      "type": "integer",
+      "exclusiveMinimum": 0
+    },
+    "height": {
+      "type": "integer",
+      "exclusiveMinimum": 0
+    },
+    "blocks": {
+      "type": "array",
+      "items": {
+        "$ref": "#/$defs/block"
+      }
+    }
+  },
+  "$defs": {
+    "block": {
+      "type": "object",
+      "required": [
+        "order",
+        "bbox_xyxy",
+        "text"
+      ],
+      "properties": {
+        "order": {
+          "type": "integer",
+          "minimum": 0,
+          "description": "Reading-order rank. Gap-free and 0-based across the page."
+        },
+        "label": {
+          "type": "string",
+          "description": "IndicDocLayout class. Restricted to the taxonomy unless 'type' is given."
+        },
+        "type": {
+          "type": "string",
+          "enum": [
+            "Text",
+            "Title",
+            "SectionHeader",
+            "Table",
+            "Equation",
+            "Caption",
+            "Footnote",
+            "PageHeader",
+            "PageFooter",
+            "PageNumber",
+            "Figure",
+            "Picture"
+          ],
+          "description": "Pipeline category; selects the prompt. Derived from 'label' when omitted."
+        },
+        "bbox_xyxy": {
+          "type": "array",
+          "items": {
+            "type": "number"
+          },
+          "minItems": 4,
+          "maxItems": 4,
+          "description": "Pixel [x0, y0, x1, y1], clamped to the page."
+        },
+        "conf": {
+          "type": "number",
+          "minimum": 0,
+          "maximum": 1
+        },
+        "text": {
+          "type": "string",
+          "description": "Transcription. \"\" means detected but deliberately not sent to the recognizer; the block is kept, not dropped."
+        }
+      },
+      "allOf": [
+        {
+          "if": {
+            "not": {
+              "required": [
+                "type"
+              ]
+            }
+          },
+          "then": {
+            "required": [
+              "label"
+            ],
+            "properties": {
+              "label": {
+                "enum": [
+                  "Question",
+                  "Paragraph",
+                  "Answer",
+                  "List",
+                  "Title",
+                  "Section-title",
+                  "Equation",
+                  "Table",
+                  "Diagram",
+                  "Image",
+                  "MCQ",
+                  "Infobox",
+                  "Sub-section-title",
+                  "Expression",
+                  "Image-caption",
+                  "Placeholder-text",
+                  "Chart",
+                  "Solved-example",
+                  "Footnote",
+                  "Table-caption",
+                  "Sub-sub-section-title",
+                  "Footer",
+                  "Header",
+                  "Code",
+                  "Page-number",
+                  "Chapter-title",
+                  "Chapter-end-section",
+                  "Folio",
+                  "Reference",
+                  "Table-of-contents",
+                  "Index",
+                  "Advertisement",
+                  "Author",
+                  "Dateline",
+                  "Contact-info",
+                  "Website-link",
+                  "Flag"
+                ]
+              }
+            }
+          }
+        }
+      ]
+    }
+  }
+}

weights/layout/config.json

@@ -0,0 +1,292 @@
+{
+  "activation_dropout": 0.0,
+  "activation_function": "silu",
+  "anchor_image_size": null,
+  "architectures": [
+    "PPDocLayoutV3Trainable"
+  ],
+  "attention_dropout": 0.0,
+  "auxiliary_loss": true,
+  "backbone": null,
+  "backbone_config": {
+    "arch": "L",
+    "depths": [
+      3,
+      4,
+      6,
+      3
+    ],
+    "dtype": "float32",
+    "embedding_size": 64,
+    "freeze_at": 0,
+    "freeze_norm": true,
+    "freeze_stem_only": true,
+    "hidden_act": "relu",
+    "hidden_sizes": [
+      256,
+      512,
+      1024,
+      2048
+    ],
+    "initializer_range": 0.02,
+    "lr_mult_list": [
+      0,
+      0.05,
+      0.05,
+      0.05,
+      0.05
+    ],
+    "model_type": "hgnet_v2",
+    "num_channels": 3,
+    "out_features": [
+      "stage1",
+      "stage2",
+      "stage3",
+      "stage4"
+    ],
+    "out_indices": [
+      1,
+      2,
+      3,
+      4
+    ],
+    "return_idx": [
+      0,
+      1,
+      2,
+      3
+    ],
+    "stage_downsample": [
+      false,
+      true,
+      true,
+      true
+    ],
+    "stage_downsample_strides": [
+      2,
+      2,
+      2,
+      2
+    ],
+    "stage_in_channels": [
+      48,
+      128,
+      512,
+      1024
+    ],
+    "stage_kernel_size": [
+      3,
+      3,
+      5,
+      5
+    ],
+    "stage_light_block": [
+      false,
+      false,
+      true,
+      true
+    ],
+    "stage_mid_channels": [
+      48,
+      96,
+      192,
+      384
+    ],
+    "stage_names": [
+      "stem",
+      "stage1",
+      "stage2",
+      "stage3",
+      "stage4"
+    ],
+    "stage_num_blocks": [
+      1,
+      1,
+      3,
+      1
+    ],
+    "stage_numb_of_layers": [
+      6,
+      6,
+      6,
+      6
+    ],
+    "stage_out_channels": [
+      128,
+      512,
+      1024,
+      2048
+    ],
+    "stem_channels": [
+      3,
+      32,
+      48
+    ],
+    "stem_strides": [
+      2,
+      1,
+      1,
+      2,
+      1
+    ],
+    "use_learnable_affine_block": false
+  },
+  "batch_norm_eps": 1e-05,
+  "box_noise_scale": 1.0,
+  "d_model": 256,
+  "decoder_activation_function": "relu",
+  "decoder_attention_heads": 8,
+  "decoder_ffn_dim": 1024,
+  "decoder_in_channels": [
+    256,
+    256,
+    256
+  ],
+  "decoder_layers": 6,
+  "decoder_n_points": 4,
+  "disable_custom_kernels": true,
+  "dropout": 0.0,
+  "dtype": "float32",
+  "encode_proj_layers": [
+    2
+  ],
+  "encoder_activation_function": "gelu",
+  "encoder_attention_heads": 8,
+  "encoder_ffn_dim": 1024,
+  "encoder_hidden_dim": 256,
+  "encoder_in_channels": [
+    512,
+    1024,
+    2048
+  ],
+  "encoder_layers": 1,
+  "eos_coefficient": 0.0001,
+  "eval_size": null,
+  "feat_strides": [
+    8,
+    16,
+    32
+  ],
+  "feature_strides": [
+    8,
+    16,
+    32
+  ],
+  "focal_loss_alpha": 0.25,
+  "focal_loss_gamma": 2.0,
+  "freeze_backbone_batch_norms": true,
+  "global_pointer_head_size": 64,
+  "gp_dropout_value": 0.1,
+  "hidden_expansion": 1.0,
+  "id2label": {
+    "0": "Question",
+    "1": "Paragraph",
+    "2": "Answer",
+    "3": "List",
+    "4": "Title",
+    "5": "Section-title",
+    "6": "Equation",
+    "7": "Table",
+    "8": "Diagram",
+    "9": "Image",
+    "10": "MCQ",
+    "11": "Infobox",
+    "12": "Sub-section-title",
+    "13": "Expression",
+    "14": "Image-caption",
+    "15": "Placeholder-text",
+    "16": "Chart",
+    "17": "Solved-example",
+    "18": "Footnote",
+    "19": "Table-caption",
+    "20": "Sub-sub-section-title",
+    "21": "Footer",
+    "22": "Header",
+    "23": "Code",
+    "24": "Page-number",
+    "25": "Chapter-title",
+    "26": "Chapter-end-section",
+    "27": "Folio",
+    "28": "Reference",
+    "29": "Table-of-contents",
+    "30": "Index",
+    "31": "Advertisement",
+    "32": "Author",
+    "33": "Dateline",
+    "34": "Contact-info",
+    "35": "Website-link",
+    "36": "Flag"
+  },
+  "initializer_bias_prior_prob": null,
+  "initializer_range": 0.01,
+  "is_encoder_decoder": true,
+  "label2id": {
+    "Advertisement": 31,
+    "Answer": 2,
+    "Author": 32,
+    "Chapter-end-section": 26,
+    "Chapter-title": 25,
+    "Chart": 16,
+    "Code": 23,
+    "Contact-info": 34,
+    "Dateline": 33,
+    "Diagram": 8,
+    "Equation": 6,
+    "Expression": 13,
+    "Flag": 36,
+    "Folio": 27,
+    "Footer": 21,
+    "Footnote": 18,
+    "Header": 22,
+    "Image": 9,
+    "Image-caption": 14,
+    "Index": 30,
+    "Infobox": 11,
+    "List": 3,
+    "MCQ": 10,
+    "Page-number": 24,
+    "Paragraph": 1,
+    "Placeholder-text": 15,
+    "Question": 0,
+    "Reference": 28,
+    "Section-title": 5,
+    "Solved-example": 17,
+    "Sub-section-title": 12,
+    "Sub-sub-section-title": 20,
+    "Table": 7,
+    "Table-caption": 19,
+    "Table-of-contents": 29,
+    "Title": 4,
+    "Website-link": 35
+  },
+  "label_noise_ratio": 0.5,
+  "lambda_order": 5.0,
+  "layer_norm_eps": 1e-05,
+  "learn_initial_query": false,
+  "loss_type": "RTDetrForObjectDetection",
+  "mask_enhanced": true,
+  "mask_feature_channels": [
+    64,
+    64
+  ],
+  "matcher_alpha": 0.25,
+  "matcher_bbox_cost": 5.0,
+  "matcher_class_cost": 2.0,
+  "matcher_gamma": 2.0,
+  "matcher_giou_cost": 2.0,
+  "model_type": "pp_doclayout_v3",
+  "normalize_before": false,
+  "num_denoising": 0,
+  "num_feature_levels": 3,
+  "num_prototypes": 32,
+  "num_queries": 300,
+  "positional_encoding_temperature": 10000,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.8.1",
+  "use_cache": false,
+  "use_focal_loss": true,
+  "weight_loss_bbox": 5.0,
+  "weight_loss_giou": 2.0,
+  "weight_loss_vfl": 1.0,
+  "x4_feat_dim": 128
+}

weights/layout/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:051fa9741b713f164eef74ef25b88defa87670ab64995cfe431d413685bf0a3f
+size 133301428

weights/ocr/chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is true %}
+        {{- '<think>\n' }}
+    {%- else %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

weights/ocr/config.json

@@ -0,0 +1,149 @@
+{
+  "_name_or_path": "/projects/data/visionteam/sherry/qwen08b/ckpts/merges_v5/S2b",
+  "architectures": [
+    "Qwen3_5ForConditionalGeneration"
+  ],
+  "chunk_size_feed_forward": 0,
+  "dtype": "bfloat16",
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
+  },
+  "image_token_id": 262155,
+  "is_encoder_decoder": false,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "model_type": "qwen3_5",
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "problem_type": null,
+  "return_dict": true,
+  "text_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_output_gate": true,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "dtype": "bfloat16",
+    "eos_token_id": 262146,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 3584,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 16,
+    "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "mlp_only_layers": [],
+    "model_type": "qwen3_5_text",
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "num_attention_heads": 8,
+    "num_hidden_layers": 24,
+    "num_key_value_heads": 2,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "pad_token_id": 262144,
+    "partial_rotary_factor": 0.25,
+    "problem_type": null,
+    "return_dict": true,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "tie_word_embeddings": true,
+    "use_cache": false,
+    "vocab_size": 262157
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.6.2",
+  "use_cache": false,
+  "video_token_id": 262156,
+  "vision_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "chunk_size_feed_forward": 0,
+    "deepstack_visual_indexes": [],
+    "depth": 12,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 768,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "model_type": "qwen3_5_vision",
+    "num_heads": 12,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 1024,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "patch_size": 16,
+    "problem_type": null,
+    "return_dict": true,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 262154,
+  "vision_start_token_id": 262153
+}

weights/ocr/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 262146,
+  "transformers_version": "5.6.2",
+  "use_cache": true
+}

weights/ocr/model-00001-of-00001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40f6a5ce321d65ce18db429c9e8c8b2d2d22d4ec7d71737d150cc0a6edf9ae5f
+size 1734372747

weights/ocr/model.safetensors.index.json

@@ -0,0 +1,480 @@
+{
+  "metadata": {
+    "total_size": 1734310016
+  },
+  "weight_map": {
+    "model.language_model.layers.0.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.dt_bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.A_log": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.self_attn.q_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.self_attn.k_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.self_attn.q_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.self_attn.k_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.self_attn.q_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.self_attn.k_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.self_attn.q_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.self_attn.k_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.self_attn.q_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.self_attn.k_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.self_attn.q_norm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.self_attn.k_norm.weight": "model-00001-of-00001.safetensors",
+    "model.visual.patch_embed.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.norm1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.norm1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.norm2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.norm2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.attn.proj.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.visual.merger.norm.weight": "model-00001-of-00001.safetensors",
+    "model.visual.merger.norm.bias": "model-00001-of-00001.safetensors",
+    "model.visual.merger.linear_fc2.bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.input_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.post_attention_layernorm.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.norm.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.attn.qkv.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.visual.merger.linear_fc1.bias": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.conv1d.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.in_proj_b.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.in_proj_a.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.self_attn.k_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.self_attn.v_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.self_attn.k_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.self_attn.v_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.self_attn.k_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.self_attn.v_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.self_attn.k_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.self_attn.v_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.self_attn.k_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.self_attn.v_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.self_attn.k_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.self_attn.v_proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.attn.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.patch_embed.proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.pos_embed.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.attn.qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.self_attn.o_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.self_attn.o_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.self_attn.o_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.self_attn.o_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.self_attn.o_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.out_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.in_proj_z.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.self_attn.o_proj.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.0.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.1.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.2.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.3.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.4.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.5.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.6.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.7.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.8.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.9.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.10.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.visual.blocks.11.mlp.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.visual.merger.linear_fc2.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.mlp.gate_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.mlp.up_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.mlp.down_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.3.self_attn.q_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.7.self_attn.q_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.11.self_attn.q_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.15.self_attn.q_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.19.self_attn.q_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.23.self_attn.q_proj.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.0.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.1.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.2.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.4.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.5.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.6.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.8.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.9.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.10.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.12.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.13.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.14.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.16.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.17.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.18.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.20.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.21.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.layers.22.linear_attn.in_proj_qkv.weight": "model-00001-of-00001.safetensors",
+    "model.visual.merger.linear_fc1.weight": "model-00001-of-00001.safetensors",
+    "model.language_model.embed_tokens.weight": "model-00001-of-00001.safetensors"
+  }
+}

weights/ocr/processor_config.json

@@ -0,0 +1,60 @@
+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 25165824,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

weights/ocr/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa502d4d33e68344b0702e5dfd90466bc1e152d36ed906bad34c48dfd3e06d41
+size 33629731

weights/ocr/tokenizer_config.json

@@ -0,0 +1,40 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<|start_of_image|>",
+  "bos_token": "[@BOS@]",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<|end_of_image|>",
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "boi_token": "<|start_of_image|>",
+    "eoi_token": "<|end_of_image|>",
+    "image_token": "<|image_pad|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "processor_class": "Qwen3VLProcessor",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}