README.md

---
license: apache-2.0
base_model: ibm-granite/granite-vision-4.1-4b
tags:
  - document-intelligence
  - structured-extraction
  - vision-language
  - ocr
  - rag
language:
  - en
pipeline_tag: image-text-to-text
---

# shrew-ocr-preview

shrew-ocr-preview converts one document page image per request into a single JSON object
containing document metadata, a summary, self-contained semantic chunks (sized for RAG ingestion,
not raw OCR lines), and figures/tables with bounding boxes and HTML. A text modality accepts
HTML/markdown/plain-text input and produces the same output schema.

Fine-tuned from [ibm-granite/granite-vision-4.1-4b](https://huggingface.co/ibm-granite/granite-vision-4.1-4b)
(merged weights, bf16). A GPTQ-8bit quantization with near-identical fidelity is released alongside
this model for faster serving.

> **Preview release.** Works well on mainstream printed documents (papers, reports, filings,
> manuals). Known failure modes are listed under Limitations; measured results under Results.
> Weights are updated in place under this name — pin a commit (`revision=`) for reproducibility.

## Output schema

One request = one page. The model returns exactly one JSON object, five keys always present:

```
{
  "metadata":        {"title", "authors" (list), "organization", "year", "doc_type"} — null where unknown,
  "summary":         str | null,
  "semantic_chunks": [{"chunk_id", "title", "content",
                       "section_type" ∈ {abstract, introduction, methodology, results,
                                          discussion, conclusion, technical_content, appendix}}],
  "figures":         [{"bbox": [x0,y0,x1,y1] | null, "caption", "description"}],
  "tables":          [{"html": "<table>…", "bbox": [...] | null, "caption", "description"}]
}
```

Bounding boxes are xyxy on a 0–1000 normalized grid over the page image. In text modality, bboxes
are null.

## Usage

**Recommended path: [shrew-server](https://github.com/btbtyler09/shrew-server)** (MIT, pin tag
`v0.2.1`), the reference server for this model. It implements the model's entire input contract
server-side — glyph-routed bucket preprocessing, the `structured_extraction` request shape, tuned
decoding with a schema-enforced retry tier, the streaming repetition guard, schema/coercion
gates, and multi-page assembly. POST a PDF, receive structured JSON. It does not serve the model
itself; point it at an OpenAI-compatible endpoint (vLLM, below):

```bash
vllm serve btbtyler09/shrew-ocr-preview --trust-remote-code \
  --served-model-name shrew-ocr-preview \
  --max-model-len 32768 --limit-mm-per-prompt '{"image":1}' --no-enable-prefix-caching
VLM_URL=http://localhost:8000 VLM_MODEL=shrew-ocr-preview shrew serve
curl -X POST localhost:8080/v1/convert -F file=@doc.pdf -F pipeline_mode=structured
```

Full instructions, including a Docker Compose quickstart, are in the repo README under "Using
with shrew-ocr-preview (recommended)".

For direct integration without shrew-server, the requirements below define the input contract.
Deviating from any of these degrades output quality:

**1. System prompt.** Set the system prompt to the literal string `structured_extraction`. Do not
send instruction text; the model was trained on this fixed prompt only.

**2. Decoding.** Set `temperature` to 0 and `max_tokens` to 20000. `presence_penalty` 0.3–0.6 is
measured fidelity-neutral; 0.3 is the reference server's first-pass default. Do not set top_p or
any other penalty parameter (measured basis under the repetition guard below). Serve with context
length ≥ 32768; dense pages need room for both the image tokens and a long completion.

**3. Input resolution ("buckets").** Resize each page image to one of three portrait tile grids,
selected by the page's measured glyph height (target ~10 px after resize). Training used exactly
this routing. Reference implementation:

```python
import cv2, statistics
import numpy as np
from PIL import Image

BUCKETS = [("B1", (1152, 1536)), ("B2", (1536, 2304)), ("B3", (2304, 3072))]
SQUARE = ("B0", (1152, 1152))            # square-ish inputs only (e.g. table crops)

def glyph_height(img, max_side=2600):
    """Median connected-component height in native px — the routing signal."""
    W, H = img.size
    s = min(1.0, max_side / max(W, H))
    im = img.convert("L")
    if s < 1.0:
        im = im.resize((int(W * s), int(H * s)), Image.BOX)
    g = cv2.adaptiveThreshold(np.asarray(im), 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                              cv2.THRESH_BINARY_INV, 31, 10)
    n, _, stats, _ = cv2.connectedComponentsWithStats(g, connectivity=8)
    hs = [stats[i][3] for i in range(1, n)
          if 2 <= stats[i][3] <= 60 and 1 <= stats[i][2] <= 60 and stats[i][4] >= 4
          and 0.08 <= stats[i][2] / max(stats[i][3], 1) <= 6.0]
    return statistics.median(hs) / max(s, 1e-6) if len(hs) >= 50 else None

def prepare_page(img, target=10.0):
    """Route to the smallest bucket that reaches ~10px effective glyph height, then enhance."""
    w, h = img.size
    if h and 0.9 <= w / h <= 1.15:
        bw, bh = SQUARE[1]
    else:
        g = glyph_height(img)
        bw, bh = BUCKETS[1][1]                       # default when unmeasurable
        if g:
            for _, (cw, ch) in BUCKETS:
                if g * min(cw / w, ch / h) >= target * 0.95:
                    bw, bh = cw, ch
                    break
            else:
                bw, bh = BUCKETS[-1][1]
    s = min(bw / w, bh / h)
    fit = img.resize((round(w * s), round(h * s)), Image.LANCZOS)
    gray = np.asarray(fit.convert("L"))
    e = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(gray)   # CLAHE
    blur = cv2.GaussianBlur(e, (0, 0), 1.2)
    e = cv2.addWeighted(e, 1.8, blur, -0.8, 0)                            # unsharp
    return Image.fromarray(e).convert("RGB")
```

The checkpoint's `config.json` and `preprocessor_config.json` carry the matching
`image_grid_pinpoints`. Do not remove or modify them; the tile packing must match training.

### Serving with vLLM

```bash
vllm serve /path/to/shrew-ocr-preview \
  --trust-remote-code --dtype bfloat16 \
  --served-model-name shrew-ocr-preview \
  --max-model-len 32768 --limit-mm-per-prompt '{"image":1}' \
  --no-enable-prefix-caching
```

**Scaling note:** the model is small (2–5 GB weights). For batch serving on multi-GPU hosts,
data-parallel replicas (`--data-parallel-size N`) outperform tensor parallelism substantially
(+54% measured on a 4-GPU node) — prefer DP unless a single GPU cannot hold the weights. On
memory-constrained GPUs keep `--max-num-batched-tokens` at 2048 or below: the vision encoder
batches image tiles, and large prefill budgets can OOM the tower on high-tile pages.

Request shape (OpenAI-compatible):

```json
{
  "model": "shrew-ocr-preview",
  "temperature": 0,
  "max_tokens": 20000,
  "messages": [
    {"role": "system", "content": "structured_extraction"},
    {"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,<prepare_page output>"}},
      {"type": "text", "text": "Extract the structured representation of this document page."}]}
  ]
}
```

**Recommended: streaming repetition guard.** On hard pages the failure mode is degenerate
repetition, not plausible-but-wrong output. Stream the completion, compute
`len(window)/len(zlib.compress(window))` over a trailing ~2,000-character window every ~800
characters, and abort after 2 consecutive windows above ~15. Clean pages measure ~2; looping
output exceeds 25.

Penalty parameters (measured: 900 first-pass runs, 150 stratified pages × 6 decode arms on the
production stack): `presence_penalty` 0.3 at first pass raised the first-pass success rate (valid
JSON passing all schema and degeneration gates, no retry) 0.880→0.887 with extraction precision
flat (0.949→0.949 median 10-gram precision vs ground truth), and is the reference server's
first-pass default; 0.3–0.6 both measured fidelity-neutral on healthy pages. In a separate rescue
experiment, a penalized retry (`presence_penalty` 0.6) recovered 12/12 sampled loop-failed
Latin-script broadsheet pages with fidelity flat; dense non-Latin broadsheets did not recover at
any penalty (see Limitations — that class is a training gap, not a decoding one). Do not use
frequency or repetition penalties or `no_repeat_ngram_size`: ngram blocking suppresses JSON
tokens that must repeat (`"bbox": [`); frequency penalties accumulate with each repeated
occurrence, and repetition penalties apply to every previously-seen token — both degrade required
schema tokens in long structured outputs. Do not use grammar-constrained (schema-enforced)
decoding at first pass: it degrades table transcription severely (table one-shot 0.975→0.225),
primarily by overrunning the token budget mid-table (grammar-constrained decoding transcribes
exhaustively), with degraded table fidelity (0.483→0.280) on the pages that do finish; the
reference server applies enforcement only on retry. Two caveats: the Results tables were measured
with penalty-free greedy decoding (the presence-penalty recommendation comes from the separate
decode matrix), and retry rescue was validated on Latin-script pages only — for pages whose text
the vision tower cannot resolve, a penalized retry can convert a detectable loop into plausible
hallucination (observed: a penalized retry on an unsupported-script page produced fluent output
with zero n-gram overlap with the page), so validate retry output with the same window check and
schema gates and mark it lower-confidence downstream.

Which borderline pages loop varies with serving configuration (kernel paths, compilation
settings); the overall loop rate does not. Compare deployments by loop rate over a fixed page
set, not by which pages failed.

### Text modality

The same model accepts born-digital text — HTML (emails, filings), markdown, source code, plain
text — and returns the same 5-key output schema with `figures[].bbox` and `tables[].bbox` null.
Do not route OCR output of scanned pages here; scanned pages go through the image modality.

Request shape: same envelope, with the raw content as a single text part in place of the image.
Send the content as-is — no wrapper, no instructions, no cleaning:

```json
{
  "model": "shrew-ocr-preview",
  "temperature": 0,
  "max_tokens": 12000,
  "messages": [
    {"role": "system", "content": "structured_extraction"},
    {"role": "user", "content": [{"type": "text", "text": "<raw HTML / markdown / plain text>"}]}
  ]
}
```

Input sizing: send 2,000–9,000 characters (~500–2,500 tokens) per request; treat 13,000
characters as the ceiling (training inputs never exceeded it). Split longer documents at
structural boundaries (headings, sections) and send one request per section. Each request in the
recommended range yields roughly 2–6 semantic chunks (median chunk ~820 characters).

## Results — OHR-Bench document RAG

Measured on the [OHR-Bench](https://huggingface.co/datasets/opendatalab/OHR-Bench) corpus
(ICCV 2025): 1,261 PDFs / 8,561 pages across 7 domains (textbook, law, finance, newspaper, manual,
academic, administration). Every page runs through our full production path (rasterize → bucket
routing → model → schema gates → assembly); each system's structured output is chunked under the
same budget, embedded with `nvidia/llama-nemotron-embed-vl-1b-v2` ("nemotron-vl"), and scored as
retrieval hit@5 / MRR@10 over OHR-Bench's ~8.5k
human-verified Q&A pairs. These are our own retrieval-harness measurements, not official OHR-Bench
generation (LCS/F1) numbers. `gt` is retrieval over OHR-Bench's human ground-truth structured
data; MinerU and PaddleOCR outputs were run through the identical chunking and indexing.

**Text retrieval, hit@5 / MRR@10 by evidence type** (higher is better, best per row in bold):

| evidence type | gt (human) | MinerU | PaddleOCR | shrew bf16 | shrew INT8 |
|---|---|---|---|---|---|
| plain text | .853 / .781 | .872 / .791 | .887 / .812 | **.896** / **.814** | .889 / .808 |
| multi-evidence | .859 / .769 | .911 / .838 | .867 / .772 | .889 / .770 | **.933** / **.840** |
| table | .864 / .756 | **.889** / **.761** | .858 / .741 | .859 / .737 | .870 / .749 |
| formula | .871 / .791 | .895 / .807 | .878 / .807 | .905 / .805 | **.913** / **.821** |
| chart | **.776** / **.665** | .594 / .491 | .546 / .434 | .673 / .551 | .711 / .582 |
| vision | .693 / .505 | .597 / .458 | .660 / .525 | .664 / .534 | **.714** / **.571** |
| reading order† | .844 / .757 | **.902** / **.825** | .877 / .786 | .122 / .105 | .077 / .070 |

† Known failure. OHR-Bench draws reading-order queries almost entirely from dense broadsheet
newspaper scans, which fall in this model's repetition-loop failure class (see Limitations); with
those pages unparsed, the attainable ceiling is ~0.14. The bf16/INT8 gap on this row is a
harness-gating artifact, not quantization: the bf16 number is inflated by hallucinated filler
that the final output gate rejects. Treat broadsheet reading order as unsupported in this
release.

**Figure/table localization** vs our own frozen human-annotated gold subset — 551 corpus pages /
1,100 boxes, not an OHR-Bench artifact (greedy match at IoU ≥ 0.5):

| arm | figure recall@0.5 | figure mean IoU | table recall@0.5 | table mean IoU |
|---|---|---|---|---|
| bf16 | 0.627 | 0.795 | 0.598 | 0.803 |
| INT8 | 0.618 | 0.796 | 0.603 | 0.804 |

**Reliability:** 84.7% (bf16) / 84.35% (INT8) of pages produce valid schema-complete JSON on the
first pass; 97.2% after mechanical schema coercion. Hard failures are ~3% of pages, concentrated
in the dense-broadsheet loop class, and terminate as repetition-guard aborts rather than silent
bad output.

**Quantization cost:** the INT8 variant matches or beats bf16 on 5 of 7 retrieval types, is within
noise on localization, and measures +0.25% domain perplexity. See the GPTQ-8bit repo for
serving-throughput numbers.

## Limitations

- **Difficult documents.** Dense broadsheet scans (historical newspapers), low-resolution scans of
  dense layouts, and pages whose text the vision tower cannot resolve can produce repetition loops
  instead of output. The streaming guard above converts these into fast, detectable failures. Work
  on this class is ongoing.
- **CJK, Cyrillic, Arabic and handwriting are unsupported.** The model is trained and evaluated on
  Latin-script print; non-Latin scripts loop or transcribe poorly. Multilingual coverage is
  planned.
- **Bounding boxes are model-supervised.** Figure/table geometry is trained from model-generated
  labels with automated repair; boxes are generally tight but can under- or over-shoot on unusual
  layouts. Pad boxes outward slightly when cropping; do not treat edges as pixel-exact.
- **One page per request.** The model has no cross-page state; feed multi-page documents page by
  page and assemble downstream.
- **Reading order on dense broadsheets** scores near the failure floor (see Results). Same
  failure class as the first bullet.

## Versions

| variant | precision | size | notes |
|---|---|---|---|
| shrew-ocr-preview | bf16 (this repo) | 7.5 GB | reference quality |
| [shrew-ocr-preview-GPTQ-8bit](https://huggingface.co/btbtyler09/shrew-ocr-preview-GPTQ-8bit) | INT8 LM / bf16 vision | 4.9 GB | ~1.8× serving throughput, +0.25% domain perplexity; serve with `--dtype half` |
| [shrew-ocr-preview-GGUF](https://huggingface.co/btbtyler09/shrew-ocr-preview-GGUF) | Q8_0 or f16 LM / f16 vision | 3.6–6.8 GB | llama.cpp; full context per slot required (`-c` = N × 32768) |
| [shrew-ocr-preview-lora](https://huggingface.co/btbtyler09/shrew-ocr-preview-lora) | LoRA adapter (r=256, bf16) | 2.0 GB | for composition / continued training — serve the merged variants instead |

This is a preview: weights update in place under these names as the model improves. Each weight
push's commit message records the training and calibration generation — pin a commit
(`revision=`) for reproducibility.

Base model: ibm-granite/granite-vision-4.1-4b (Apache 2.0). The vision tower is unchanged from the
base; all fine-tuning lives in the language model.