shrew-ocr-preview-GGUF

GGUF conversions of shrew-ocr-preview for llama.cpp. The model converts one document page image per request into a single JSON object containing document metadata, a summary, RAG-ready semantic chunks, and figures/tables with bounding boxes and HTML — see the merged model card for the output schema and full input contract.

Converted from the bf16 merged checkpoint with llama.cpp release b10453 (native Granite4Vision support). Only the language model is quantized; the vision tower is f16 in the mmproj — the same layout as the GPTQ variant.

Files

file	size	validated envelope
shrew-ocr-preview-Q8_0.gguf	3.6 GB	12 GB-class GPUs at the full contract; up to 2 slots with KV quantization (see Serving)
shrew-ocr-preview-f16.gguf	6.8 GB	reference precision; requires ≥16 GB VRAM at full 32k context (CUDA OOM on 12 GB with large-tile pages)
shrew-ocr-preview-mmproj-f16.gguf	1.16 GB	required alongside either LM file

SHA256SUMS is included; verify downloads against it.

Serving

llama-server splits -c evenly across parallel slots, and a slot with less than the full 32k context crashes on dense pages (exit 139). The rule: set -c to N × 32768 for -np N — full context per slot. This is a correctness requirement independent of VRAM.

# single slot — safe everywhere (~2.8 pages/min on a 12 GB RTX 3060)
llama-server -m shrew-ocr-preview-Q8_0.gguf \
  --mmproj shrew-ocr-preview-mmproj-f16.gguf \
  -ngl 99 -c 32768 -np 1 --host 0.0.0.0 --port 8080

# two slots on 12 GB — KV-cache quantization REQUIRED (f16 KV at 64k crashes under load);
# measured fidelity-neutral, ~2x throughput (~5.7 pages/min)
llama-server -m shrew-ocr-preview-Q8_0.gguf \
  --mmproj shrew-ocr-preview-mmproj-f16.gguf \
  -ngl 99 -c 65536 -np 2 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --host 0.0.0.0 --port 8080

Multi-slot validation (50-page gate, 12 GB RTX 3060, paired per-page fidelity vs single-slot): -np 2 + q8_0 KV scored first-pass 46/50, precision 0.939, paired fidelity delta 0.0000 — vs 44/50 / 0.935 single-slot. q8_0 KV quantization alone is also fidelity-neutral (paired delta 0.0000). Larger cards can scale -np further under the same -c = N × 32768 rule (unmeasured beyond N=2). -np 2 with f16 KV on 12 GB crashes under load; do not run it without the KV quantization flags.

Zero-setup alternative: docker run --gpus all -p 8000:8000 btbtyler09/shrew-ocr-cuda:v0.2.1 serves this model with the Q8_0 files baked in. Env vars NP / CTX / KV_QUANT expose the same knobs (default NP=1, always safe).

The client contract is unchanged from the merged model card: glyph-routed bucket preprocessing, the structured_extraction system prompt, temperature 0 (plus presence_penalty 0.3 per the reference server). Point shrew-server at this endpoint, or follow the merged card's Usage section for direct integration.

Validation

Q8_0 gate — 50 pages against the vLLM GPTQ-8bit variant on identical pages: first-pass ok 44/50 vs 43/50, extraction precision 0.935 vs 0.937, paired table-fidelity delta median 0.000, bbox emission verified. Throughput ~30 s/page single-stream on an RTX 3060 (vLLM: ~28 s/page).

f16 — quality spot-check 9/10 pages ok, precision 0.942 (measured at -c 16384 on an RTX 3060; the full 32k contract needs ≥16 GB VRAM).

Versions

variant	precision	size	notes
shrew-ocr-preview	bf16	7.5 GB	reference quality
shrew-ocr-preview-GPTQ-8bit	INT8 LM / bf16 vision	4.9 GB	~1.8× serving throughput, +0.25% domain perplexity; serve with --dtype half
shrew-ocr-preview-GGUF	Q8_0 or f16 LM / f16 vision (this repo)	3.6–6.8 GB	llama.cpp; full context per slot required (-c = N × 32768)
shrew-ocr-preview-lora	LoRA adapter (r=256, bf16)	2.0 GB	for composition / continued training

This is a preview: weights update in place under these names as the model improves, and the repos are pushed in lockstep from the same training generation. Each push's commit message records the generation — pin a commit (revision=) for reproducibility.

Base model: ibm-granite/granite-vision-4.1-4b (Apache 2.0). All fine-tuning lives in the language model; the vision tower is unchanged from the base.