Configuration Parsing Warning:In UNKNOWN_FILENAME: "quantization_config.config_groups.group_0.format" must be a string
shrew-ocr-preview-FP8
W8A8 FP8 quantization of shrew-ocr-preview v0.5 for vLLM: page image in, structured JSON out. The 280 language-model linear layers are float8 (e4m3) with per-output-channel weight scales and a static per-tensor input scale each; the vision tower, projectors, embeddings, norms and lm_head stay bf16. Benchmarks, schema, decoding and input contract are on the main card.
Preview: weights update in place under this name. Pin the tag
v0.5or a commit (revision=) to reproduce.
Quick start
Needs shrew-server โฅ 0.5.0. Validated on one NVIDIA RTX PRO 6000 with a vLLM 0.34 build; the checkpoint loads without a --quantization flag.
vllm serve btbtyler09/shrew-ocr-preview-FP8 --revision v0.5 \
--served-model-name shrew-ocr-preview \
--max-model-len 32768 --max-num-seqs 256 --max-num-batched-tokens 2048 \
--limit-mm-per-prompt '{"image":1}' --no-enable-prefix-caching \
--chat-template-content-format openai
VLM_URL=http://localhost:8000 VLM_MODEL=shrew-ocr-preview shrew serve
curl -X POST localhost:8080/v1/convert -F file=@doc.pdf -F pipeline_mode=structured
- This is the flag set used in validation, plus the repo, revision and served name. The validation run also set a fixed KV-cache memory budget (6 GiB) suited to that card; size it for yours. If your vLLM build asks for
--trust-remote-code(the repo ships its model code), add it. - Not tested: GPUs without native FP8 (for example AMD gfx908), AMD RDNA, or plain transformers. Behaviour on hardware without an FP8 kernel is untested; use the bf16 or GPTQ-8bit repo there.
- Decode with temperature 0.4 and presence_penalty 0.3, never greedy; output budgets are on the main card.
How it was built
- Format: compressed-tensors,
float-quantized, written with compressed-tensors 0.20.1 from the v0.5 bf16 weights: static activation scales, no dynamic quantization, no KV-cache quantization. - Calibration: 512 real document pages drawn from the training split (480 used for calibration, 32 held out for the comparison below), 29 source types, 2.92 M tokens. Each page was run through the checkpoint's processor with the system prompt
structured_extractionand the page image as the only user content; the assistant side was the bf16 model's own sampled answer, teacher forced. No benchmark or held-out evaluation pages were used. Pages did not go through the shrew-server bucket routing or image enhancement. - Written by a short script with compressed-tensors 0.20.1 configuration classes, not with llm-compressor. Weight scales are per output channel (symmetric, max of the row / 448); activation scales are static per layer, from the largest input magnitude seen over the calibration set.
- Every non-weight file (config text, tokenizer, chat template, model code, generation config) is byte-identical to the bf16 repo; only
model.safetensorsandconfig.json(the quantization block) differ.
Measured on this build
One RTX PRO 6000, vLLM 0.34 build, as measured; none of these are OmniDocBench or localization numbers.
| measurement | FP8 | bf16 |
|---|---|---|
| boot, GPU memory | 92 s, 12.7 GB | 82 s, 15.7 GB |
| 24 synthetic pages, numbers read back | 956 / 960 | 956 / 960 |
| 32 held-out pages, text similarity to bf16 (temperature 0, 3,000-token cap), mean / median | 0.921 / 0.961 | bf16 vs bf16 (two boots): 0.958 / 0.995 |
| throughput, pages per second at 16 and 64 concurrent requests (same card, synthetic text pages: compares the arms with each other only) | 1.53 / 1.58 | 1.20 / 1.24 |
FP8 differs from bf16 by more than two bf16 boots differ from each other (worst held-out page 0.70). FP8 with scales computed at run time instead of the static calibrated ones scored the same (mean 0.923), so the difference comes from 8-bit weights and activations, not the calibration. Similarity to bf16 is not accuracy: these pages have no labels, and no benchmark score was measured for this checkpoint. The 3,000-token cap in the similarity comparison is a comparison device, not a serving setting; do not cap outputs in production.
Files
| file | size | for |
|---|---|---|
model.safetensors |
4.9 GB | FP8 language model, bf16 vision and embeddings |
config.json |
small | model definition with the quantization block |
tokenizer, chat template, modeling.py and related files |
small | identical to the bf16 repo |
SHA256SUMS |
small | verify downloads |
Other variants: main card.
Limitations
- Dense broadsheets, low-resolution dense scans and pages the vision tower cannot resolve can still loop or stop early; the guard makes these detectable failures.
- v0.5 splits large tables (and occasionally figures) into stacked strips more often than v0.4.x, has more far false positives on figures and tables, and lower figure recall (bf16 numbers on the main card); the loop edit's effect is not statistically significant on held-out pages.
- OmniDocBench, localization, retrieval, reliability and reading order were not measured on this FP8 build; the bf16 numbers on the main card do not transfer automatically (see Measured on this build).
- Character accuracy on dense Chinese is below Latin print. Cyrillic and Arabic were not evaluated: treat as unsupported. Handwriting has limited evaluation (a separate 300-page check) and no published score.
- Boxes come from model-generated labels: pad them outward when cropping; edges are not pixel-exact.
- One page per request: no cross-page state, so assemble multi-page documents downstream.
Changelog
Per-version detail is on the main card.
- v0.5 (2026-11-14): first FP8 release: W8A8 FP8 (compressed-tensors) of the v0.5 weights, calibrated on 480 real training-split pages.
Attribution
Text from the following OpenStax books (Rice University and named co-authors), each retrieved 2026-10-11 under the Creative Commons Attribution 4.0 International licence (https://creativecommons.org/licenses/by/4.0/), is used as prose in some synthetic document renders, and as original page images in a tables-and-formulas training set. Only the books whose licence was CC BY 4.0 at retrieval were used; the CC BY-NC-SA books were never fetched. No OpenStax photographs or third-party credited figures are used; OpenStax's own graphs, diagrams and tables appear only as printed on the original pages described in the Changes note.
Changes: some rows use passages excerpted and re-typeset into synthetic document renders (tables, layout and surrounding content are our own); other rows are original OpenStax book pages (CC BY 4.0), full pages rasterised at 200 dpi and otherwise unmodified, with the page text, tables, graphs, diagrams and illustrations as printed, each such row carrying its own attribution (book title, licence, licence URL, page); for every row the chunk boundaries and titles, summaries, keywords and descriptions are ours; photographs and figures credited to a third party (courtesy, flickr/wikimedia/getty-style credits) are refused and not used, while OpenStax's own tables and graphs that cite a data source are kept as printed with that citation.
- Algebra and Trigonometry, OpenStax, Rice University (openstax.org/details/books/algebra-and-trigonometry) - CC BY 4.0
- American Government, OpenStax, Rice University (openstax.org/details/books/american-government) - CC BY 4.0
- American Government 2e, OpenStax, Rice University (openstax.org/details/books/american-government-2e) - CC BY 4.0
- American Government 3e, OpenStax, Rice University (openstax.org/details/books/american-government-3e) - CC BY 4.0
- Anatomy and Physiology, OpenStax, Rice University (openstax.org/details/books/anatomy-and-physiology) - CC BY 4.0
- The AP Physics Collection, OpenStax, Rice University (openstax.org/details/books/college-physics-ap-courses) - CC BY 4.0
- Astronomy, OpenStax, Rice University (openstax.org/details/books/astronomy) - CC BY 4.0
- Biology, OpenStax, Rice University (openstax.org/details/books/biology) - CC BY 4.0
- Chemistry, OpenStax, Rice University (openstax.org/details/books/chemistry) - CC BY 4.0
- Chemistry: Atoms First, OpenStax, Rice University (openstax.org/details/books/chemistry-atoms-first) - CC BY 4.0
- College Algebra, OpenStax, Rice University (openstax.org/details/books/college-algebra) - CC BY 4.0
- College Algebra with Corequisite Support, OpenStax, Rice University (openstax.org/details/books/college-algebra-corequisite-support) - CC BY 4.0
- College Physics, OpenStax, Rice University (openstax.org/details/books/college-physics) - CC BY 4.0
- Elementary Algebra, OpenStax, Rice University (openstax.org/details/books/elementary-algebra) - CC BY 4.0
- Intermediate Algebra, OpenStax, Rice University (openstax.org/details/books/intermediate-algebra) - CC BY 4.0
- Introduction to Intellectual Property, OpenStax, Rice University (openstax.org/details/books/introduction-intellectual-property) - CC BY 4.0
- Introduction to Sociology 2e, OpenStax, Rice University (openstax.org/details/books/introduction-sociology-2e) - CC BY 4.0
- Introductory Business Statistics, OpenStax, Rice University (openstax.org/details/books/introductory-business-statistics) - CC BY 4.0
- Introductory Statistics, OpenStax, Rice University (openstax.org/details/books/introductory-statistics) - CC BY 4.0
- Physics, OpenStax, Rice University (openstax.org/details/books/physics) - CC BY 4.0
- Prealgebra, OpenStax, Rice University (openstax.org/details/books/prealgebra) - CC BY 4.0
- Precalculus, OpenStax, Rice University (openstax.org/details/books/precalculus) - CC BY 4.0
- Principles of Economics, OpenStax, Rice University (openstax.org/details/books/principles-economics) - CC BY 4.0
- Principles of Economics 2e, OpenStax, Rice University (openstax.org/details/books/principles-economics-2e) - CC BY 4.0
- Principles of Macroeconomics, OpenStax, Rice University (openstax.org/details/books/principles-macroeconomics) - CC BY 4.0
- Principles of Macroeconomics 2e, OpenStax, Rice University (openstax.org/details/books/principles-macroeconomics-2e) - CC BY 4.0
- Principles of Macroeconomics for APยฎ Courses, OpenStax, Rice University (openstax.org/details/books/principles-macroeconomics-ap-courses) - CC BY 4.0
- Principles of Macroeconomics for APยฎ Courses 2e, OpenStax, Rice University (openstax.org/details/books/principles-macroeconomics-ap-courses-2e) - CC BY 4.0
- Principles of Microeconomics, OpenStax, Rice University (openstax.org/details/books/principles-microeconomics) - CC BY 4.0
- Principles of Microeconomics 2e, OpenStax, Rice University (openstax.org/details/books/principles-microeconomics-2e) - CC BY 4.0
- Principles of Microeconomics for APยฎ Courses, OpenStax, Rice University (openstax.org/details/books/principles-microeconomics-ap-courses) - CC BY 4.0
- Principles of Microeconomics for APยฎ Courses 2e, OpenStax, Rice University (openstax.org/details/books/principles-microeconomics-ap-courses-2e) - CC BY 4.0
- Psychology, OpenStax, Rice University (openstax.org/details/books/psychology) - CC BY 4.0
- Statistics, OpenStax, Rice University (openstax.org/details/books/statistics) - CC BY 4.0
- Fรญsica universitaria volumen 1, OpenStax, Rice University (openstax.org/details/books/fรญsica-universitaria-volumen-1) - CC BY 4.0
License and citation
Apache-2.0. Base: ibm-granite/granite-vision-4.1-4b (Apache 2.0).
@misc{shrew-ocr-preview,
title = {shrew-ocr-preview: page image to structured JSON},
author = {btbtyler09},
year = {2026},
howpublished = {\url{https://huggingface.co/btbtyler09/shrew-ocr-preview}},
note = {Version v0.5}
}
- Downloads last month
- 64
Model tree for btbtyler09/shrew-ocr-preview-FP8
Base model
ibm-granite/granite-vision-4.1-4b