Hugging Face's logo

nutrientdocs
/
doc-img-classification

Zero-Shot Image Classification
English
image-classification
document-ai
document-image-classification
open-vocabulary
clip
Model card Files
xet
 Community
doc-img-classification
File size: 5,081 Bytes
2f41dd7
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
---
license: other
license_name: nutrient-commercial
pipeline_tag: zero-shot-image-classification
language:
  - en
tags:
  - zero-shot-image-classification
  - image-classification
  - document-ai
  - document-image-classification
  - open-vocabulary
  - clip
datasets:
  - nutrientdocs/doc-openvocab-benchmark
metrics:
  - accuracy
---

# doc-img-classification · _commercial_

**Classify any document or figure image against labels you choose at runtime.** `doc-img-classification` is a
document-adapted dual-encoder (image ↔ text) that scores an image against an *arbitrary* set of text
labels and returns the best match — invoices, resumes, scientific papers, financial reports, receipts,
charts, signatures, screenshots, and whatever label you type next. No fixed class list, no retraining to
add a category: it is **open-vocabulary**.

It is the document specialist in a field of generalists. General open-vocab models (CLIP, SigLIP) were
trained on web photos; this one is adapted to documents and figures — and it shows on the
[leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-leaderboard). **Weights are
commercial** (not downloadable here); this page is a spec + scorecard.

- 🎯 **Try it:** [doc-openvocab-demo](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-demo)
- 🏆 **Leaderboard:** [doc-openvocab-leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-leaderboard)
- 📊 **Benchmark:** [doc-openvocab-benchmark](https://huggingface.co/datasets/nutrientdocs/doc-openvocab-benchmark)

## Results

On the open [doc-openvocab-benchmark](https://huggingface.co/datasets/nutrientdocs/doc-openvocab-benchmark)
— document & figure images scored against a ~48-label open vocabulary — against the strongest general
open-vocab image classifiers. Top-1 accuracy; higher is better.

<!-- RESULTS-TABLE:START — filled from leaderboard/leaderboard.json by the scoring run (docs/PUBLISH.md Phase B) -->
| Track | `doc-img-classification` | OpenAI CLIP L/14 | LAION CLIP-H/14 | SigLIP2 SO400M |
| --- | ---: | ---: | ---: | ---: |
| **Broad** (48-label open vocab) | **.880** | .726 | .654 | .801 |
| **Specialized** (document categories) | **.712** | .463 | .250 | .536 |
| **Synonym** (paraphrase robustness) | .730 | **.738** | .672 | .689 |
<!-- RESULTS-TABLE:END -->

The point of the benchmark: this is the open-vocabulary task — labels supplied at inference, not baked
into a head — which a fixed-label classifier structurally cannot do. Among models that *can* do it, the
document-adapted one leads on document and figure imagery. Full ranking on the
[leaderboard](https://huggingface.co/spaces/nutrientdocs/doc-openvocab-leaderboard).

**Concepts, not strings.** On the *synonym* track the same images are scored against held-out paraphrases
of every label ("invoice" → "bill", "manuals" → "user guide"). Accuracy holds — the model matches the
*concept*, so your exact wording doesn't have to match ours.

## Calibrating the score

Contrastive training maximizes *ranking*, which can leave the raw similarity overconfident. For a score
you can threshold, apply **temperature scaling** — divide the logits by a fitted `T` before the sigmoid.
It is monotonic, so ranking/accuracy is untouched and only the confidence values are repaired. On our
serving distribution the fitted temperature drives pairwise calibration error from **0.027 → 0.002**, and
it ships folded directly into the exported graph. Re-fit `T` if your input pipeline changes.

## Intended use & limits

- **Use it for:** zero-shot classification of document page images and figure/zone crops in a free-label
  setting — the caller supplies the candidate labels. Built for the Nutrient Maestro SDK
  image-classification contract (a single combined ONNX graph, on-prem).
- **Limits:** optimized for **document and figure** imagery; general natural-image zero-shot ability is
  preserved but not the focus. English label strings are the primary target.

## License & data

The model **weights** are offered under a commercial Nutrient license — deployed on-prem, so your
documents never leave your infrastructure. The training set is not redistributed. The **evaluation** data
is public and reproducible:
[doc-openvocab-benchmark](https://huggingface.co/datasets/nutrientdocs/doc-openvocab-benchmark)
(CC-BY-SA-4.0, test-only).

> ### 📩 Get access
>
> `doc-img-classification` is commercial and its weights are not downloadable here. To run it on-prem —
> open-vocabulary, calibrated, private — **contact Nutrient:
> [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).**

## About the author

<a href="https://nutrient.io/">
  <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" />
</a>

This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.