README.md

---
license: other
license_name: nutrient-commercial
pipeline_tag: zero-shot-image-classification
language:
  - en
tags:
  - zero-shot-image-classification
  - image-classification
  - document-ai
  - document-image-classification
  - open-vocabulary
datasets:
  - nutrientdocs/document-classification-benchmark
metrics:
  - accuracy
---

# document-classification-v2 · _commercial_

**Classify any document image against labels you choose at runtime.** `document-classification-v2` is an
open-vocabulary, zero-shot document classifier: supply one label + a short description per class at
inference, and it scores a document page against *any* such set — invoices, letters, memos, forms,
scientific articles, resumes, and whatever label you type next. No fixed class list, no per-class training.

It is the document specialist in a field of generalists. General open-vocab models were trained on web
photos; this one is built for document pages — and it shows on the
[leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard). **Weights are
commercial** (not downloadable here); this page is a spec + scorecard.

- 🎯 **Try it:** [document-classification-demo](https://huggingface.co/spaces/nutrientdocs/document-classification-demo)
- 🏆 **Leaderboard:** [document-classification-leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard)
- 📊 **Benchmark:** [document-classification-benchmark](https://huggingface.co/datasets/nutrientdocs/document-classification-benchmark)

## Results

Macro-F1, zero-shot, on the held-out benchmark — higher is better. The flagship vs the previously-shipped
in-house model and a cloud frontier VLM reference.

<!-- RESULTS-TABLE:START — filled from leaderboard/leaderboard.json by the scoring run -->
| Benchmark (macro-F1) | `document-classification-v2` | best generalist | best cloud VLM |
| --- | ---: | ---: | ---: |
| **DocLayNet** (page types) | **0.97** | 0.63 | 0.83 |
| **Forms** | **1.00** | 0.11 | 1.00 |
| **Tobacco** (doc types) | 0.74 | 0.43 | **0.85** |
| **OOD** (unseen doc types) | **0.95** | — | — |
| **OOV** (synonym wording) | **0.83** | — | — |

Latency (fused image+text, A40): **~4.9 pages/s** on text-heavy pages; the image branch alone runs ~8 docs/s
when no OCR is present. The open-weight
[`v1`](https://huggingface.co/nutrientdocs/document-classification-v1) runs ~5.7 pages/s fused on the same GPU.
<!-- RESULTS-TABLE:END -->

Every model — ours and cloud — is scored by the same open macro-F1 scorer; full per-model ranking on the
[leaderboard](https://huggingface.co/spaces/nutrientdocs/document-classification-leaderboard).

- **Matches/leads the cloud on visual document-type tracks** (DocLayNet, Forms) at **zero per-request API
  cost** — and runs on a single GPU.
- **Trails on Tobacco** (0.74 vs 0.85): that track rewards *reading* fine header text (memo vs letter vs
  email) — a large VLM reads it; an embedding model can't.
- **OOD robustness:** on document types absent from training (invoices, handwriting, charts, tables), 0.95
  macro-F1. **Open-vocab (OOV):** under never-seen synonym label wording, 0.83 — matches the *concept*, not
  your exact string. (Cloud VLMs train on ~all data, so these held-out axes aren't reported for them.)

## Intended use & limits

- **Use it for:** zero-shot classification of document page images in a free-label setting — the caller
  supplies the candidate labels (and optional descriptions). Multi-page documents supported; optional page
  OCR sharpens fine-grained form / tax-code distinctions.
- **Limits:** optimized for **document** imagery; English label strings are the primary target. Scores are
  per-label match probabilities (independent per label), not a softmax across the set.

## License & data

The model **weights** are offered under a commercial Nutrient license — deployed on-prem, so your documents
never leave your infrastructure. The training set is not redistributed. Evaluation runs on the held-out
[document-classification-benchmark](https://huggingface.co/datasets/nutrientdocs/document-classification-benchmark).

> ### 📩 Get access
>
> `document-classification-v2` is commercial and its weights are not downloadable here. To run it on-prem —
> open-vocabulary, calibrated, private — **contact Nutrient:
> [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).**

## About the author

<a href="https://nutrient.io/">
  <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" />
</a>

This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks.