| --- |
| license: other |
| license_name: nutrient-commercial |
| pipeline_tag: image-to-text |
| language: |
| - multilingual |
| tags: |
| - document-ai |
| - form-understanding |
| - multilingual |
| - object-detection |
| - vision-language-model |
| datasets: |
| - nutrientdocs/form-field-vlm-benchmark |
| metrics: |
| - f1 |
| --- |
| |
| # form-field-vlm · _commercial_ |
|
|
| **Detect and understand interactive form fields from a page image.** The Nutrient system returns each |
| field's bounding box, type, text label, and radio-group identity as constrained JSON. The production |
| system can combine a document-specialized VLM with FF-DETR. The Nutrient VLM performs the semantic work: |
| field detection, fine field typing, text labeling, and radio-group linking. FF-DETR is an optional |
| localization component that supplies object boxes only; it does not label fields or link radio buttons. |
|
|
| The runnable weights are commercial and are not downloadable from this repository. This public page is |
| the product specification and scorecard. |
|
|
| - 🎯 **Try it:** [form-field-vlm-demo](https://huggingface.co/spaces/nutrientdocs/form-field-vlm-demo) |
| - 🏆 **Leaderboard:** [form-field-vlm-leaderboard](https://huggingface.co/spaces/nutrientdocs/form-field-vlm-leaderboard) |
| - 📊 **Benchmark:** [form-field-vlm-benchmark](https://huggingface.co/datasets/nutrientdocs/form-field-vlm-benchmark) |
|
|
| ## Headline result |
|
|
| Models are ranked by **Typed F1 at IoU 0.5**. A prediction must overlap the gold widget by at |
| least 0.5 IoU and return the correct fine field type. This strict threshold rewards usable, tightly |
| grounded fields; IoU 0.2 is also reported to expose models that find the right area but draw loose boxes. |
|
|
|  |
|
|
| | System | Typed F1 `.5 / .2` | Box F1 `.5 / .2` | Box Recall `.5 / .2` | sec/page | |
| |---|---:|---:|---:|---:| |
| | **Nutrient Hybrid** | **0.477 / 0.516** | **0.594 / 0.671** | **0.790 / 0.892** | 3.24 | |
| | **Nutrient VLM** | **0.406 / 0.478** | **0.514 / 0.646** | **0.551 / 0.692** | 3.04 | |
| | **GPT-5.6 Sol** | 0.357 / **0.521** | 0.403 / 0.580 | 0.364 / 0.524 | — | |
|
|
| The evaluation contains 100 held-out clean form pages and 701 annotated fields. The Nutrient hybrid |
| achieves **0.477 F1 at IoU 0.5**. The Nutrient VLM alone reaches **0.406**, ahead of the strongest cloud |
| VLM in this evaluation, **GPT-5.6 Sol** (**0.357**). The hybrid also narrows the strict-to-loose Typed F1 gap from |
| **0.072** for the VLM alone to **0.039**, evidence that FF-DETR localization converts more approximately |
| correct detections into strict IoU 0.5 matches. |
|
|
| Cells show `IoU 0.5 / IoU 0.2`; ranking uses only the first Typed F1 value. Typed F1 requires the correct |
| fine field type. Box F1 measures localization while penalizing missed and extra boxes. Box Recall measures |
| field coverage without penalizing extra boxes. IoU 0.2 is diagnostic for approximately correct but loose |
| localization. |
|
|
| ## Detailed result |
|
|
| Each detail cell is `Typed F1 @ IoU 0.5 / Typed F1 @ IoU 0.2`. Seconds per page are shown only for models measured in |
| the same local batch environment; provider batch APIs do not provide comparable request latency. |
|
|
|  |
|
|
| The benchmark is intentionally small for the first release. Treat it as a transparent, reproducible |
| comparison set, not a complete estimate of every form distribution. The leaderboard exposes additional |
| precision, recall, density, return-count, and Box Recall breakdowns. |
|
|
| Cloud systems are labeled with the model generation returned by the evaluated batch: GPT-5.6; |
| Gemini 3.1 Flash-Lite, Gemini 3.5 Flash, and Gemini 3.1 Pro Preview; and Claude 4.5/4.8/5 as applicable. |
| Leaderboard details retain both the resolved Gemini `model_version` and the submitted `gemini-*-latest` |
| alias, so future alias changes cannot alter this benchmark snapshot. |
|
|
| ## Output |
|
|
| ```json |
| [ |
| { |
| "box": [82, 164, 418, 205], |
| "type": "text", |
| "label": "Account number", |
| "group_id": null |
| } |
| ] |
| ``` |
|
|
| Coordinates are `[x0, y0, x1, y1]` on a 0–1000 page grid. Supported types are `text`, |
| `choice_checkbox`, `choice_radio`, `choice_select`, and `signature`. |
|
|
| ## Intended use and limits |
|
|
| - Empty digital forms are the supported release target. |
| - Multilingual forms and field labels are supported. |
| - Very dense pages can require adaptive tiling and longer generation budgets. |
|
|
| ## License and attribution |
|
|
| The Nutrient VLM weights are offered under a commercial Nutrient license. The evaluation set is public |
| and reproducible through |
| [`nutrientdocs/form-field-vlm-benchmark`](https://huggingface.co/datasets/nutrientdocs/form-field-vlm-benchmark). |
|
|
| The hybrid uses <a href="https://huggingface.co/jbarrow/FFDetr" target="_blank" rel="noopener"><strong>FF-DETR / CommonForms</strong></a> |
| by jbarrow, Apache-2.0, for object detection and box localization. Benchmark pages derive from CommonForms, |
| CC-BY-4.0. |
|
|
| > ### 📩 Get access |
| > |
| > `form-field-vlm` is commercial and its weights are not downloadable here. To evaluate or deploy it |
| > on-prem, with documents remaining in your infrastructure, **contact Nutrient: |
| > [nutrient.io/contact-sales](https://www.nutrient.io/contact-sales/).** |
|
|
| ## About the author |
|
|
| <a href="https://nutrient.io/"> |
| <img src="https://avatars2.githubusercontent.com/u/1527679?v=3&s=200" height="80" /> |
| </a> |
|
|
| This project is maintained and funded by [Nutrient](https://nutrient.io/) - The deterministic document infrastructure enterprises run their highest-stakes workflows on: replayable output, clear exceptions, and full audit trails on the messy, regulated documents where AI alone breaks. |
|
|
