| --- |
| license: apache-2.0 |
| base_model: jbarrow/FFDetr |
| pipeline_tag: object-detection |
| tags: |
| - rf-detr |
| - form-fields |
| - document-ai |
| - commonforms |
| --- |
| |
| # FFDetr — filled-form fine-tune |
|
|
| Fine-tune of [jbarrow/FFDetr](https://huggingface.co/jbarrow/FFDetr) (RF-DETR Medium) that detects |
| form fields **even when they already contain values** — typed text, handwriting, checkbox marks, |
| signatures — on both born-digital and scanned pages. The base model only detects blank fields |
| reliably; on filled fields its recall collapses to near zero. |
|
|
| ## Classes |
|
|
| Same head layout and index order as the base model: |
|
|
| | id | class | |
| |----|-------| |
| | 0 | Text (TextBox) | |
| | 1 | CheckBox (ChoiceButton) | |
| | 2 | Signature | |
|
|
| ## Results |
|
|
| Evaluated on a held-out 300-image synthetic validation set (including 60 field-less negative |
| pages) built from the CommonForms `valid` split — document-disjoint from training, with disjoint |
| fill-content generator profiles. IoU 0.5, confidence 0.5: |
|
|
| | Slice | Base P / R / F1 | This model P / R / F1 | |
| |---|---|---| |
| | filled_flat | 0.34 / 0.05 / 0.09 | **0.77 / 0.71 / 0.74** | |
| | filled_scan | 0.62 / 0.02 / 0.04 | **0.96 / 0.86 / 0.91** | |
| | hybrid_flat | 0.70 / 0.42 / 0.53 | **0.80 / 0.75 / 0.78** | |
| | blank_flat | 0.90 / 0.78 / 0.83 | 0.89 / 0.76 / 0.82 | |
| | blank_scan | 0.87 / 0.82 / 0.84 | 0.92 / 0.71 / 0.80 | |
| | **overall** | 0.84 / 0.53 / 0.65 | **0.87 / 0.77 / 0.82** | |
| |
| - Overall AP@0.5: 0.635 (base) → **0.815** |
| - False positives on field-less pages: 1.47 → **1.32** per negative page |
| - On pages that do contain fields, absolute FP rises 438 → 502 (+15%) because the model now |
| makes far more detections there — true positives rise 2669 → 3833 (+44%) — so the share of |
| false boxes among all detections **falls from 16.5% to 13.2%**, and precision improves on |
| every slice except blank_flat (flat: 0.90 → 0.89). The base model's low absolute FP on |
| filled pages is an artifact of it barely detecting anything there. |
| - Inference latency unchanged (~57 ms/img on A100 at resolution 1024; same architecture, |
| weights-only checkpoint) |
|
|
| On real filled documents (scanned bank forms with handwriting, typewriter-filled government |
| forms), the fine-tune matches the base model on strongly-outlined blank fields and additionally |
| detects value-bearing and graphics-obstructed fields the base model misses, without producing |
| detections on field-less pages. |
|
|
| ## Training data |
|
|
| 8,000 images derived from 4,000 pages of [jbarrow/CommonForms](https://huggingface.co/datasets/jbarrow/CommonForms) |
| (3,200 pages with fields + 800 negatives, revision-pinned). Every page is paired with a synthetic |
| augmented counterpart: values rendered into field boxes (multilingual text incl. CJK/Arabic/Cyrillic, |
| choice marks, signature strokes; partial-fill variants) and/or scan degradation |
| (rotation, blur, contrast shift, noise, JPEG artifacts). |
|
|
| ## Recipe |
|
|
| Initialized from the base FFDetr checkpoint; detection head reinitialized for 3 classes. |
| 8 epochs, lr 1e-5, batch 4 with grad-accum 4 (effective 16), resolution 1024, on 1× A100 40GB. |
|
|
| ## Usage |
|
|
| Pin a revision in production so a re-upload can never silently change your model: |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| from commonforms.inference import FFDetrDetector |
| |
| path = hf_hub_download( |
| "pdf-net/ffdetr-filled", |
| "FFDetr.pth", |
| revision="e4fc1a8654915fe2f9bae63327062bc15711ecd6", |
| ) |
| detector = FFDetrDetector(path) |
| ``` |
|
|
| The checkpoint is a drop-in replacement for the base model: same class indices, same input |
| resolution, same inference code path. |
|
|
| ## Known limitations |
|
|
| - Detected boxes follow the CommonForms field-rect convention (the fillable area). Ink written |
| outside that area — on or below an underline, spilling past the box — may not be fully covered |
| by the box. A future revision trains with overflowing fills and ink-union target boxes. |
| - `blank_scan` recall is ~0.11 below the base model at conf 0.5 (precision is higher); expected |
| to recover with the larger final training run. |
|
|
| ## Status |
|
|
| Interim checkpoint from an 8k-image pilot. A final fine-tune on a larger dataset is planned and |
| will be published here as a new revision — pin a specific commit revision in production. |
|
|
| License: Apache-2.0, same as the base model (jbarrow/FFDetr), the CommonForms dataset, and the |
| rfdetr training library. |
