CaliDoc: FUNSD

LayoutLMv3-Large fine-tuned for key information extraction on FUNSD, plus the CaliDoc calibration artifacts fitted for this exact checkpoint. Paper: CaliDoc: Calibrated Field-Level Risk and Annotation Auditing for Document Information Extraction, WACV 2027 (Evaluations and Datasets Track). Code: https://github.com/MHHamdan/CaliDoc · Supplement: PDF · Collection: CaliDoc — WACV 2027

Token-level reliability is not complete-field reliability. CaliDoc calibrates the object a downstream system consumes: the complete field.

Model Description

Extractor. LayoutLMv3ForTokenClassification (microsoft/layoutlmv3-large), 7 BIO labels, trained on noisy scanned administrative forms (FUNSD), labels HEADER, QUESTION, ANSWER and O. This is a dataset-specific checkpoint, and CaliDoc does not provide a universal model. Each dataset has its own repository in the collection.
Checkpoint. This repository contains the seed-42 canonical checkpoint, the one the paper uses for every ablation, SpanCRC and audit result. Headline WACV results report mean ± standard deviation over three training seeds (42, 123, 456). Those aggregates are not metrics of this single checkpoint.
Calibration (post hoc, no retraining). Temperature scaling followed by field-conditional SpanCRC, a conformal risk control over max-token span scores. The neural extractor and the calibration are separate: the weights are a standard Transformers model, and the thresholds in calibration/ turn its probabilities into calibrated label sets.

Intended Use

Research on uncertainty quantification, conformal prediction and annotation auditing for document information extraction.
Reproducing and extending the WACV 2027 results.
Prototyping human-in-the-loop review routing on documents from the same distribution as FUNSD.

Out-of-Scope Use

Commercial use: the weights inherit CC BY-NC-SA 4.0 from LayoutLMv3.
Documents from other domains, layouts, languages or OCR engines. The calibration guarantee assumes exchangeability with the calibration documents and does not transfer to out-of-domain inputs.
Autonomous decisions in consequential workflows (finance, legal, medical, identity). CaliDoc is meant to decide what a human should check, not to replace that check.
Treating calibrated-set status as a per-field probability of correctness.

How CaliDoc Works

Input: a document image, plus OCR words and word boxes on LayoutLMv3's 0–1000 grid. OCR is not included: the paper uses the dataset-provided OCR.
Extractor: LayoutLMv3 token classification. Each word is scored by its first sub-word, and probabilities are softmax(logits / T) with T = 1.2598.
Calibration: the calibrated set of word i is C(x_i) = { y : 1 − p(y|x_i) ≤ λ_type(y) }. A gold field is fully covered exactly when its worst word clears the threshold, which makes this a max-token complete-field score.
Reliability: taking the document as the exchangeable unit, SpanCRC bounds the expected per-document fraction of missed gold fields by α = 0.1. The bound holds on fresh documents exchangeable with the calibration documents that contain at least one gold field.
Audit (corpus level): at the pooled threshold, a boundary-escape test asks whether threshold escapes concentrate at field edges, which is a symptom of alignment and annotation errors. It runs over a calibration corpus with Holm control and a document-blocked bootstrap. It is a dataset diagnostic, not a per-document error detector, and it is not part of single-document inference.

FUNSD thresholds are large (pooled λ = 0.951), because form fields are long and the max-token score is conservative for long spans. Most fields therefore get calibrated sets with more than one label and are routed to review. The calibration split is small (35 forms), so the document-level bound is loose.

Quick Start

pip install "calidoc @ git+https://github.com/MHHamdan/CaliDoc"

Plain Transformers (the extractor only):

from transformers import AutoModelForTokenClassification, AutoProcessor

model_id = "MHamdan/calidoc-layoutlmv3-large-funsd"
processor = AutoProcessor.from_pretrained(model_id, apply_ocr=False)
model = AutoModelForTokenClassification.from_pretrained(model_id)
print(model.config.id2label)

CaliDoc calibrated extraction, given an image and an OCR JSON {"words": [...], "boxes": [[x0, y0, x1, y1], ...], "box_format": "pixel" | "normalized_1000"}:

import json
from PIL import Image
from calidoc.inference import CaliDocPredictor, normalize_boxes

predictor = CaliDocPredictor.from_pretrained("MHamdan/calidoc-layoutlmv3-large-funsd")
image = Image.open("document.png").convert("RGB")
ocr = json.load(open("ocr.json"))
boxes = ocr["boxes"] if ocr["box_format"] == "normalized_1000" else normalize_boxes(ocr["boxes"], *image.size)
result = predictor.predict(image, ocr["words"], boxes)
for field in result["fields"]:
    print(field["type"], "|", field["text"], "|", field["status"], field["set_sizes"])
print(result["summary"]["route"])

or from the command line:

python -m calidoc.predict --model MHamdan/calidoc-layoutlmv3-large-funsd --image document.png --ocr ocr.json --out prediction.json

This model's training data is not cleared for redistribution, so no example image is shipped. The GitHub repository has a CORD example with the same input format, and assets/examples/funsd/ holds this model's output on a selected test document with the text removed.

Each predicted field gets one of these statuses:

status	meaning	suggested action
`covered`	every word's calibrated set is exactly the predicted label	pass downstream
`ambiguous_review`	the predicted label is in every set, but some set also admits other labels	validate / review
`conflict_review`	some word's set is non-empty but excludes the predicted label	review
`empty_abstain`	some word's set is empty	abstain / review

unassigned_field_candidates lists words predicted O whose set still admits a field label, which are possible missed fields. These statuses are properties of calibrated prediction sets, not confidence scores.

Calibration Artifacts

file	content
`calidoc_config.json`	α = 0.1, m0 = 25, T, pooled λ, supported/fallback field types, released-checkpoint test metrics
`calibration/temperature.json`	T = 1.259807, fitted by NLL on the `es_holdout` early-stopping split
`calibration/pooled_threshold.json`	pooled SpanCRC λ = 0.951165 and the token-marginal q̂ for comparison
`calibration/field_thresholds.json`	λ per field type with its calibration support
`calibration/split_manifest.json`	calibration document IDs (identifiers only, no data)

The thresholds were fitted on calibration split seed 42 (the paper's first calibration unit) and match the committed paper artifacts exactly. Field types with fewer than m0 = 25 calibration spans fall back to the pooled λ, so they inherit only the pooled document-level bound and get no per-type guarantee.

field type	λ	calibration spans	threshold
`ANSWER`	0.9563	476	own threshold
`HEADER`	0.9686	82	own threshold
`QUESTION`	0.9351	717	own threshold

Evaluation

Paper: three training seeds (headline, mean ± s.d. over seeds 42/123/456, full test split):

metric	value
entity F1	82.39 ± 0.65

Paper: CaliDoc on this seed-42 checkpoint (90% target, α = 0.10, mean over three calibration splits; the paper's SpanCRC table):

	token-marginal threshold	field-conditional SpanCRC
realized field-miss risk (the bounded quantity)	0.054	0.058
full-span coverage (empirical)	0.950	0.944
average set size	1.71	2.10

This released checkpoint and calibration (computed from the release artifacts):

metric	value
test entity F1 / precision / recall	82.70 / 81.13 / 84.34 (50 test documents)
test ECE (after temperature)	5.49%
SpanCRC on the evaluation side of the released calibration split: field-miss risk / span coverage / set size	0.046 / 0.951 / 2.34 (698 gold spans)
same split, token-marginal threshold: token / span coverage	0.950 / 0.954

The released weights reproduce the paper pipeline's cached probabilities to within 3e-5 and the recorded test F1 exactly (see docs/release/HF_MODEL_ARTIFACT_AUDIT.md in the GitHub repository).

Limitations

Dataset and domain specific. One checkpoint per benchmark. Results do not transfer to other document types, and the guarantee assumes exchangeability with the calibration documents.
English-script benchmarks only (CORD receipts are Indonesian with mostly Latin-script menu text). One backbone family (LayoutLMv3).
OCR dependence. Inputs are OCR words and boxes, and the paper uses dataset-provided OCR. A different OCR engine changes tokenization and boxes and voids the calibration.
Max-score conservatism. The worst word decides a field, so long fields (FUNSD answers, SROIE addresses) need larger thresholds and larger sets.
Support-limited field conditioning. Types below m0 = 25 calibration spans fall back to the pooled threshold and receive no per-type guarantee (most of CORD).
Short-span audit blind spot. The boundary-escape test cannot see spans of two words or fewer, so a clean result on short fields means untested.
Audit independence assumption. The randomization null assumes independence across spans, which the method states rather than derives. Findings are claimed only when a document-blocked bootstrap agrees.
The guarantee is in expectation, over calibration draws and fresh documents. A single document, or a single calibration split, can exceed α.
Not a replacement for human review in consequential workflows.

License

Model weights: CC BY-NC-SA 4.0, inherited from microsoft/layoutlmv3-large (CC BY-NC-SA 4.0). This repository fine-tunes that model for token classification and adds calibration files. Non-commercial use only, with attribution, under the same license.
CaliDoc source code: Apache-2.0, in the GitHub repository.
Training data: FUNSD is distributed under the FUNSD License for non-commercial research and educational use; its images (from RVL-CDIP) remain under third-party copyright. Lineage used: the official FUNSD archive (https://guillaumejaume.github.io/FUNSD/), 149/50 train/test, with 15 training forms held out for early stopping and temperature. Cite Jaume, Ekenel, Thiran, FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents, ICDAR-OST 2019. No dataset images or annotations are included in this repository.

Citation

WACV 2027 proceedings metadata (pages, DOI) will be added once published.

@inproceedings{hamdan2027calidoc,
  author    = {Hamdan, Mohammed and Cheriet, Mohamed},
  title     = {{CaliDoc}: Calibrated Field-Level Risk and Annotation Auditing for Document Information Extraction},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

@inproceedings{huang2022layoutlmv3,
  author    = {Huang, Yupan and Lv, Tengchao and Cui, Lei and Lu, Yutong and Wei, Furu},
  title     = {{LayoutLMv3}: Pre-training for Document {AI} with Unified Text and Image Masking},
  booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},
  year      = {2022}
}