Lumen Analytics invoices — an extraction evaluation set

Fictional. Every company, person, address, VAT id and IBAN here is invented; every page says so along its edge. Generated for the Docling × Confluent Cloud demo (example 04, examples/04-invoice-extraction in docling-project/docling-confluent-cloud), and usable on its own to evaluate structured extraction from invoices.

174 supplier invoices of a fictional Munich company, Lumen Analytics GmbH, dated 2025-12-01 to 2026-06-19: 17 vendors, 7 layouts, 139 in EUR, 21 in USD, 7 in GBP, 7 in CHF; 10 scans (rasterized, tilted, noisy, no text layer) and 8 two-page invoices; 749 line items in all. Each comes with its ground truth — what is printed on it, field by field and line by line — because the generator drew it from that.

Files

This is a private Hugging Face Storage Bucket, docling-project/docling-confluent-ex04:

hf buckets sync hf://buckets/docling-project/docling-confluent-ex04 ./lumen-invoices

The invoices

The variety is the point: the same fields printed seven ways — German and Swiss number formats, US and European dates, net/tax/total blocks with one, two or three VAT rates, a two-page cloud bill whose metered lines spill over, a hotel folio whose lines are nights. One template seventeen times would be template matching, not extraction.

Ground truth

ground_truth/manifest.jsonl, one JSON object per invoice:

{"doc_id": "lumen-inv-0001", "file": "invoices/lumen-inv-0001.pdf", "layout": "uk_legal",
 "vendor_key": "kestrel", "scanned": false, "pages": 1,
 "department": "Finance", "category": "Professional Services", "submitted_at": "2026-01-05T08:20:51Z",
 "truth": {"vendor_name": "Kestrel & Finch LLP", "invoice_number": "KF/2025/0351",
           "invoice_date": "2025-12-30", "currency": "GBP", "total_amount": 7510.5, "…": "…",
           "line_items": [{"description": "…", "quantity": 6.3, "unit_price": 395.0, "amount": 2488.5,
                           "page": 1}]}}

truth holds vendor_name, vendor_vat_id, vendor_address, customer_name, invoice_number, invoice_date, due_date, currency, subtotal, tax_amount, total_amount, iban and line_items (description, quantity, unit_price, amount, and the page it is printed on). Amounts are numbers in the invoice's currency, dates ISO. department, category and submitted_at are the demo's (who submitted the invoice, when); they are not on the page.

What is asked: the extraction target

target/extraction_target.json is a Docling ExtractionTarget (docling-project/docling#4218): output_schema (the contract, validated per page), template (an example_json: one made-up invoice shown as an example answer) and instructions. It asks for a subset of the truth:

LineItem:

Evaluating

Per invoice, the pages' answers are merged — header fields from the first page that has them, totals from the last, line items from all — and compared with truth:

ok is the demo pipeline's own verdict on the merged record: vendor, date, currency and total present, and subtotal + tax = total. The reference implementation is scripts/extract.py score with invoice_udf.invoice.build. Line items are counted, not matched line by line — scores.jsonl and results.jsonl have what is needed for a finer measure.

Runs

2026-10-05_granite-vision-v4

174 invoices, Docling for IBM watsonx (staging), POST /v1/extract/source/async, preset granite_vision_v4_kserve, extracted 2026-10-05; median 10.34 s per invoice. Files: extractions/docling/2026-10-05_granite-vision-v4/.

Things worth knowing about these runs:

• Multi-page invoices: page 1 of each carries 20+ line items, and the service's model server timed out on it (20 s) on every attempt — the page has an error and no data, so dates and line counts miss. A service limit at the time, not the model's answer.
• Vendor names miss where the model "corrects" a name (Volterra for Voltara, Biurebedarf for Bürobedarf) or drops a suffix (, Inc.).
• Dates come back as printed (20.02.2026, 30 December 2025), despite the schema's format: date and the instructions; Docling does not validate format. They are scored after parsing.
• A page whose answer breaks the schema has extracted_data: null in Docling's result and the answer only in raw_text; the scoring reads it from there.

Generated by scripts/dataset.py on 2026-10-05; edit that, not this file.