README.md

metadata

license: apache-2.0
language:
  - en
  - ko
library_name: onnxruntime
pipeline_tag: image-to-text
tags:
  - onnx
  - onnxruntime
  - receipt
  - key-information-extraction
  - document-understanding

tiny-receipt-reader-digit-slots-2m

A question-free receipt reader. One small CNN reads the image once and emits a fixed record; questions are answered afterwards by regular expressions and array indexing.

There is no transformer, no decoder, no autoregressive loop, no tokenizer, and no question encoder. The graph has one input: the image.

image ──▶ CNN ──▶ slots 0..11  phone digits
                  slots 12..15 street-number digits

question ──▶ regex ──▶ (family, op) ──▶ index the record ──▶ answer

Several questions about one receipt cost one forward pass, because the record does not depend on the question.

Files

model.onnx           FP32 graph
model_int8.onnx      static W8A8 (QDQ, U8S8) graph
config.json          input contract and slot layout
manifest.json        graph contract and the full export verification report
question_router.py   regex router, required to answer a question
inference.py         onnxruntime runtime
examples/            two synthetic receipts and a runnable check
eval/                held-out reports and runtime benchmarks

Usage

pip install -r requirements.txt

English question, FP32:

python inference.py --model-dir . --image examples/receipt_en.jpg \
  --question "What is the first number of the store's phone number?" \
  --question "What is the street number in the store address?"

{ "record": { "phone": "4234929", "street": "732" },
  "forward_passes": 1,
  "answers": [ { "family": "phone",   "op": "front_1",   "answer": "4" },
               { "family": "address", "op": "street_no", "answer": "732" } ] }

Korean question, INT8:

python inference.py --model-dir . --precision int8 \
  --image examples/receipt_ko.jpg \
  --question "가게 전화번호의 뒤에서 1번째 숫자는 무엇입니까?" \
  --question "영수증의 가게 주소에서 도로명 뒤 숫자는 무엇입니까?"

{ "record": { "phone": "5008936", "street": "699" },
  "forward_passes": 1,
  "answers": [ { "family": "phone",   "op": "back_1",    "answer": "6" },
               { "family": "address", "op": "street_no", "answer": "699" } ] }

Both questions are served from the one forward pass that produced record. The router understands 앞에서 N번째, 뒤에서 N번째, 앞자리, 뒷자리, 끝자리, and the English ordinals.

A question outside the two supported fields routes to other and returns an empty answer rather than guessing:

python inference.py --model-dir . --image examples/receipt_ko.jpg \
  --question "상호명이 무엇입니까?"

{ "family": "other", "op": "unsupported", "answer": "" }

Check the bundled examples end to end:

python examples/test_examples.py

Input contract: grayscale, resized to 672x320 (width x height) with bilinear resampling, scaled to [0,1], then normalized (x - 0.5) / 0.5. The graph takes image as float32 [batch, 1, 320, 672] and returns slot_logits as float32 [batch, 16, 11]. Classes 0..9 are digits and class 10 is blank; read each group until the first blank. Slots 0..11 are the phone number and slots 12..15 are the street number.

Evaluation

2,000 held-out receipt questions, excluded from training, from checkpoint selection, and from quantization calibration.

precision	answer_exact	target_exact	address	phone	full phone read	street
FP32	0.9875	0.9690	0.9777	0.9979	0.9940	0.9750
INT8 W8A8	0.9845	0.9585	0.9709	0.9990	0.9900	0.9670

answer_exact is the regex route plus index compared with the annotation answer. target_exact is stricter still: every phone digit and every street digit correct on the same receipt, whether or not the question asked for them. 131 of the 2,000 answers are right while the rest of the record is not, which is what the gap between the two columns measures.

Note that target_exact here is not the transformer baseline's target_exact. That one also required transcribing the full address text, which this model never attempts. The comparable pair is full phone read: 0.9194 for the transformer against 0.9574 here.

Runtime

onnxruntime CPU execution provider, one thread, batch 1. Timed on a shared host, so these are per-image minima over 9 runs; see eval/runtime_benchmark_*.json for medians as well.

precision	median ms	parameters
FP32	34.6	2.46M
INT8	23.7	2.46M

Comparison with the transformer baseline

ivere27/tiny-receipt-vqa-structured-qa-21m is the encoder-decoder VQA transformer this model replaces. Both were measured on the same 2,000-item held-out split and the same CPU protocol (onnxruntime, one thread, batch 1).

	VQA transformer	this model
parameters	21.8M	2.46M
answer_exact FP32	0.9725	0.9875
answer_exact INT8	0.9710	0.9845
full record read (target_exact)	not comparable	0.9690
address	0.9535	0.9777
phone	0.9928	0.9979
full phone-number read	0.9618	0.9940
CPU latency FP32	288.5 ms	34.6 ms
CPU latency INT8	173.7 ms	23.7 ms
ONNX graphs	2, plus a per-token decode loop	1
tokenizer	byte-fallback BPE, 1536 tokens	none
passes for N questions on one receipt	N encoder runs	1, record cached

Same or better on every accuracy column at 8.9x fewer parameters and 8.3x lower latency.

Two caveats belong with that table.

The comparison favours this model by construction. The transformer answers eight question families — store name, item rows, item arithmetic, item lookup, and more. This model answers two. The held-out set happens to test only those two; on anything else this model returns an empty string. It is a specialist measured on a specialist's benchmark.

In one respect the comparison is conservative. The transformer's release notes describe its checkpoint as chosen by comparing two candidates on the held-out split. This checkpoint was chosen on schedule completion, without reference to held-out accuracy.

The transformer also transcribes the full address and store name, which this model never attempts. That subtask is where it struggles: its address transcription exact-match is 0.0155. Dropping it is what makes the small model both faster and, on digits, more accurate.

Quantization

model_int8.onnx quantizes convolutions only. The readout is deliberately left in float: it is about 5% of runtime, and quantizing its MatMuls costs six points of answer_exact and fifty-six of target_exact for 0.7 ms and 1.2 MB.

quantized ops	answer_exact	target_exact	size
none (FP32)	0.9875	0.9690	9.9 MB
Conv (shipped)	0.9845	0.9585	4.1 MB
Conv,MatMul,Gemm	0.9260	0.3990	2.9 MB
Conv,MatMul,Gemm,Add,Mul	0.8475	0.0590	2.6 MB

The readout MatMuls are the attention itself — a softmax choosing among 420 grid cells — not a residual-wrapped feature transform, so quantization noise moves a slot to a different cell instead of averaging out.

Limitations

• Reads two fields only: phone number and street number. Store names, item rows, and item arithmetic are out of scope.
• The output alphabet is 0-9 plus blank. No text is transcribed, so full addresses and store names cannot be produced.
• Input geometry is fixed at 672x320; the stem's final (2,1) stride ties the graph to that size.
• question_router.py must ship with the model. The graph alone cannot answer a question.
• Questions the router maps to neither phone nor address return an empty answer.
• The evaluation annotations are machine generated, not human adjudicated.