README.md

---
license: apache-2.0
language:
- en
- ko
library_name: onnxruntime
pipeline_tag: image-to-text
tags:
- onnx
- onnxruntime
- receipt
- key-information-extraction
- document-understanding
---

# tiny-receipt-reader-digit-slots-2m

A question-free receipt reader. One small CNN reads the image once and
emits a fixed record; questions are answered afterwards by regular
expressions and array indexing.

There is no transformer, no decoder, no autoregressive loop, no tokenizer,
and no question encoder. The graph has one input: the image.

```text
image ──▶ CNN ──▶ slots 0..11  phone digits
                  slots 12..15 street-number digits

question ──▶ regex ──▶ (family, op) ──▶ index the record ──▶ answer
```

Several questions about one receipt cost one forward pass, because the
record does not depend on the question.

## Files

```text
model.onnx           FP32 graph
model_int8.onnx      static W8A8 (QDQ, U8S8) graph
config.json          input contract and slot layout
manifest.json        graph contract and the full export verification report
question_router.py   regex router, required to answer a question
inference.py         onnxruntime runtime
examples/            two synthetic receipts and a runnable check
eval/                held-out reports and runtime benchmarks
```

## Usage

```bash
pip install -r requirements.txt
```

English question, FP32:

```bash
python inference.py --model-dir . --image examples/receipt_en.jpg \
  --question "What is the first number of the store's phone number?" \
  --question "What is the street number in the store address?"
```

```json
{ "record": { "phone": "4234929", "street": "732" },
  "forward_passes": 1,
  "answers": [ { "family": "phone",   "op": "front_1",   "answer": "4" },
               { "family": "address", "op": "street_no", "answer": "732" } ] }
```

Korean question, INT8:

```bash
python inference.py --model-dir . --precision int8 \
  --image examples/receipt_ko.jpg \
  --question "가게 전화번호의 뒤에서 1번째 숫자는 무엇입니까?" \
  --question "영수증의 가게 주소에서 도로명 뒤 숫자는 무엇입니까?"
```

```json
{ "record": { "phone": "5008936", "street": "699" },
  "forward_passes": 1,
  "answers": [ { "family": "phone",   "op": "back_1",    "answer": "6" },
               { "family": "address", "op": "street_no", "answer": "699" } ] }
```

Both questions are served from the one forward pass that produced
`record`. The router understands `앞에서 N번째`, `뒤에서 N번째`, `앞자리`,
`뒷자리`, `끝자리`, and the English ordinals.

A question outside the two supported fields routes to `other` and returns
an empty answer rather than guessing:

```bash
python inference.py --model-dir . --image examples/receipt_ko.jpg \
  --question "상호명이 무엇입니까?"
```

```json
{ "family": "other", "op": "unsupported", "answer": "" }
```

Check the bundled examples end to end:

```bash
python examples/test_examples.py
```

Input contract: grayscale, resized to 672x320 (width x height) with
bilinear resampling, scaled to `[0,1]`, then normalized `(x - 0.5) / 0.5`.
The graph takes `image` as float32 `[batch, 1, 320, 672]` and returns
`slot_logits` as float32 `[batch, 16, 11]`. Classes `0..9` are
digits and class `10` is blank; read each group until the first
blank. Slots `0..11` are the phone number and slots
`12..15` are the street number.

## Evaluation

2,000 held-out receipt questions, excluded from training, from
checkpoint selection, and from quantization calibration.

| precision | answer_exact | target_exact | address | phone | full phone read | street |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| FP32 | 0.9875 | 0.9690 | 0.9777 | 0.9979 | 0.9940 | 0.9750 |
| INT8 W8A8 | 0.9845 | 0.9585 | 0.9709 | 0.9990 | 0.9900 | 0.9670 |

`answer_exact` is the regex route plus index compared with the annotation
answer. `target_exact` is stricter still: every phone digit *and* every
street digit correct on the same receipt, whether or not the question
asked for them. 131 of the 2,000 answers are right while the rest of the
record is not, which is what the gap between the two columns measures.

Note that `target_exact` here is not the transformer baseline's
`target_exact`. That one also required transcribing the full address text,
which this model never attempts. The comparable pair is `full phone read`:
0.9194 for the transformer against 0.9574 here.

## Runtime

`onnxruntime` CPU execution provider, one thread, batch 1. Timed on a
shared host, so these are per-image minima over 9 runs; see
`eval/runtime_benchmark_*.json` for medians as well.

| precision | median ms | parameters |
| --- | ---: | ---: |
| FP32 | 34.6 | 2.46M |
| INT8 | 23.7 | 2.46M |

## Comparison with the transformer baseline

[ivere27/tiny-receipt-vqa-structured-qa-21m](https://huggingface.co/ivere27/tiny-receipt-vqa-structured-qa-21m) is the encoder-decoder VQA transformer this
model replaces. Both were measured on the same 2,000-item held-out split
and the same CPU protocol (`onnxruntime`, one thread, batch 1).

| | VQA transformer | this model |
| --- | ---: | ---: |
| parameters | 21.8M | 2.46M |
| `answer_exact` FP32 | 0.9725 | **0.9875** |
| `answer_exact` INT8 | 0.9710 | 0.9845 |
| full record read (`target_exact`) | not comparable | 0.9690 |
| address | 0.9535 | 0.9777 |
| phone | 0.9928 | 0.9979 |
| full phone-number read | 0.9618 | 0.9940 |
| CPU latency FP32 | 288.5 ms | **34.6 ms** |
| CPU latency INT8 | 173.7 ms | 23.7 ms |
| ONNX graphs | 2, plus a per-token decode loop | 1 |
| tokenizer | byte-fallback BPE, 1536 tokens | none |
| passes for N questions on one receipt | N encoder runs | 1, record cached |

Same or better on every accuracy column at 8.9x fewer
parameters and 8.3x lower latency.

Two caveats belong with that table.

**The comparison favours this model by construction.** The transformer
answers eight question families — store name, item rows, item arithmetic,
item lookup, and more. This model answers two. The held-out set happens to
test only those two; on anything else this model returns an empty string.
It is a specialist measured on a specialist's benchmark.

**In one respect the comparison is conservative.** The transformer's
release notes describe its checkpoint as chosen by comparing two candidates
on the held-out split. This checkpoint was chosen on schedule completion,
without reference to held-out accuracy.

The transformer also transcribes the full address and store name, which
this model never attempts. That subtask is where it struggles: its address
transcription exact-match is 0.0155. Dropping it is what makes the small
model both faster and, on digits, more accurate.

## Quantization

`model_int8.onnx` quantizes convolutions only. The readout is deliberately
left in float: it is about 5% of runtime, and quantizing its MatMuls costs
six points of `answer_exact` and fifty-six of `target_exact` for 0.7 ms
and 1.2 MB.

| quantized ops | answer_exact | target_exact | size |
| --- | ---: | ---: | ---: |
| none (FP32) | 0.9875 | 0.9690 | 9.9 MB |
| `Conv` (shipped) | 0.9845 | 0.9585 | 4.1 MB |
| `Conv,MatMul,Gemm` | 0.9260 | 0.3990 | 2.9 MB |
| `Conv,MatMul,Gemm,Add,Mul` | 0.8475 | 0.0590 | 2.6 MB |

The readout MatMuls are the attention itself — a softmax choosing among
420 grid cells — not a residual-wrapped feature transform, so quantization
noise moves a slot to a different cell instead of averaging out.

## Limitations

- Reads two fields only: phone number and street number. Store names, item
  rows, and item arithmetic are out of scope.
- The output alphabet is `0-9` plus blank. No text is transcribed, so full
  addresses and store names cannot be produced.
- Input geometry is fixed at 672x320; the stem's final `(2,1)` stride ties
  the graph to that size.
- `question_router.py` must ship with the model. The graph alone cannot
  answer a question.
- Questions the router maps to neither `phone` nor `address` return an
  empty answer.
- The evaluation annotations are machine generated, not human adjudicated.