README.md

---
license: apache-2.0
language:
- en
- ko
library_name: onnxruntime
pipeline_tag: visual-question-answering
tags:
- onnx
- onnxruntime
- document-question-answering
- receipt
- key-information-extraction
- lora
- routing
---

# TinyReceiptVQA-StructuredQA-22M-BPE1536-DirectNoValue-LoRA-Router-e100

TinyReceiptVQA is a compact receipt-domain visual question-answering model. It
takes a full receipt image and a question, then generates a structured rationale
and final `<answer>` value. A learned router selects one of eight family slots;
each slot has a post-encoder memory adapter and a post-decoder adapter for phone,
address, store, item-row, item-math, item-lookup, math, or other questions.

## Files

```text
encoder_model.onnx       # CNN + multimodal encoder + router + memory adapter
decoder_model.onnx       # autoregressive decoder + selected decoder adapter
encoder_model_int8.onnx  # static W8A8 U8S8 QDQ encoder
decoder_model_int8.onnx  # static W8A8 U8S8 QDQ decoder
manifest.json            # graph contract, hashes, and runtime validation
config.json
vocab.json
byte_bpe.py              # dependency-free tokenizer runtime
inference.py             # ONNX Runtime greedy decoding
examples/
  receipt_en.jpg         # verified English synthetic receipt
  receipt_ko.jpg         # verified Korean synthetic receipt
  family_examples.json   # 16 explicit/AUTO checks and expected answers
  test_all_families.py   # emits decoded text, token IDs, and BPE tokens
eval/
  heldout_summary.json   # heldout evaluation summary
  int8_w8a8_summary.json
  onnx_parity_summary.json
  comparison.json
```

The encoder is run once. The host runtime then calls the decoder with the
growing output prefix until EOS; this avoids recomputing the receipt image and
question encoder for every generated token.

## Usage

```bash
pip install -r requirements.txt

python inference.py \
  --model-dir . \
  --image examples/receipt_en.jpg \
  --question "What is the street number in the store address?"
# Expected JSON fields: "answer": "837", "well_formed": true

# Use the Korean example with the static W8A8 U8S8 QDQ ONNX files.
python inference.py \
  --model-dir . \
  --precision int8 \
  --image examples/receipt_ko.jpg \
  --question "영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까?"
# Expected JSON fields: "answer": "875", "well_formed": true
```

`--family auto` is the default and uses the learned router. An explicit family
such as `phone` or `address` can be supplied for controlled debugging.

The input image contract is float32 `[batch, 1, 320, 672]`: convert to
grayscale, resize to 672x320 (width x height), scale to `[0,1]`, then normalize
with `(x - 0.5) / 0.5`.

## Runtime interface

The opset is 18. Batch, question length, and decoder-prefix
length are dynamic; image height and width are fixed.

- Encoder inputs: `image`, `question_ids`, and `family_ids` (`-1` means AUTO).
- Encoder outputs: cached `memory`, padding mask, router logits, selected family.
- Decoder inputs: generated prefix, cached memory/mask, selected family.
- Decoder output: token logits.

Use the included `inference.py` entry point to handle preprocessing,
AUTO-routing, and greedy generation.

The JSON result sets `well_formed` to `true` only when generation contains a
complete `<answer>...</answer>` block. If that block is missing, `answer` is an
empty string and `well_formed` is `false`; inspect `generated` for diagnostics.

## Byte-Fallback BPE and vocabulary

This release uses only the packaged **Byte-Fallback BPE1536** tokenizer
contract. `vocab.json` declares `type=byte_fallback_bpe`, version 1, NFC
normalization, and exactly 1,536 tokens. Character-vocabulary artifacts are
rejected instead of falling back silently. Its SHA-256 tokenizer fingerprint is
`5801826ac9092bdc984210af805d2ff20f0398b80adaf5963ef13b04771a8584`.

| Token IDs | Count | Purpose |
| --- | ---: | --- |
| 0–3 | 4 | `<pad>`, `<bos>`, `<eos>`, `<unk>` |
| 4–11 | 8 | Atomic `<field>`, `<value>`, `<op>`, and `<answer>` open/close tags |
| 12–21 | 10 | Atomic digits `0` through `9` |
| 22–277 | 256 | Every UTF-8 byte as `<0x00>` through `<0xFF>` |
| 278–1018 | 741 | Unicode characters learned from training questions and targets |
| 1019–1535 | 517 | Learned BPE merge outputs |

The structured tags and digits never participate in merges. Whitespace is a
hard merge boundary; in this vocabulary an ASCII space is `<0x20>` (ID 54).
If a Unicode character is absent from the learned alphabet, its UTF-8 bytes are
emitted instead, so valid Unicode input can round-trip without using `<unk>`:
`decode(encode(text)) == NFC(text)`. Byte fallback guarantees representability,
not that the vision model can recognize an unseen character correctly.

Only training records were used to learn the alphabet and merges. The optional
`tokenizers` library was used while building the vocabulary, but packaged
loading, encoding, and decoding use the dependency-free `byte_bpe.py` reference
runtime. Browser implementations must reproduce its NFC, whitespace-boundary,
atomic-token, merge-order, and byte-fallback behavior exactly.

The target schema is `direct_answer_no_value_v1`. For direct reads such as a
phone number, store name, or item name, `<value>` would merely duplicate the
answer and is omitted. It remains for address extraction, lookup, and arithmetic
when it carries distinct evidence.

Training used BF16 autocast on the GPU. Saved source tensors and the FP32 ONNX
graphs are FP32; this is expected and does not mean training was performed in
FP32. The additional deployment graphs are static W8A8 INT8.

## Verified bilingual examples

These receipts come from the synthetic validation split, not heldout data or
INT8 calibration data. They were selected as readable demonstrations and are
not a family-level benchmark.

| English | Korean |
| --- | --- |
| ![English synthetic receipt](examples/receipt_en.jpg) | ![Korean synthetic receipt](examples/receipt_ko.jpg) |

| Listed family / case | English question → answer | Korean question → answer |
| --- | --- | --- |
| `phone` | What is the store phone number? → `6533831` | 가게 전화번호는 무엇입니까? → `9113125` |
| `address` | What is the street number in the store address? → `837` | 영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까? → `875` |
| `store` | What is the store name? → `Best price Mall` | 가게 이름은 무엇입니까? → `착한 도매슈퍼마켓` |
| `item_row` | What is the name of item 1? → `Packed Yogurt` | 1번째 품목명은 무엇입니까? → `과일류` |
| `item_math` | What is the sum of item totals? → `2200` | 품목 총합을 모두 더하면 얼마입니까? → `3600` |
| `item_lookup` | What is the quantity of Packed Yogurt? → `3` | 과일류 개수는 몇 개입니까? → `4` |
| `math` | Calculate item 1 unit price multiplied by quantity. → `1800` | 1번째 품목의 가격 곱하기 개수는 얼마입니까? → `400` |
| extra `phone` position | What is the second number of the store's phone number? → `5` | 가게 전화번호의 뒤에서 3번째 숫자는 무엇입니까? → `1` |

All 16 questions passed in every tested combination:

| Routing | FP32 | Static W8A8 INT8 | Listed-family agreement |
| --- | ---: | ---: | ---: |
| Explicit family override | 16/16 | 16/16 | 16/16 |
| Learned AUTO router | 16/16 | 16/16 | 14/16 |

AUTO routes the two listed `math` questions to trained `item_math`; their final
answers and decoded outputs remain exact. The first six listed families have
training records. `math` and `other` have zero family-specific training records,
so the explicit `math` rows are execution smoke tests, not evidence of general
math-family accuracy. `other` is not included.

### Observed decoded output by family

The following outputs were observed identically for FP32 and INT8, with both
explicit and AUTO routing, on the English example. These are complete decoder
outputs after removing `<bos>`/`<eos>` for display.

| Family | Decoded output |
| --- | --- |
| `phone` | `<field>phone</field><op>identity</op><answer>6533831</answer>` |
| `address` | `<field>addr</field><value>Seoul Hwasun Jodonan St. 837</value><op>street_no</op><answer>837</answer>` |
| `store` | `<field>store</field><op>identity</op><answer>Best price Mall</answer>` |
| `item_row` | `<field>item</field><op>item_1_name</op><answer>Packed Yogurt</answer>` |
| `item_math` | `<field>items</field><value>1800,400</value><op>sum</op><answer>2200</answer>` |
| `item_lookup` | `<field>item</field><value>Packed Yogurt</value><op>lookup_quantity</op><answer>3</answer>` |
| `math` | `<field>item</field><value>600*3</value><op>product</op><answer>1800</answer>` |

The Korean outputs use the same schema:

| Family | Decoded output |
| --- | --- |
| `phone` | `<field>phone</field><op>identity</op><answer>9113125</answer>` |
| `address` | `<field>addr</field><value>의정부시 도봉구 엄치길 875</value><op>street_no</op><answer>875</answer>` |
| `store` | `<field>store</field><op>identity</op><answer>착한 도매슈퍼마켓</answer>` |
| `item_row` | `<field>item</field><op>item_1_name</op><answer>과일류</answer>` |
| `item_math` | `<field>items</field><value>400,3200</value><op>sum</op><answer>3600</answer>` |
| `item_lookup` | `<field>item</field><value>과일류</value><op>lookup_quantity</op><answer>4</answer>` |
| `math` | `<field>item</field><value>100*4</value><op>product</op><answer>400</answer>` |

### Actual emitted BPE tokens

These are the actual FP32 decoder tokens for the seven English family rows,
not strings decoded and then re-encoded. The selected examples emitted the same
token sequences in INT8.

| Family | Emitted tokens, including generation boundaries |
| --- | --- |
| `phone` | `<bos> · <field> · phone · </field> · <op> · identity · </op> · <answer> · 6 · 5 · 3 · 3 · 8 · 3 · 1 · </answer> · <eos>` |
| `address` | `<bos> · <field> · addr · </field> · <value> · Se · ou · l · <0x20> · H · w · as · un · <0x20> · J · od · on · an · <0x20> · St. · <0x20> · 8 · 3 · 7 · </value> · <op> · street_no · </op> · <answer> · 8 · 3 · 7 · </answer> · <eos>` |
| `store` | `<bos> · <field> · store · </field> · <op> · identity · </op> · <answer> · B · est · <0x20> · price · <0x20> · M · all · </answer> · <eos>` |
| `item_row` | `<bos> · <field> · item · </field> · <op> · item_ · 1 · _name · </op> · <answer> · P · ack · ed · <0x20> · Y · o · g · u · r · t · </answer> · <eos>` |
| `item_math` | `<bos> · <field> · items · </field> · <value> · 1 · 8 · 0 · 0 · , · 4 · 0 · 0 · </value> · <op> · sum · </op> · <answer> · 2 · 2 · 0 · 0 · </answer> · <eos>` |
| `item_lookup` | `<bos> · <field> · item · </field> · <value> · P · ack · ed · <0x20> · Y · o · g · u · r · t · </value> · <op> · lookup_quantity · </op> · <answer> · 3 · </answer> · <eos>` |
| `math` | `<bos> · <field> · item · </field> · <value> · 6 · 0 · 0 · * · 3 · </value> · <op> · product · </op> · <answer> · 1 · 8 · 0 · 0 · </answer> · <eos>` |

For the English phone-position question, the observed decoded output was:

```text
<field>phone</field><value>6533831</value><op>front_2</op><answer>5</answer>
```

Its actual token IDs and tokens were:

```text
IDs:    [1, 4, 1038, 5, 6, 18, 17, 15, 15, 20, 15, 13, 7,
         8, 1063, 14, 9, 10, 17, 11, 2]
Tokens: [<bos>, <field>, phone, </field>, <value>, 6, 5, 3, 3, 8, 3, 1,
         </value>, <op>, front_, 2, </op>, <answer>, 5, </answer>, <eos>]
```

The Korean back-position question generated
`<field>phone</field><value>9113125</value><op>back_3</op><answer>1</answer>`.

Run the packaged test from the repository root. It reports decoded text,
actual decoder IDs, token strings, selected families, and router logits:

```bash
python examples/test_all_families.py \
  --model-dir . \
  --precision both \
  --routing both

# Learned router only (family_ids=-1).
python examples/test_all_families.py --model-dir . --routing auto
```

The machine-readable questions and recorded answers are in
[`examples/family_examples.json`](examples/family_examples.json).

The INT8 variant uses static U8S8 W8A8 QDQ quantization for convolution,
attention, FFN, adapter, router, and output-head `Conv`, `MatMul`, and `Gemm`
operators. Weights are symmetric signed INT8 per output channel, activations
are asymmetric UINT8 per tensor, and integer accumulation is INT32. Calibration
uses only synthetic training records; heldout records are not calibration
inputs. ONNX Runtime can fuse supported QDQ regions into integer kernels, while
normalization, softmax, positional, and shape operations remain floating point.
The two FP32 graphs total 84.3 MiB; the matching W8A8 graphs total
27.4 MiB, a 67.5% reduction. These are executable
quantized ONNX graphs, not storage-only weights that are dequantized back to
FP32 before inference.

## Validation

All four model graphs pass ONNX Checker and CPU ONNX Runtime execution. Validation
covered AUTO routing, all eight explicit adapters, dynamic sequence shapes, and
greedy generation. The selected adapter ID matched exactly.

On the 2,000-question AUTO phone/address heldout evaluation, the model achieved
answer exact 97.05%,
recomputed answer exact 97.00%, and
target exact 45.30%.

The source PyTorch reference and the final ONNX FP32 rerun agree exactly on all
nine reported accuracy metrics across the overall, address, and phone subsets.

### Character error rate

CER is the standard OCR edit-distance metric:

`CER = (substitutions + deletions + insertions) / ground-truth characters`.

For readability, the table reports character accuracy, `1 - CER`, alongside
CER. These are NFC-normalized PyTorch FP32 AUTO results for the same 2,000
heldout examples. Lower CER and higher character accuracy are better.

| Scope | Character accuracy | CER |
| --- | ---: | ---: |
| Overall target | 95.94% | 4.06% |
| Overall `<value>` | 78.26% | 21.74% |
| Overall `<answer>` | 98.17% | 1.83% |
| Address target | 93.07% | 6.93% |
| Address `<value>` | 72.84% | 27.16% |
| Address `<answer>` | 97.83% | 2.17% |
| Phone target | 99.82% | 0.18% |
| Phone `<value>` | 98.19% | 1.81% |
| Phone `<answer>` | 99.28% | 0.72% |

Full-target character metrics include repeated structured markup such as
`<field>`, `<op>`, and `<answer>`. Those common tag characters are easy to
generate and therefore make full-target character accuracy look better than the
model's actual OCR/copy quality. Use `<value>` CER to assess transcription of
the visual evidence and `<answer>` exact/CER to assess final QA output.

The final split ONNX files were also evaluated directly with
ONNX Runtime 1.23.2 using CUDAExecutionProvider with CPU fallback
enabled, with learned AUTO routing:

| ONNX variant | Answer exact | Recomputed exact | Target exact |
| --- | ---: | ---: | ---: |
| FP32 | 97.05% | 97.00% | 45.30% |
| W8A8 U8S8 QDQ | 96.80% | 96.90% | 45.05% |

The two variants had 80.60% full
prediction agreement, 99.25% answer agreement,
and 100.00% selected-family agreement.
There were 388 changed full predictions out of
2000; variant-specific accuracy is reported independently above.

Accumulated model execution time for all records was 25.75 seconds
for FP32 and 32.62 seconds for W8A8. W8A8 was
26.7% slower in this server-side provider configuration.
This CUDA timing is not a proxy for browser WASM latency; validate performance
in the target browser.

## Model design and architecture

This configuration has **21,834,104 train-time parameters** (about 21.8M,
rounded to 22M in the model name), and all of them were trainable. The token
embedding and output projection share one weight matrix, so it is counted once.
The LoRA branches account for 256,000 parameters during training; they are
folded into dense FFN weights for ONNX inference and add no runtime branch.

```text
672x320 grayscale receipt -> stride-32 CNN -> 10x21 visual tokens -----------+
question -> Byte-Fallback BPE -> question embeddings -----------------------+--> concat
question embeddings -> mean-pool router -> selected family -----------------+      |
                                                                                   v
                              6-layer multimodal encoder -> selected memory adapter
                                                                                   |
                                                                               cached memory
                                                                                   |
BOS / growing prefix -> 4-layer causal decoder + cross-attention -----------------+
                      -> selected decoder adapter -> tied vocabulary head -> next token
```

### Vision and multimodal encoder

The input is float32 grayscale `[B, 1, 320, 672]`. Five bias-free 3x3
convolutions downsample through `1 -> 48 -> 96 -> 192 -> 320 -> 320` channels.
Conv blocks use GroupNorm and SiLU; four two-convolution residual blocks are
interleaved with the downsampling stages. The total stride is 32, producing a
`10x21` feature map that is flattened into 210 visual tokens of width 320.

Visual tokens receive learned image positions and an image-type embedding.
Question tokens use the shared `1536x320` token embedding, learned question
positions, and a question-type embedding. The model concatenates
`[visual tokens, question tokens]` and applies a 6-layer pre-LayerNorm
Transformer encoder with 8 attention heads (head width 40), FFN width 1280,
GELU, and dropout 0.1. The encoded memory shape is `[B, 210 + Q, 320]`.

### Router and family adapters

The router sees question embeddings only, before image/question fusion. It
mean-pools non-padding question tokens, then applies `Linear(320 -> 160)`, GELU,
and `Linear(160 -> 8)`. AUTO routing takes the hard argmax;
`family_ids=0..7` explicitly overrides it. The ordered slots are `phone`,
`address`, `store`, `item_row`, `item_math`, `item_lookup`, `math`, and `other`.

One selected residual bottleneck adapter is applied after the encoder and one
after the decoder. Each is `320 -> 64 -> 320` with GELU, so this is eight routed
family slots backed by 16 adapter MLPs, not a soft mixture of all adapters.
Training used token cross-entropy plus `0.1 *` router-family cross-entropy.
Family-specific training records cover the first six slots. `math` and `other`
are untrained extension slots whose encoder and decoder adapter up-projections
are exactly zero. Explicitly selecting either slot therefore applies an
identity adapter; AUTO can instead route such prompts to a trained family.

### Decoder, LoRA, and deployment split

The 4-layer pre-LayerNorm Transformer decoder uses causal self-attention and
cross-attention to multimodal memory, with the same 320-wide, 8-head,
1280-wide-FFN design. A final LayerNorm and output bias produce token logits.
The output projection is weight-tied to the input token embedding.

Training added rank-8 LoRA (`alpha=16`, scale 2, dropout 0.05) to `linear1` and
`linear2` in every encoder and decoder FFN: 20 Linear modules in total. This was
LoRA-augmented **full-model training**, not a frozen-base PEFT run. Export folded
all 20 deltas into their dense weights; the maximum pre/post-merge absolute
difference was `5.245208740234375e-06`. The ONNX graphs contain no separate
LoRA operators.

The split encoder runs the CNN, multimodal encoder, router, and memory adapter
once. It returns memory, its padding mask, router logits, and the selected family
ID. The decoder receives those cached encoder outputs plus the growing prefix.
There is **no decoder KV/past cache**: each decoding step recomputes the full
prefix, while the image and question encoder are not recomputed. Generation
uses greedy argmax from BOS until EOS or 192 tokens.

The source checkpoint is epoch-100 `last.pt`. Heldout was excluded from training
and intermediate checkpointing, then used post hoc to compare e79 `best.pt`
with e100 `last.pt`; e100 `last.pt` was selected for higher heldout answer exact.

## Target serialization: `direct_answer_no_value_v1`

This is the current training-target serialization identifier; it is not a
model-architecture or tokenizer version. The complete serialized target is
teacher-forced into the decoder and contributes to token cross-entropy loss:

```text
<field>...</field>[optional <value>...</value>]<op>...</op><answer>...</answer>
```

`<field>`, `<op>`, and `<answer>` are always retained. `<value>` is omitted only
when it would repeat the final answer: `identity`, `copy`, `count`, and
`item_N_name`, `item_N_unit_price`, `item_N_quantity`, or `item_N_total`.
It remains when it carries distinct evidence for extraction, lookup, or
arithmetic. For example:

```text
<field>store</field><op>identity</op><answer>ABC마트</answer>
<field>phone</field><value>6533831</value><op>front_2</op><answer>5</answer>
```

The separate `answer` property in an annotation is used for evaluation; it is
not a second decoder-loss target. Using the `v1` identifier does not alter the
learned output behavior or model weights, so retraining is unnecessary.


## Limitations

### Training and task scope

This checkpoint learned a finite bilingual `structured_qa` task grammar for
receipts. It is not an open-world document-VQA system, a general-purpose OCR
engine, or an arbitrary table-query/calculation model. Training used synthetic
receipts together with annotated real receipts, but this release does not ship
the exact training-record count or per-family training distribution.

| Trained family | Supervised coverage | Boundary not claimed |
| --- | --- | --- |
| `phone` | Full digits-only number; one digit from the front or back at positions **1–4**; prefix/suffix lengths **2–4**; digit count and digit sum | Position 5 or later, arbitrary substring extraction, and unseen wording are out of the generated task range |
| `address` | Full address; final street/building number; first and second whitespace token; road name; whitespace-removed character positions **1–3** from either end | Geocoding and free-form address semantics are not trained; the component parser assumes a simple road name and a final digit group |
| `store` | Full store name; first/last word; whitespace-removed character count; character positions **1–4** from either end | Other character positions, arbitrary spans, and merchant/entity normalization are not trained |
| `item_row` | Item count; name, unit price, quantity, and row total for rows **1–4**; first character of those item names | Direct row-index questions for row 5 or later and fields outside this schema are not trained |
| `item_lookup` | Name or sampled partial-name lookup of price, quantity, or total from the first **6** eligible non-address-like rows; absent-item negatives with the fixed answer `없음`; reverse lookup when a numeric value identifies one unique row | Ambiguous duplicate values, arbitrary fuzzy matching, and name-based lookup beyond row 6 are not covered |
| `item_math` | Per-row `unit_price × quantity` for rows **1–4**; sums and min/max values or item names over valid total, unit-price, and quantity rows; sum and absolute difference of the first two row totals | General arithmetic, arbitrary expressions, and unconstrained table reasoning are not trained |

The prompt generator selects from finite English and Korean template banks;
question language may differ from the receipt text language. Byte fallback makes
other Unicode text encodable, but it does not add multilingual OCR or QA
capability. Close human paraphrases are therefore not guaranteed. For example,
prefer an explicit learned form such as `전화번호 앞에서 1번째 숫자는?` or
`What is the first digit of the store phone number?`. A shorter form such as
`전화번호 첫번째 숫자`, wording with `index`, or a request for the fifth digit
is outside the guaranteed template/range and may produce a plausible but wrong
operation.

Only `phone`, `address`, `store`, `item_row`, `item_math`, and `item_lookup`
have family-specific training records. `math` and `other` are untrained
extension slots with exact identity adapters; explicitly selecting either slot
does not provide a separately trained math or catch-all capability. Direct
extraction of dates, times, tax, payment method, printed subtotal/grand total,
change, and other fields outside the table above has no task-specific accuracy
claim.

Images are converted to grayscale and resized to the fixed 672x320 input.
Training augmentation covered modest contrast/brightness changes, blur, and
small rotations, so tiny text, heavy blur, large rotation, unusual layouts, or
very dense receipts may degrade recognition. CER in this card is informational
for transcription within the evaluated tasks; it is not evidence of unrestricted
full-document OCR quality.

### Output and evaluation scope

- `well_formed: true` means only that a complete `<answer>...</answer>` block
  was generated. It does not verify that the selected family, `<op>`, copied
  `<value>`, or answer is semantically correct.
- The 2,000-record heldout evaluation contains only supported phone/address
  prompts (1,032 address and 968 phone). It does not validate store/item
  families, every trained operation, or novel paraphrases. The bilingual family
  examples are selected smoke tests, not a family-level benchmark.
- Heldout was excluded from training and intermediate checkpointing. After
  training it was used to compare e79 `best.pt` and e100 `last.pt` and to select
  e100 `last.pt`, so it is a development test rather than a final blind
  benchmark.
- W8A8 output can differ from FP32 output because both weights and activations
  are quantized. Validate accuracy and latency in the target browser.
- Integer-kernel fusion depends on the ONNX Runtime execution provider and
  target hardware.