TinyReceiptVQA is a compact receipt-domain visual question-answering model. It
takes a full receipt image and a question, then generates a structured rationale
and final <answer> value. A learned router selects one of eight family slots;
each slot has a post-encoder memory adapter and a post-decoder adapter for phone,
address, store, item-row, item-math, item-lookup, math, or other questions.
The FP32 and INT8 models use KV caching. The encoder runs once and emits
per-layer cross-attention K/V. The decoder then consumes one token at a time
while carrying its self-attention K/V and target-padding-mask outputs.
These four ONNX graphs are the complete distributed model set.
Usage
bash
1pip install -r requirements.txt
23python inference.py \4 --model-dir .\5 --image examples/receipt_en.jpg \6 --question "What is the street number in the store address?"7# Expected JSON fields: "answer": "837", "well_formed": true89# Use the Korean example with the static W8A8 U8S8 QDQ ONNX files.10python inference.py \11 --model-dir .\12 --precision int8 \13 --image examples/receipt_ko.jpg \14 --question "영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까?"15# Expected JSON fields: "answer": "875", "well_formed": true
--family auto is the default and uses the learned router. An explicit family
such as phone or address can be supplied for controlled debugging.
The input image contract is float32 [batch, 1, 320, 672]: convert to
grayscale, resize to 672x320 (width x height), scale to [0,1], then normalize
with (x - 0.5) / 0.5.
Runtime interface
The opset is 18. Batch, question length, memory length, and past/present cache
lengths are dynamic; image height and width are fixed.
Encoder inputs: image, question_ids, and family_ids (-1 means AUTO).
Encoder outputs: memory, padding mask, router logits, selected family, and
cross_k_0/v_0 through cross_k_3/v_3.
Decoder inputs: one decoder_input_ids token, per-sample position_ids,
selected family, encoder/target padding masks, per-layer Cross K/V, and
per-layer prior Self K/V.
Decoder outputs: one-token logits, updated target padding mask, and per-layer
present Self K/V. External cache tensors remain float32 in the W8A8 variant.
Use the included inference.py entry point to handle preprocessing,
AUTO-routing, and greedy generation.
The JSON result sets well_formed to true only when generation contains a
complete <answer>...</answer> block. If that block is missing, answer is an
empty string and well_formed is false; inspect generated for diagnostics.
Byte-Fallback BPE and vocabulary
This release uses only the packaged Byte-Fallback BPE1536 tokenizer
contract. vocab.json declares type=byte_fallback_bpe, version 1, NFC
normalization, and exactly 1,536 tokens. Character-vocabulary artifacts are
rejected instead of falling back silently. Its SHA-256 tokenizer fingerprint is
5801826ac9092bdc984210af805d2ff20f0398b80adaf5963ef13b04771a8584.
Token IDs
Count
Purpose
0–3
4
<pad>, <bos>, <eos>, <unk>
4–11
8
Atomic <field>, <value>, <op>, and <answer> open/close tags
12–21
10
Atomic digits 0 through 9
22–277
256
Every UTF-8 byte as <0x00> through <0xFF>
278–1018
741
Unicode characters learned from training questions and targets
1019–1535
517
Learned BPE merge outputs
The structured tags and digits never participate in merges. Whitespace is a
hard merge boundary; in this vocabulary an ASCII space is <0x20> (ID 54).
If a Unicode character is absent from the learned alphabet, its UTF-8 bytes are
emitted instead, so valid Unicode input can round-trip without using <unk>:
decode(encode(text)) == NFC(text). Byte fallback guarantees representability,
not that the vision model can recognize an unseen character correctly.
Only training records were used to learn the alphabet and merges. The optional
tokenizers library was used while building the vocabulary, but packaged
loading, encoding, and decoding use the dependency-free byte_bpe.py reference
runtime. Browser implementations must reproduce its NFC, whitespace-boundary,
atomic-token, merge-order, and byte-fallback behavior exactly.
The target schema is direct_answer_no_value_v1. For direct reads such as a
phone number, store name, or item name, <value> would merely duplicate the
answer and is omitted. It remains for address extraction, lookup, and arithmetic
when it carries distinct evidence.
Training used BF16 autocast on the GPU. Saved source tensors and the FP32 ONNX
graphs are FP32; this is expected and does not mean training was performed in
FP32. The additional deployment graphs are static W8A8 INT8.
Verified bilingual examples
These receipts come from the synthetic validation split, not heldout data or
INT8 calibration data. They were selected as readable demonstrations and are
not a family-level benchmark.
English
Korean
English synthetic receipt
Korean synthetic receipt
Listed family / case
English question → answer
Korean question → answer
phone
What is the store phone number? → 6533831
가게 전화번호는 무엇입니까? → 9113125
address
What is the street number in the store address? → 837
영수증의 가게 위치에서 도로명 뒤 숫자는 무엇입니까? → 875
store
What is the store name? → Best price Mall
가게 이름은 무엇입니까? → 착한 도매슈퍼마켓
item_row
What is the name of item 1? → Packed Yogurt
1번째 품목명은 무엇입니까? → 과일류
item_math
What is the sum of item totals? → 2200
품목 총합을 모두 더하면 얼마입니까? → 3600
item_lookup
What is the quantity of Packed Yogurt? → 3
과일류 개수는 몇 개입니까? → 4
math
Calculate item 1 unit price multiplied by quantity. → 1800
1번째 품목의 가격 곱하기 개수는 얼마입니까? → 400
extra phone position
What is the second number of the store's phone number? → 5
가게 전화번호의 뒤에서 3번째 숫자는 무엇입니까? → 1
All 16 questions passed in every tested combination:
Routing
FP32
Static W8A8 INT8
Listed-family agreement
Explicit family override
16/16
16/16
16/16
Learned AUTO router
16/16
16/16
14/16
AUTO routes the two listed math questions to trained item_math; their final
answers and decoded outputs remain exact. The first six listed families have
training records. math and other have zero family-specific training records,
so the explicit math rows are execution smoke tests, not evidence of general
math-family accuracy. other is not included.
Observed decoded output by family
The following outputs were observed identically for FP32 and INT8, with both
explicit and AUTO routing, on the English example. These are complete decoder
outputs after removing <bos>/<eos> for display.
These are the actual FP32 decoder tokens for the seven English family rows,
not strings decoded and then re-encoded. The selected examples emitted the same
token sequences in INT8.
The Korean back-position question generated
<field>phone</field><value>9113125</value><op>back_3</op><answer>1</answer>.
Run the packaged test from the repository root. It reports decoded text,
actual decoder IDs, token strings, selected families, and router logits:
bash
1python examples/test_all_families.py \2 --model-dir .\3 --precision both \4 --routing both
56# Learned router only (family_ids=-1).7python examples/test_all_families.py --model-dir . --routing auto
The INT8 variant uses static U8S8 W8A8 QDQ quantization for convolution,
attention, FFN, adapter, router, and output-head Conv, MatMul, and Gemm
operators. Weights are symmetric signed INT8 per output channel, activations
are asymmetric UINT8 per tensor, and integer accumulation is INT32. Calibration
uses only synthetic training records; heldout records are not calibration
inputs. ONNX Runtime can fuse supported QDQ regions into integer kernels, while
normalization, softmax, positional, and shape operations remain floating point.
The two FP32 graphs total 86.2 MiB; the matching W8A8 graphs total
27.4 MiB, a 68.2% reduction. These are executable
quantized ONNX graphs, not storage-only weights that are dequantized back to
FP32 before inference.
Validation
All four model graphs pass ONNX Checker and CPU ONNX Runtime execution. Validation
covered AUTO routing, all eight explicit adapters, dynamic sequence shapes, and
greedy generation. The selected adapter ID matched exactly.
On the 2,000-question AUTO phone/address heldout evaluation, the model achieved
answer exact 97.25% and target exact 47.35% (FP32); the static W8A8 graph
achieves 97.10% and 47.20%.
Subset
n
FP32 answer exact
FP32 target exact
INT8 answer exact
overall
2000
97.25%
47.35%
97.10%
phone
968
99.28%
96.18%
99.28%
address
1032
95.35%
1.55%
95.06%
Address target exact is low because the target includes a full transcription
of the address text in <value>, which this model rarely reproduces character
for character. The <answer> it is asked for — the street number — is correct
95.35% of the time.
The source PyTorch reference and the final ONNX FP32 rerun agree exactly on all
nine reported accuracy metrics across the overall, address, and phone subsets.
For readability, the table reports character accuracy, 1 - CER, alongside
CER. These are NFC-normalized PyTorch FP32 AUTO results for the same 2,000
heldout examples, computed from the FP32 ONNX rerun. Lower CER and higher
character accuracy are better.
Scope
Character accuracy
CER
Overall target
96.00%
4.00%
Overall <value>
78.55%
21.45%
Overall <answer>
98.35%
1.65%
Address target
93.11%
6.89%
Address <value>
72.96%
27.04%
Address <answer>
98.06%
1.94%
Phone target
99.92%
0.08%
Phone <value>
99.29%
0.71%
Phone <answer>
99.28%
0.72%
Full-target character metrics include repeated structured markup such as
<field>, <op>, and <answer>. Those common tag characters are easy to
generate and therefore make full-target character accuracy look better than the
model's actual OCR/copy quality. Use <value> CER to assess transcription of
the visual evidence and <answer> exact/CER to assess final QA output.
The final default KV-cache ONNX files were also evaluated directly on all 2,000
heldout questions with ONNX Runtime 1.23.2 and CPUExecutionProvider, using
learned AUTO routing:
ONNX variant
Answer exact
Recomputed exact
Target exact
FP32
97.05%
97.00%
45.30%
W8A8 U8S8 QDQ
96.80%
96.80%
45.15%
The two variants had 74.45% full
prediction agreement, 97.95% answer agreement,
and 100.00% selected-family agreement.
There were 511 changed full predictions out of 2,000. The INT8 answer-exact
result was 0.25 percentage points below FP32. Variant-specific accuracy is
reported independently above.
With batch size 24 and 12 CPU threads, accumulated model execution time for all
records was 273.27 seconds for FP32 and 257.15 seconds for W8A8. These aggregate
evaluation timings are not single-query latency measurements; the one-thread
per-VQA runtime measurements are reported below. See
eval/fp32_int8_heldout_summary.json for the complete summary.
Model design and architecture
This configuration has 21,834,104 train-time parameters (about 21.8M,
rounded to 22M in the model name), and all of them were trainable. The token
embedding and output projection share one weight matrix, so it is counted once.
The LoRA branches account for 256,000 parameters during training; they are
folded into dense FFN weights for ONNX inference and add no runtime branch.
The input is float32 grayscale [B, 1, 320, 672]. Five bias-free 3x3
convolutions downsample through 1 -> 48 -> 96 -> 192 -> 320 -> 320 channels.
Conv blocks use GroupNorm and SiLU; four two-convolution residual blocks are
interleaved with the downsampling stages. The total stride is 32, producing a
10x21 feature map that is flattened into 210 visual tokens of width 320.
Visual tokens receive learned image positions and an image-type embedding.
Question tokens use the shared 1536x320 token embedding, learned question
positions, and a question-type embedding. The model concatenates
[visual tokens, question tokens] and applies a 6-layer pre-LayerNorm
Transformer encoder with 8 attention heads (head width 40), FFN width 1280,
GELU, and dropout 0.1. The encoded memory shape is [B, 210 + Q, 320].
Router and family adapters
The router sees question embeddings only, before image/question fusion. It
mean-pools non-padding question tokens, then applies Linear(320 -> 160), GELU,
and Linear(160 -> 8). AUTO routing takes the hard argmax;
family_ids=0..7 explicitly overrides it. The ordered slots are phone,
address, store, item_row, item_math, item_lookup, math, and other.
One selected residual bottleneck adapter is applied after the encoder and one
after the decoder. Each is 320 -> 64 -> 320 with GELU, so this is eight routed
family slots backed by 16 adapter MLPs, not a soft mixture of all adapters.
Training used token cross-entropy plus 0.1 * router-family cross-entropy.
Family-specific training records cover the first six slots. math and other
are untrained extension slots whose encoder and decoder adapter up-projections
are exactly zero. Explicitly selecting either slot therefore applies an
identity adapter; AUTO can instead route such prompts to a trained family.
Decoder, LoRA, and deployment split
The 4-layer pre-LayerNorm Transformer decoder uses causal self-attention and
cross-attention to multimodal memory, with the same 320-wide, 8-head,
1280-wide-FFN design. A final LayerNorm and output bias produce token logits.
The output projection is weight-tied to the input token embedding.
Training added rank-8 LoRA (alpha=16, scale 2, dropout 0.05) to linear1 and
linear2 in every encoder and decoder FFN: 20 Linear modules in total. This was
LoRA-augmented full-model training, not a frozen-base PEFT run. Export folded
all 20 deltas into their dense weights; the maximum pre/post-merge absolute
difference was 5.245208740234375e-06. The ONNX graphs contain no separate
LoRA operators.
The split encoder runs the CNN, multimodal encoder, router, memory adapter, and
four per-layer cross-attention K/V projections once. The default FP32 and W8A8
decoders consume one token per call and carry headed self-attention K/V plus the
target padding mask between calls. The encoder padding mask remains active in
every cross-attention operation. On CPUExecutionProvider with one thread,
median-of-per-VQA median model E2E latency was 288.50 ms for FP32 and 173.67 ms
for W8A8. The reports include measurements for each of the 20 VQAs:
eval/runtime_benchmark_fp32_cpu.json and
eval/runtime_benchmark_int8_cpu.json. The 2,000-question accuracy comparison
above is the quality check for both variants. Generation uses greedy argmax from
BOS until EOS or 192 tokens.
The source checkpoint is epoch-100 last.pt. Heldout was excluded from training
and intermediate checkpointing, then used post hoc to compare e79 best.pt
with e100 last.pt; e100 last.pt was selected for higher heldout answer exact.
Target serialization: direct_answer_no_value_v1
This is the current training-target serialization identifier; it is not a
model-architecture or tokenizer version. The complete serialized target is
teacher-forced into the decoder and contributes to token cross-entropy loss:
<field>, <op>, and <answer> are always retained. <value> is omitted only
when it would repeat the final answer: identity, copy, count, and
item_N_name, item_N_unit_price, item_N_quantity, or item_N_total.
It remains when it carries distinct evidence for extraction, lookup, or
arithmetic. For example:
The separate answer property in an annotation is used for evaluation; it is
not a second decoder-loss target. Using the v1 identifier does not alter the
learned output behavior or model weights, so retraining is unnecessary.
Limitations
Training and task scope
This checkpoint learned a finite bilingual structured_qa task grammar for
receipts. It is not an open-world document-VQA system, a general-purpose OCR
engine, or an arbitrary table-query/calculation model. Training used synthetic
receipts together with annotated real receipts, but this release does not ship
the exact training-record count or per-family training distribution.
Trained family
Supervised coverage
Boundary not claimed
phone
Full digits-only number; one digit from the front or back at positions 1–4; prefix/suffix lengths 2–4; digit count and digit sum
Position 5 or later, arbitrary substring extraction, and unseen wording are out of the generated task range
address
Full address; final street/building number; first and second whitespace token; road name; whitespace-removed character positions 1–3 from either end
Geocoding and free-form address semantics are not trained; the component parser assumes a simple road name and a final digit group
store
Full store name; first/last word; whitespace-removed character count; character positions 1–4 from either end
Other character positions, arbitrary spans, and merchant/entity normalization are not trained
item_row
Item count; name, unit price, quantity, and row total for rows 1–4; first character of those item names
Direct row-index questions for row 5 or later and fields outside this schema are not trained
item_lookup
Name or sampled partial-name lookup of price, quantity, or total from the first 6 eligible non-address-like rows; absent-item negatives with the fixed answer 없음; reverse lookup when a numeric value identifies one unique row
Ambiguous duplicate values, arbitrary fuzzy matching, and name-based lookup beyond row 6 are not covered
item_math
Per-row unit_price × quantity for rows 1–4; sums and min/max values or item names over valid total, unit-price, and quantity rows; sum and absolute difference of the first two row totals
General arithmetic, arbitrary expressions, and unconstrained table reasoning are not trained
The prompt generator selects from finite English and Korean template banks;
question language may differ from the receipt text language. Byte fallback makes
other Unicode text encodable, but it does not add multilingual OCR or QA
capability. Close human paraphrases are therefore not guaranteed. For example,
prefer an explicit learned form such as 전화번호 앞에서 1번째 숫자는? or
What is the first digit of the store phone number?. A shorter form such as
전화번호 첫번째 숫자, wording with index, or a request for the fifth digit
is outside the guaranteed template/range and may produce a plausible but wrong
operation.
Only phone, address, store, item_row, item_math, and item_lookup
have family-specific training records. math and other are untrained
extension slots with exact identity adapters; explicitly selecting either slot
does not provide a separately trained math or catch-all capability. Direct
extraction of dates, times, tax, payment method, printed subtotal/grand total,
change, and other fields outside the table above has no task-specific accuracy
claim.
Images are converted to grayscale and resized to the fixed 672x320 input.
Training augmentation covered modest contrast/brightness changes, blur, and
small rotations, so tiny text, heavy blur, large rotation, unusual layouts, or
very dense receipts may degrade recognition. CER in this card is informational
for transcription within the evaluated tasks; it is not evidence of unrestricted
full-document OCR quality.
Output and evaluation scope
well_formed: true means only that a complete <answer>...</answer> block
was generated. It does not verify that the selected family, <op>, copied
<value>, or answer is semantically correct.
The 2,000-record heldout evaluation contains only supported phone/address
prompts (1,032 address and 968 phone). It does not validate store/item
families, every trained operation, or novel paraphrases. The bilingual family
examples are selected smoke tests, not a family-level benchmark.
Heldout was excluded from training and intermediate checkpointing. After
training it was used to compare e79 best.pt and e100 last.pt and to select
e100 last.pt, so it is a development test rather than a final blind
benchmark.
W8A8 output can differ from FP32 output because both weights and activations
are quantized. Validate accuracy and latency in the target browser.
Integer-kernel fusion depends on the ONNX Runtime execution provider and
target hardware.
