EVE-OCR-1.0-NPL

A QLoRA adapter for Qwen2.5-VL-7B-Instruct that turns a scanned page of Vietnamese non-performing-loan (NPL) collateral paperwork into a fixed 30-field JSON record. Field names stay constant across Vietnamese, Korean and English documents; only the values change language.

Trained entirely on synthetic pages. Evaluated entirely on genuine scans the model never saw.

Results

Three genuine VAMC valuation-request appendix pages, 86 scoreable fields, exact match after Unicode NFC normalisation. Same prompt, same fields, same pages for every row. Greedy decoding — the base row was measured twice in separate runs and reproduced to the decimal.

	Field exact match	Gold string present
Base model, zero-shot	64.0%	88.4%
+ adapter, 1 epoch	75.6%	86.0%
+ adapter, 2 epochs	81.4%	90.7%
+ adapter, 3 epochs	81.4%	90.7%

+17.4 pp. Two and three epochs are identical field-for-field on every page, so training converged at two; the 2-epoch adapter is the one published here.

Per page, so variance is visible rather than averaged away:

Page	Fields	Base	2 epochs	Delta
1406, appendix p.2	30	50.0%	83.3%	+33.3 pp
1348, appendix p.2	29	69.0%	79.3%	+10.3 pp
1309, appendix p.2	27	74.1%	81.5%	+7.4 pp

A principal + interest = total consistency check passes on every page, before and after tuning.

What the training actually fixed

The base model already reads these pages — 88.4% of gold values appear somewhere in its output. What it lacked was field discipline: it returns 8994/QD-CT ngay 22/11/2019 where the schema asks for 8994/QD-CT, and it fills fields the page does not contain instead of returning null. The gap between "present somewhere" and "exactly right" fell from 24.4 pp to 9.3 pp, and the substring figure rose as well — precision was not bought by dropping recall.

Multilingual behaviour

Adapting to one language usually costs the others. Measured on 60 synthetic pages held out of training, 20 per language:

	Vietnamese	Korean	English
Base model	77.8%	63.3%	75.0%
2 epochs	93.3%	92.2%	91.1%

Korean gains 28.9 pp, English 16.1 pp. The training mix was deliberately 60% Vietnamese / 20% Korean / 20% English for exactly this reason.

Held-out synthetic Vietnamese reaches 93.3% against 81.4% on real pages. That 11.9 pp difference is the distance between generated pages and genuine scanner output — the clearest remaining lever.

Pages mixing languages in one layout. No such page appeared in training — every training page carries a single language. Fourteen held-out mixed-layout pages were scored to find out what that omission cost:

Layout	Pages	Fields	Exact match
One language	6	167/180	92.8%
Two languages	6	167/180	92.8%
Three languages	2	55/60	91.7%

Mixing languages costs nothing measurable, and the 1.1 pp on three languages is two fields on two pages. Scan quality costs far more: from the same generator, a clean Vietnamese page scored 100% while a low-quality scan scored 83.3% — 16.7 pp from image quality alone against 1.1 pp from adding two languages. Self-consistency between a greedy and a sampled pass was flat across all three groups (2.5 / 2.0 / 2.5 disagreeing fields), so the model is not merely getting lucky on the mixed pages.

On these pages the headings and field labels are multilingual while the values stay in one language. This measures robustness to multilingual labels, not the reading of values in several languages on one page.

Handwritten entries

The document number and date are handwritten into dotted blanks on these forms. Those fields were in training and evaluation from the start; they were scored separately afterwards.

One row below measures handwriting. The other two measure fonts.

Evaluation set	Glyphs	Fields	Base	2 epochs
Genuine VAMC pages	real handwriting	6	33.3%	66.7%
Synthetic, first set	4 fonts	80	35.0%	58.8% withdrawn
Synthetic, after fix	2 fonts	80	22.5%	80.0% not a handwriting figure

Six fields is the entire evidence for reading handwriting — two correct became four. The synthetic rows are font renderings: glyph shapes are fixed, so every 4 across all forty pages is the same 4. They measure whether the model separates a dotted blank from the printed text beside it, which is worth measuring, but it is not handwriting recognition. Naming that section after handwriting overstated it.

The 80.0% is inflated further: DejaVuSans-Bold — a plain bold sans used for printed body text, present in the font list only as a crash-guard fallback — was selected for 17 of the 40 pages. Those pages score 91.2% because there is no handwriting on them to read. The pen-like subset scores 71.7%.

The synthetic set was gated before its numbers were used: a set easier than the real pages would make any improvement measured on it meaningless, so the base model had to fail on it comparably. It did — 35.0% against 33.3% on real pages.

The character of the errors changed more than the number did. Before tuning, the model read the dotted line itself as characters, absorbed the following sentence into the field, and on one page returned 010623885 — the business registration number printed inside the red seal — as the document number. After tuning, what remains is single-digit confusion between visually similar handwritten figures: 4 read as 9 and 3 read as 8 on real pages, 7 as 9 and 6 as 1 on synthetic pages.

The 58.8% figure has been withdrawn — it measured a defect in our own generator.

Of the twelve errors in the evaluation log, ten had the handwritten digits correct and failed on the adjacent printed suffix: 1670/AMC-DG with the V dropped. The errors were spread evenly across the four pen fonts (8 / 9 / 9 / 7), pointing to a structural cause rather than to particular handwriting.

The cause was geometric. The dotted blank allots a fixed 88 px to the handwritten document number, while four digits at the sizes the generator draws occupy 92–129 px. The overflow was painted straight over the printed /VAMC-ĐG that follows it. On five of six pages inspected the / was destroyed, and on one the V with it — so the gold string was not present in the image at all, and a model that read the page correctly was scored wrong.

The generator now measures the handwriting on a separate layer and pushes the printed suffix out to clear it. The real-page row (66.7%) is unaffected: genuine documents have the suffix printed cleanly.

The fix worked, and the proof is in what the errors became. Of twelve logged errors before, ten were the printed suffix and two were misread digits. After the fix that is reversed: two suffix errors, ten misread digits — 1251 read as 1254, 1465 as 1565, 9/2 as 4/2. The gold string is now present in the image and the model is being scored on whether it read it.

Zooming into the three genuine pages settles what the synthetic data should imitate. All three write 1 with a flag at the upper left and 7 with a crossbar — continental European forms, not the bare stroke and uncrossed seven typical of US handprint. Document numbers here are all in the 1000s, so 1 is the most frequent glyph in the field, and that is exactly where the convention diverges. Compositing US handprint crops without selecting for these forms would repeat the address- abbreviation mistake described above: training on a shape the real documents never use. Caveat: three pages from one institution, possibly few writers.

Work on this axis is paused pending genuine handwriting samples. Enlarging a font-rendered set does not help — every 4 stays the same 4.

The same defect reaches the training pages, more mildly — the base generator overruns by 4–14 px at two of its three font sizes. The model was therefore taught to emit /VAMC-ĐG from pages where the / was damaged: to complete a template rather than read one. Whether removing that changes real-page accuracy is now a testable question rather than a suspicion.

An earlier version of this set was discarded for a related reason — calligraphic fonts produced pages no human could read. A label that does not exist in the image makes its scoring meaningless, however the label came to be missing.

How a regression was found and fixed

The first training run improved the average but made one page worse — page 1348 fell from 69.0% to 65.5%. Its address fields are printed in abbreviated form (Q. Tan Phu, TP.HCM) while the first synthetic corpus always spelled the administrative prefix out (huyen ..., tinh ...). The adapter had learned to normalise toward the only form it had seen.

Adding abbreviated forms to the generator — along with varied currency units (4 variants), numbered appendix titles, three term-of-use phrasings, unnumbered mortgage contracts whose gold value is null, and certificate numbers containing spaces — lifted that page to 79.3%, above its untuned score, and every other page improved too.

All of those variations were taken from features observed in real documents, not guessed. This is the whole method: measure, find where it fails, identify why, change the data, measure again.

Intended use

Extracting structured records so a person can verify and confirm them before they reach a database. Output must be reviewed. At 81.4% exact match roughly one field in five still needs correction, and the model does not know which one.

Pair it with validation rules. The principal + interest = total check above catches row-level confusion that an accuracy figure hides.

Training

• Base: Qwen/Qwen2.5-VL-7B-Instruct, 4-bit NF4, bfloat16 compute, flash-attn 2
• LoRA r=16, alpha=32, dropout 0.05, on q,k,v,o,gate,up,down
• 47,589,376 trainable parameters — 0.57% of 8.34B
• AdamW, lr 1e-4 cosine, effective batch 4 (grad accumulation 4)
• 1,740 synthetic pages, 2 epochs, ~55 min on one H200 (about 42 GB available)
• Data: ysmeta/EVE-Train-NPL-1.1, which ships the generator source so the corpus is reproducible

extraction_prompt.txt here is the exact prompt behind every number above. The prompt matters: one line telling the model which table row the debt totals come from fixed a row-confusion error before any training happened. All reported gains are measured against the base model using this same prompt.

Why the evaluation set is not published

Evaluation uses letters VAMC posts publicly on sbvamc.vn. Those pages carry personal data — staff names and phone numbers, borrower details, collateral property addresses. Collecting them into a downloadable dataset would turn scattered public postings into a searchable corpus, which is a different act from the original publication. The evaluation set is kept private. Document numbers and the source site are cited so results stay checkable. The synthetic training data is published in full.

Limitations

• Three evaluation pages. One field is 1.16% of the score. Every figure is provisional; the per-page table shows how wide the spread is.
• One document family — VAMC valuation-request appendices. Other Vietnamese financial paperwork is untested.
• Flat 30-field schema. Real filings sometimes list many collateral assets across several pages; a flat record cannot represent that. Four fields were excluded from scoring on two pages where the document was genuinely ambiguous — for instance an address printed in both pre- and post-reform administrative forms.
• Korean and English were evaluated on synthetic pages only. No genuine Korean or English NPL document was tested.
• Mixed-language pages were tested on labels only. The fourteen pages above carry multilingual headings but single-language values. Documents whose values appear in different languages on the same page remain untested.
• Handwriting in the training data is font-rendered, not genuine handwriting. Per-glyph rotation and baseline jitter vary where each character sits, but not its shape — every 4 across all forty evaluation pages is the same glyph. Real writing varies in stroke weight, slant and pressure with every instance. The handwriting figures above are weak evidence for reading real handwriting styles.
• Only short handwritten digits were tested — document numbers and dates in dotted blanks. Free-form handwritten text does not appear in the source documents and was not evaluated.
• No source coordinates. The adapter does not output bounding boxes, so a value cannot be highlighted on the original page. Qwen2.5-VL supports grounding natively; this adapter was simply not trained for it.
• null discipline improved but is not solved.

Usage

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct", device_map="auto", dtype="bfloat16")
model = PeftModel.from_pretrained(model, "ysmeta/EVE-OCR-1.0-NPL")
proc = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

prompt = open("extraction_prompt.txt").read()
msgs = [{"role": "user", "content": [
    {"type": "image", "image": "page.png"}, {"type": "text", "text": prompt}]}]
text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
imgs, vids = process_vision_info(msgs)
inputs = proc(text=[text], images=imgs, videos=vids, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=900, do_sample=False)
print(proc.batch_decode(out[:, inputs.input_ids.shape[-1]:], skip_special_tokens=True)[0])

Render scanned PDFs to roughly 1,100–1,250 px on the short edge, matching the evaluation pages. Pages that already carry a text layer should be parsed directly — running a model over them replaces a correct answer with a guess.