DocFields-F32-KO

πŸ‡°πŸ‡· ν•œκ΅­μ–΄ μΉ΄λ“œ: MODEL_CARD.ko.md

Status: EXPERIMENTAL / RESEARCH PREVIEW. Quality verdict: F32_TARGETS_MISSED β€” this checkpoint met 6 of its 7 pre-registered development targets and missed the phone-number target. It is published as a research artifact, not as a product. There is no accuracy guarantee for production documents.

DocFields-F32-KO labels the OCR boxes of a Korean financial form (claim forms, applications, confirmations) with a field role: address, phone number, account number, claim amount, issue date, other dates and so on. It takes text boxes and their coordinates; it does not read pixels and it does not include OCR. It is a full fine-tune of LiLT + XLM-RoBERTa with a 15-label token-classification head.

What F3.2 changed over its predecessor: one checkpoint now recovers generic dates and account numbers that the previous single model mostly dropped, without calling a second model or a rule engine at inference time.

0. How to use

Input is a list of OCR boxes for one page. bbox is [x0, y0, x1, y1] normalized to 0..1 (x to the right, y down).

pip install -r requirements.txt
python docfields_f32.py --selftest
python docfields_f32.py --model . --example examples/synthetic_form.json
from huggingface_hub import snapshot_download
import sys
d = snapshot_download("ThakiCloud/DocFields-F32-KO"); sys.path.insert(0, d)
import docfields_f32 as df

model, tok, labels, dec = df.load(d, device="cpu")
boxes = [{"text": "κ³„μ’Œλ²ˆν˜Έ", "bbox": [0.44, 0.31, 0.56, 0.34]},
         {"text": "000-00-000000", "bbox": [0.60, 0.31, 0.84, 0.34]}]
for r in df.predict(model, tok, labels, dec, boxes):
    print(r["text"], "->", r["label"])

Things you must keep the same as training, all handled by docfields_f32.py:

  • One box is one unit. Only the first sub-token of each box carries the prediction; at most 24 sub-tokens per box.
  • Pages longer than 510 sub-tokens are split on box boundaries, without overlap.
  • Decoding is not plain argmax. decode.json holds {"tau_date": null, "tau_acct": 0.25}: a box whose argmax is not a date or account label is promoted to ACCOUNT_NO when p(ACCOUNT_NO) β‰₯ 0.25. Date promotion is off. The numbers below were measured with this decoding. Plain argmax or the stock pipeline() gives different results.
  • Reading order matters to the encoder. Feed boxes top-to-bottom, left-to-right, as your OCR emits them.

What the shipped example returns

examples/synthetic_form.json is a hand-written 19-box claim form. On CPU the released weights label it as follows (field keys such as "μ„±λͺ…" come out as KEY):

Box text Label
010-0000-0000 PHONE
μ„œμšΈνŠΉλ³„μ‹œ κ°€λ‚˜κ΅¬ λ‹€λΌλ‘œ 123, 4μΈ΅ ADDRESS
1,250,000원 CLAIM_AMOUNT
κ°€λ‚˜μ€ν–‰ BANK_NAME
000-00-000000 ACCOUNT_NO
2025λ…„ 3μ›” 10일 (signing date) ISSUE_DATE
2025. 3. 4. (the value next to "μ‚¬κ³ μΌμž", accident date) ISSUE_DATE β€” arguably should be OTHER_DATE
홍길동 (applicant name) O β€” missed

One easy form is a smoke test, not evidence of accuracy. It does show two typical weak spots: issue date versus other dates, and labels that were learned from synthetic forms only.

Labels (15)

O, KEY, APPLICANT_NAME, RRN_OR_BIRTH, ACCOUNT_NO, BANK_NAME, CONTRACT_NO, INS_PERIOD, CLAIM_AMOUNT, CONSENT_MARK, SIGNATURE, ISSUE_DATE, PHONE, ADDRESS, OTHER_DATE

Only six of these were trained and scored on real forms: ADDRESS, PHONE, ACCOUNT_NO, CLAIM_AMOUNT, ISSUE_DATE, OTHER_DATE. The others (APPLICANT_NAME, RRN_OR_BIRTH, BANK_NAME, CONTRACT_NO, INS_PERIOD, SIGNATURE, and nearly all of KEY) were learned from synthetic forms only, and CONSENT_MARK had no training examples at all. Treat those outputs as unverified.

1. What was measured

All numbers come from one evaluation set, called sealed7_fin in our ledger: 204 financial-form documents (198 with complete labels). Read section 2 before using any of them.

Metric (sealed7_fin) F3.2 (this model) Previous single model (F30m) Previous 3-model routing (F3.1) Target Met
Macro-F1 over 5 roles 0.9035 0.879 0.890 β‰₯ 0.890 yes
Date detection recall 0.703 0.273 0.606 β‰₯ 0.606 yes
Date detection precision 0.417 0.417 0.337 β‰₯ 0.337 yes
ACCOUNT_NO F1 0.797 0.694 0.767 β‰₯ 0.767 yes
CLAIM_AMOUNT F1 0.9231 0.9231 – β‰₯ 0.9181 yes
ADDRESS F1 0.9978 0.9985 – β‰₯ 0.9935 yes
PHONE F1 0.9147 0.9219 – β‰₯ 0.9169 no
ISSUE_DATE F1 0.885 0.857 – – –
  • The five roles in macro-F1 are account number, address, claim amount, issue date and phone.
  • "Date detection" asks whether a box holding a complete date string gets any date label (ISSUE_DATE or OTHER_DATE). The reference is a deterministic date-format checker, not a human. Precision of 0.417 means more than half of the boxes the model calls a date are not complete dates by that checker: 0.79 false date boxes per document.
  • The phone target was the previous model's score minus 0.005. F3.2 lost 0.007. Every phone error we inspected was a confusion between PHONE and ACCOUNT_NO. That is consistent with the account-number recovery pulling some phone numbers along, but we did not isolate the cause.

Paired bootstrap over documents (2,000 resamples, 95% interval):

Difference vs previous single model (F30m) vs 3-model routing (F3.1)
Date detection recall +0.430 [0.329, 0.534] +0.097 [0.018, 0.181]
Date detection precision – +0.081 [0.044, 0.121]
ACCOUNT_NO F1 +0.104 [0.016, 0.219] +0.030 [βˆ’0.016, 0.087]
Macro-F1 – +0.014 [βˆ’0.016, 0.042]

Against the three-model routing, macro-F1 and account F1 intervals include zero. We do not claim F3.2 is better than the routing on those; we claim it reaches a similar level with one checkpoint. All rows were scored by us with the same code and the same files.

Development gate that failed

Checkpoint selection used a separate development split (823 documents) with seven gates. The final checkpoint passed six. It failed the claim-amount gate (0.8946 against a floor of 0.9013). It is the last-round checkpoint, evaluated once as the pre-registration required, and reported as is. On that development split it scored macro-F1 0.858, date recall 0.691 and date precision 0.166 β€” noticeably lower precision than on sealed7_fin.

On held-out synthetic forms F3.2 scored 0.843 against 0.844 for the previous model, i.e. no change.

2. Limits β€” observed improvement is not proven generalization

Please read the numbers above with all of the following in mind.

  1. The gold labels are pseudo-labels. They were produced by a large language model (an internal Qwen-family deployment we call "marm"), not by human annotators. The scores measure agreement with that labeler.
  2. sealed7_fin is an exposed evaluation set. It was used to evaluate earlier versions (F2.7, F3.0, F3.1) and was opened twice during F3.2 work. It is not a fresh test set and cannot support a release decision.
  3. The initial weights had seen evaluation groups. F3.2 was initialized from F30m, whose training data overlapped 259 form groups (same template family) with the evaluation splits. F3.2's own training documents were filtered to zero id and zero group overlap, but exposure through the initial weights is not removed.
  4. Date detection recall is not an independent check. The same deterministic date checker was used as a condition when building training labels.
  5. No external evaluation exists. We have no independent set of real customer forms with human labels (E3 in our ledger is DATA_UNAVAILABLE). Accuracy on your forms is unknown.
  6. Synthetic-form scores are not real-form scores, and are reported separately for that reason.
  7. Inputs are OCR boxes. OCR errors, different box granularity (line versus word), rotated or multi-column pages and forms outside insurance/banking paperwork were not measured.
  8. Numbers were measured on GPU with bf16 autocast. CPU fp32 inference, which the script defaults to, can differ in the last decimals.

What this release does not claim: readiness for production, accuracy on real forms, generalization to unseen templates or issuers, or superiority over the three-model routing on metrics whose interval includes zero.

3. How it was trained

  • Base: nielsr/lilt-xlm-roberta-base, full fine-tune, 15 labels. Initialized from our previous checkpoint (F30m); the 14 existing head rows were copied and only the new OTHER_DATE row was freshly initialized.
  • OTHER_DATE label restored. The predecessor had one date label (issue date) and pushed every other date to O, which taught the model to ignore dates. F3.2 gives non-issue dates their own label.
  • Uncertain boxes are masked out of the loss instead of being forced to O: boxes where the pseudo-label and the date checker disagree, and account-shaped strings the pseudo-labeler did not mark. About 1,200 boxes were masked this way (another 3,147 carried a pseudo-role that has no label in this head and were masked as before).
  • Date/account discrimination. Account-shaped strings next to a contract-number, resident-number or phone key are kept as hard negatives for ACCOUNT_NO.
  • Distillation arm (the released one). For masked boxes, labels from earlier specialist models were used only when they passed a confidence condition; confirmed boxes were never overwritten. No new LLM labeling was done.
  • Data: 1,059 real financial-form documents (567 more were excluded because their form group appeared in an evaluation split) replayed twice, plus 4,121 synthetic forms. AdamW, lr 2e-5, batch 8, 3 epochs, warmup 6% then linear decay, bf16. Two arms Γ— two seeds Γ— two rounds; this is arm B, seed 29, epoch 1 of round 2, chosen on the development split by a rule fixed before training.

4. Data and license

  • Weights and code: MIT. Base model nielsr/lilt-xlm-roberta-base: MIT.
  • Training data source: AI ν—ˆλΈŒ(AI-Hub) β€” OCR 데이터(금육 및 λ¬Όλ₯˜) (71301), and Korean statutory form templates (law.go.kr) filled with synthetic values.
  • The data is not redistributed. The AI-Hub dataset is for the approved applicant's internal use only and may not be redistributed. This repository contains weights, tokenizer, label names, the decoding threshold, code and one hand-written synthetic example β€” no dataset image, text, label, crop, subset or sample.
  • Training forms contain personal-information fields. The model predicts field roles; do not use it to harvest personal data without a lawful basis.

5. Files

model.safetensors (sha256 2bc3269f764d248666414608dda10d4c836336e2eaadb289db5de98c54829f9f), config.json, tokenizer.json, tokenizer_config.json, special_tokens_map.json, labels.json, decode.json, docfields_f32.py, requirements.txt, examples/synthetic_form.json, LICENSE, NOTICE.

Downloads last month
36
Safetensors
Model size
0.3B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for ThakiCloud/DocFields-F32-KO

Finetuned
(33)
this model

Collection including ThakiCloud/DocFields-F32-KO