Aadhi852/lightonocr-w9-finetuned

LightOnOCR-2-1B — W-9 Form Extraction

Fine-tuned version of LightOnOCR-2-1B-base for structured field extraction from IRS W-9 tax forms.

Model Description

• Base model: lightonai/LightOnOCR-2-1B-base
• Fine-tuning method: LoRA (r=16, alpha=32) via PEFT
• Training hardware: NVIDIA RTX 4000 Ada (24GB)
• Precision: bfloat16
• Output format: Minified JSON (not Markdown)
• Organization: Span Technology Services

Training Details

Parameter	Value
Epochs	3
Train samples	439
Val samples	61
Learning rate	2e-4
Effective batch size	8
LR scheduler	Cosine
Optimizer	AdamW 8-bit
Final train loss	~0.15

Evaluation Results

Evaluated on 61 held-out synthetic W-9 forms.

Metric	Baseline (base model)	Fine-tuned	Target
CER (Character Error Rate)	1.0000	0.0353	< 0.05
JSON Parse Rate	0.0000	1.0000	> 0.95
Field Exact Match	0.0000	0.9594	> 0.90
Empty Field Accuracy	1.0000	1.0000	> 0.95
Hallucination Rate	0.0000	0.0000	~0.00

Per-Field Accuracy

Field	Accuracy
form_type	1.0000
form_version	1.0000
line_1_name	1.0000
line_2_business_name	0.8361
line_3a_tax_classification	1.0000
line_3a_llc_class	1.0000
line_3a_other_desc	1.0000
line_3b_foreign_partners	1.0000
line_4_exempt_code	0.8852
line_4_fatca_code	0.9672
line_5_address	1.0000
line_6_city_state_zip	1.0000
requester_name_address	0.8033
line_7_account_numbers	0.7377
tin_type	1.0000
social_security_number	1.0000
employer_identification_number	1.0000
signature	1.0000
date	1.0000

Dataset

Trained on 500 synthetic W-9 forms generated programmatically:

• Official IRS W-9 PDF filled via PyMuPDF
• Realistic fake data via Faker (names, addresses, TINs)
• Rendered to PNG at 200 DPI
• Ground truth JSON derived directly from fill record (no re-OCR)
• No real PII used anywhere in training

Intended Use

• Extracting structured fields from IRS W-9 forms
• Self-hosted tax form processing pipelines
• Drop-in replacement for cloud VLM APIs in document extraction

Limitations

• Trained on synthetic data only — real-world scan quality may vary
• Optional fields (line_7_account_numbers, requester_name_address) have lower accuracy due to sparse training examples
• Designed for W-9 forms only — not tested on other form types

How to Use

import torch
from transformers import LightOnOcrProcessor, LightOnOcrForConditionalGeneration
from PIL import Image

model_id = "Aadhi852/lightonocr-w9-finetuned"

processor = LightOnOcrProcessor.from_pretrained(model_id)
model = LightOnOcrForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

image = Image.open("w9_form.png").convert("RGB")
prompt = "Extract all fields from this form and return a valid JSON object only. Do not include any explanation or additional text."

messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": prompt},
]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
encoding = processor(images=image, text=text, return_tensors="pt")
encoding = {k: v.to(model.device) for k, v in encoding.items() if hasattr(v, 'to')}

with torch.no_grad():
    output_ids = model.generate(**encoding, max_new_tokens=512, do_sample=False)

input_len = encoding['input_ids'].shape[1]
result = processor.decode(output_ids[0][input_len:], skip_special_tokens=True)
print(result)  # {"name": "...", "tin": "...", ...}

Citation

If you use this model, please cite the base model:

• LightOnOCR-2-1B by LightOn