gemma3_12b_deedexp

Reads the OCR text of a Karnataka property registration document (Kannada + English) and returns one JSON record: buyers, sellers, property details and document details.

A fine-tune of Gemma 3 12B-it in three stages:

1. domain-adaptive pre-training on registered-document text — 1,771 documents, ~16.0M tokens, every validation document excluded;
2. an instruction bridge stage that teaches what the document wording means (roles, identifiers, dates);
3. supervised extraction training across seven prompt styles, each also in a two-request split form (parties / transaction + property).

This repo carries the unquantised bf16 merged weights (~23 GB). Running them needs roughly 24 GB of VRAM; on smaller cards load in 4-bit. Quantised builds follow.

Status: for testing. No aggregate benchmark yet — see Evaluation. Contact: kamananand98@gmail.com

Prompts

One request per document: the user message is the prompt text + "\n\n" + the document's OCR text. Greedy decoding, up to 2,500 new tokens, one <bos> (the chat template already adds it).

file	use
prompts/micro_full.txt	single-request default — returns the whole record
prompts/micro_parties.txt + prompts/micro_txn_property.txt	split mode: two requests on the same OCR; merge the two JSON objects
prompts/short_full.txt, prompts/medium_full.txt	longer wordings of the same rules

All 21 prompts are shipped: 7 styles (micro, brief, short, medium, long, fields, user) × 3 scopes (full, parties, txn_property). The model was trained on all of them with the style drawn independently per row, so any style works with any scope. A split request must use the split prompts, never the full prompt — each returns only its own keys.

micro is carried over from the earlier 4B benchmark, where it won on the same validation set. It has not been re-benchmarked at 12B, and neither has single vs split. Treat the choice as inherited, not established.

pip install "transformers>=4.50" torch accelerate
python inference.py document_ocr.txt                  # micro_full, one request
python inference.py document_ocr.txt --split          # parties + txn_property, merged
python inference.py document_ocr.txt --prompt short   # any of the 7 styles

Output schema

{
 "buyer_details":  [{"name", "gender": "Male|Female|null", "father_name", "aadhaar_number", "pan_card_number", "address"}],
 "seller_details": [ ...same fields... ],
 "property_details": {"schedule_c_property_address", "sale_consideration", "registration_fee"},
 "document_details": {"transaction_date": "YYYY-MM-DD"}
}

Main rules it is trained on:

• Roles come from what the document says each party does, not print order. Confirming parties and consenting witnesses are not sellers. A promoter or developer is not a seller unless the document names it as vendor, seller or owner, or it is the only transferring party.
• transaction_date = the registration date from the Sub-Registrar endorsement, not the execution date.
• father_name of a married woman = her husband when her own description gives him.
• Text is copied as printed — never translated, never normalised. An OCR character error in a name is still copied; only genuinely missing or incomplete information becomes null.
• Identifiers are stricter: a malformed, masked or partial Aadhaar or PAN becomes null and is never repaired.

Evaluation

The 60-document held-out run has not been completed, so there are no aggregate F1 figures yet. What follows is everything that has been measured, including the failures.

Party extraction on documents the previous 4B model got wrong

Five held-out documents, chosen because the earlier 4B model mis-assigned parties on them — putting a seller in the buyer list or the reverse. Each run twice, single-request and split.

document	earlier 4B behaviour	this model
2 buyers / 1 seller	roles swapped	roles correct
1 buyer / 10 sellers	collapsed to 9 buyers / 7 sellers	exact, both modes
4 buyers / 1 seller	swapped to 2 / 2	exact, both modes
1 buyer / 5 sellers	swapped; a later attempt returned invalid JSON	exact, both modes
1 buyer / 9 sellers	an earlier attempt inverted the document	follows the Party Type: (Executant) tag (note below)

No role swap occurred on any of the five, in either mode, and single-request matched split everywhere — the split form was not needed to get these right. One name was copied with a mis-rendered initial.

Note on the last row: OCR flattens the registration table, so a party arrives as a bare Party Type: (Executant) line with no table header or column context anywhere in the document. The model treats that as a statement that the person executed the document, which is the only reading the text supports. A rule that depends on telling table from body is not implementable from this input.

JSON schema: the prompt is required

Three documents, three prompt conditions:

condition	valid JSON	schema correct
no schema given, "return JSON"	3/3	0/3
shipped prompt (schema described)	3/3	3/3
one worked JSON example, no description	3/3	3/3

Without a schema the model invents its own keys — singular buyer/seller, an invented currency, a field this schema does not contain, and in one case the whole record collapsed into a single key. Use the shipped prompts. A single worked example works as well as the written schema, so the shape can be demonstrated rather than described.

Identifiers

Eight identifiers, valid and invalid: 8/8 judged correctly — a well-formed PAN and Aadhaar accepted; a wrong-shape PAN, a PAN with the letter O in a digit position, a lowercase PAN, an 11-digit Aadhaar, a 16-digit Aadhaar and a masked Aadhaar all rejected. Two of the eight explanations were wrong although the verdicts were right, so the stated reasoning should not be relied on as an audit trail.

Long-range mapping. Where parties are named on one page and their identifiers appear many pages later as a bare numbered list with no names, the model mapped them to the right people. Given five sellers and only four identifiers it assigned four and left the fifth null. No identifier was fabricated or shifted in any test.

Dates

Asked about a document executed on one date, presented on another and registered on a third, it returned the registration date, with the correct reasoning.

Kannada domain knowledge — strong on documents, unreliable on law

Strong where it matters for extraction. Answers in Kannada distinguish sale, gift and partition deeds by consideration and name the parties to each correctly, explain ಚಕ್ಕುಬಂದಿ and ಅಳತೆ, translate registration and stamp-duty vocabulary in both directions, and give real reasons a consenting party appears in a deed (family settlement, a share in the property, an existing claim).

Unreliable on general legal procedure — do not use it as a legal reference. Asked open questions beyond document reading it produced confident but wrong answers: an invented registration deadline ("within 7 days"), a circular and incorrect account of whether a power-of-attorney holder is the seller — which also contradicts the rule it was trained on — an incoherent list of who may sell a minor's property, and one question answered with a preamble and no content. Occasional glosses are garbled.

Treat it as a document reader, not an adviser.

Weaknesses found

• Indian currency handling is weak. Given four separate part-payments and no stated total, it listed every payment correctly with its date and mode but summed them wrongly, by roughly a factor of ten. Copied amounts are reliable; derived or summed amounts are not, and should be recomputed downstream.
• Addresses are the weakest field — observed dropping a project-name prefix, reordering components, and omitting a PIN.
• Output shape drifts whenever the prompt does not carry the schema (above).
• Training loss was still falling when training ended, and the checkpoint was not selected on a validation curve.

Serving and throughput

serving/ has everything needed to reproduce this on a single 16 GiB GPU: exact hardware, driver and package versions, both serving paths, the quantization and serve commands, and the pitfalls that cost us time.

path	documents/hour	context	notes
plain transformers, 4-bit (NF4)	114 sequential	full 20,480	simplest; no quantized artifact needed
vLLM, W4A16, 4 concurrent	480 (4.2x)	12,288	needs a quantized build; long documents must be routed elsewhere

Measured on an RTX 5060 Ti (16 GiB), 20 held-out documents, warm-up discarded. Valid JSON on every request at every concurrency, and no output lost to batching (abort/error/repetition counters all zero).

The binding constraint is KV cache, not weights: vLLM charges 6.57 GiB for a single 20,480-token sequence, so even with 4-bit weights the pool holds ~28,500 tokens — about 3-4 documents of median length. W8A16 leaves too little pool to batch at all on 16 GiB and is only useful on a larger card. See serving/REPRODUCE.md.

Limitations

• Trained with a 12,288-token context. About 4.8% of our own corpus is longer (up to 20,453 tokens); behaviour above the trained window is largely untested.
• Works on OCR text, so OCR errors in identifiers and names carry through. Always check extracted identifiers against the document.
• Trained on Karnataka property transfer documents. Other document types and other states are out of scope.
• The training documents contain personal data. Use only on documents you are authorised to process.

License

Derivative of Gemma 3, distributed under the Gemma Terms of Use; use is subject to the Gemma Prohibited Use Policy.