NayanHarod/LeagalAI-section-138

Section 138 Expert (Qwen2.5-1.5B, SFT + GRPO)

A small open model fine-tuned to check whether an Indian cheque bounce complaint under Section 138 of the Negotiable Instruments Act, 1881 meets the legal deadlines. It reads the case documents, reasons step by step, and returns a structured JSON answer.

It is the first test case of Expert Factory, a pipeline that turns a general open model into a domain expert using a code verifier, generated data, SFT and GRPO. Code, data pipeline and the full story: https://github.com/NayanHarod/LeagalAI-section-138-

Not legal advice. This is a research prototype trained on synthetic cases. It still makes date mistakes. Always verify its output.

Results

All results are on 120 held-out test cases. Every test case comes from a seed case that was never used in training.

Model	Score	Verdict accuracy	Correct format
Qwen2.5-1.5B-Instruct (no training)	0.000	34% (random guessing)	0 / 120
+ SFT	0.833	76%	120 / 120
+ SFT + GRPO (this model)	0.877	81%	120 / 120

The score is the average of a weighted checker. Each answer is graded on six fields: presentation, notice, cause-of-action date, complaint status, jurisdiction, and verdict. The verdict counts double.

Field	Correct (out of 120)
presentation_ok	119
notice_ok	112
cause_of_action_date	92
complaint_status	100
verdict	97
jurisdiction_city	120

The model's 120 test answers are in eval/eval_grpo.jsonl. The SFT-only answers are in eval/eval_sft.jsonl.

What the model does

Input: a fixed instruction followed by five case documents: the cheque, the bank return memo, the legal notice, postal tracking and the complaint.

Output: step-by-step reasoning inside <reasoning> tags, then a JSON answer inside <answer> tags:

Field	Values	Rule the model must apply
presentation_ok	true / false	Cheque presented within 3 months of its date
notice_ok	true / false	Demand notice sent within 30 days of receiving the return memo
cause_of_action_date	YYYY-MM-DD	The day after the 15-day payment period (from notice delivery) ends
complaint_status	premature / in_time / late	Complaint filed within 1 month of the cause of action
verdict	maintainable / condonation_required / not_maintainable	Combines the above. Only a late complaint can be condoned
jurisdiction_city	city name	Section 142(2): payee's branch if deposited through their account, otherwise the drawer's branch

The legal rules are not in the prompt. The model learned them in training.

Example

Input documents (the instruction is shown in Prompt format):

[cheque]
Cheque dated 09 June 2027, drawn on the drawer's account at the Bengaluru branch.

[return_memo]
The payee presented the cheque directly at the drawer's bank, Bengaluru branch, on 30 July 2027. The cheque was returned unpaid with the remark 'Funds Insufficient'. The payee received the return memo on 04 August 2027.

[legal_notice]
Legal notice demanding payment of the cheque amount was dispatched to the drawer by Speed Post on 04 September 2027.

[postal_tracking]
Speed Post tracking: item delivered to addressee on 07 September 2027.

[complaint]
Complaint under Section 138 of the NI Act filed on 09 October 2027.

Model output. The notice was sent one day late, a boundary case, and the model gets every field right:

<reasoning>
1. Presentation: ... It was presented on 30 July 2027, within the validity period. presentation_ok = true.
2. Notice: ... The memo was received on 04 August 2027, so the last day was 03 September 2027. The notice was dispatched on 04 September 2027, after the 30-day limit. notice_ok = false.
3. Cause of action: ... The notice was delivered on 07 September 2027, so the 15 days ended on 22 September 2027. The cause of action arose on 23 September 2027.
4. Complaint: ... It was filed on 09 October 2027, within the limitation period. complaint_status = in_time.
5. Verdict: not_maintainable, because the notice was not sent in time, and such defects cannot be cured by condonation.
6. Jurisdiction: ... the cheque was presented directly to the drawer's bank, so the court where the drawer's branch is located has jurisdiction. jurisdiction_city = Bengaluru.
</reasoning>
<answer>
{"presentation_ok": true, "notice_ok": false, "cause_of_action_date": "2027-09-23", "complaint_status": "in_time", "verdict": "not_maintainable", "jurisdiction_city": "Bengaluru"}
</answer>

Files

File	What it is
section138-expert-v3-q8_0.gguf	Full merged model, 8-bit GGUF (1.6 GB), for Ollama or llama.cpp
Modelfile	Ollama config: chat template, temperature 0, repeat penalty 1.1, stop tokens
adapters/sft/	LoRA adapter 1 (SFT), on top of Qwen2.5-1.5B-Instruct
adapters/grpo/	LoRA adapter 2 (GRPO), trained on top of the SFT-merged model
eval/	The model's answers on the 120 test cases, with scores

How to use

Option 1: Ollama (easiest)

hf download NayanHarod/LeagalAI-section-138 section138-expert-v3-q8_0.gguf Modelfile --local-dir section138-expert
cd section138-expert
ollama create section138-expert -f Modelfile
ollama run section138-expert

Then send a prompt in the format below. It takes about 8 seconds per answer on a laptop GPU.

Option 2: transformers + peft

The two adapters must be merged in order: SFT first, then GRPO. The GRPO adapter was trained on the SFT-merged model, so loading it alone on the base model gives a different, wrong model.

import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = snapshot_download("NayanHarod/LeagalAI-section-138", allow_patterns=["adapters/*"])
base = "Qwen/Qwen2.5-1.5B-Instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype=torch.float16 if device == "cuda" else torch.float32).to(device)
for name in ["sft", "grpo"]:  # order matters
    model = PeftModel.from_pretrained(model, f"{repo}/adapters/{name}").merge_and_unload()
model.eval()

prompt = "..."  # instruction + case documents, see "Prompt format"
text = tokenizer.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=700, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use greedy decoding (do_sample=False). Qwen's default generation config adds a repetition penalty of 1.1, which matches the evaluation.

Prompt format

The model was trained on one exact prompt format. In the GitHub repo, rule_engine/prompt.py builds it from the case dates, and local/api.py wraps it in an API that takes plain JSON dates.

Full instruction text

You are assisting the complainant's lawyer in a cheque bounce case under Section 138 of the Negotiable Instruments Act, 1881.

Read the case documents. First explain your reasoning step by step inside <reasoning></reasoning> tags. Then give your final answer inside <answer></answer> tags as a JSON object containing exactly these fields:
- "presentation_ok": true or false. Was the cheque presented to the bank within the legally allowed period?
- "notice_ok": true or false. Was the demand notice given within the legally allowed period?
- "cause_of_action_date": "YYYY-MM-DD". The date on which the cause of action arose.
- "complaint_status": "premature", "in_time" or "late".
- "verdict": "maintainable", "condonation_required" or "not_maintainable".
- "jurisdiction_city": the city whose court has jurisdiction under Section 142(2).

Case documents:

[cheque]
...

The five documents follow in the order shown in the example.

Training

Base model	Qwen/Qwen2.5-1.5B-Instruct
Hardware	1 × NVIDIA T4 (Kaggle, free tier)
Method	LoRA r=16, alpha=32, dropout 0.05, all linear layers
Libraries	trl 1.16.0, peft 0.21.2, transformers 5.14.1

Data. 2,400 synthetic cases were generated from 10 hand-written seed cases. Every label comes from a rule engine (the law written as code), so labels are always correct. The cases are balanced by case type: 800 valid cases and 400 each of late presentation, late notice, premature complaint and late complaint. Many cases fall exactly on a deadline or one day after it. Split: SFT 1,000 / GRPO 784 / test 616. The test cases all come from a separate held-out seed.

Step 1: SFT. 1,000 examples with worked reasoning, 1 epoch, learning rate 2e-4, effective batch size 16.

Step 2: GRPO. The SFT adapter was merged into the base model, then a fresh LoRA was trained on top.

Setting	Value
Steps	250
Answers per prompt	8
Learning rate	2e-5
Temperature	1.0
KL penalty (beta)	0.04
Rewards	checker score (weight 1.0) and consistency (weight 0.5)

The checker score grades the answer against the rule engine's answer. The consistency reward checks that the verdict follows from the model's own fields.

Why the consistency reward and KL penalty. An earlier run without them learned a shortcut: it always answered "true" for presentation and notice, and never said "maintainable". Its score went up, but it was not learning the task. Balancing by case type, adding the consistency reward and keeping the model close to the SFT model with the KL penalty removed this shortcut. This model predicts every value in roughly the right proportions, and all 120 of its verdicts match its own fields.

Limitations

• Date arithmetic and comparison. The cause-of-action date is the weakest field (92/120). Cases that fall exactly on a deadline are the hardest. Sometimes the model computes the right dates and then compares them wrongly, for example by calling 28 August "before" 22 August. A production system should give the model a date-calculation tool.
• Too cautious. Of 43 valid test cases, it calls 18 either not maintainable or condonation required.
• Sometimes too optimistic. In 4 of 120 test cases it calls an invalid complaint maintainable, which is the costly error for a complainant.
• Synthetic, template documents. It has only seen documents in one fixed template, so real case papers with different wording are out of distribution.
• Narrow scope. It handles only the deadline and jurisdiction checks of Section 138. It does not cover other defences, the amount, the debt or the evidence.
• Not reviewed by a lawyer. The rule engine's date-counting conventions have not yet been checked by a practising lawyer.

Intended use

Research and education: a worked example of verifier-driven fine-tuning (SFT + GRPO) for a narrow professional task. It is not for real legal decisions.