AravArun/dhara-reranker

Dhara reranker: Indian criminal law

A cross-encoder that scores how well a passage from a Supreme Court of India criminal judgment, or a section of the criminal codes, answers a legal question. It is bge-reranker-base fine-tuned on question–passage pairs mined from the Dhara corpus, and it is the reranker in the Dhara research tool, live at dhara.aravarun.in.

It was built for the 1 July 2024 change of codes (IPC, CrPC and the Evidence Act replaced by BNS, BNSS and BSA). Questions are asked with new-code section numbers, but most judgments cite the old ones, so the model has to judge a passage on what it says, not on the number it quotes.

Use

from sentence_transformers import CrossEncoder

model = CrossEncoder("AravArun/dhara-reranker", max_length=512)
question = "Can a conviction under BNS 103 rest on circumstantial evidence alone?"
passages = [
    "Where the case rests on circumstantial evidence, the chain must be so complete that it excludes every hypothesis but guilt...",
    "The appellant was granted anticipatory bail subject to conditions...",
]
scores = model.predict([(question, p) for p in passages])

Higher is more relevant. In Dhara it reorders the top 50 passages from a rank fusion of BM25, a citation-graph retriever and bge-m3 dense and sparse search.

Training

• Questions: written for training-split judgments and sections only, the same way as the DharaBench benchmark, so no judgment or section in dev or test is seen in training.
• Positives: passages of the source judgment that cite the provisions behind the question, or the provision's own text for statute questions.
• Hard negatives: top results from the live retriever that are not the answer. A passage from another judgment citing the same law is never used as a negative, since it is often a right answer too; for statute questions the old/new counterpart is never one.
• Size: 1,596 questions, 9,235 pairs (about 4.8 negatives per question).
• Setup: binary cross-entropy with positives weighted by the negative ratio, 2 epochs, learning rate 2e-5, effective batch 16, 512 tokens.

The first version used negatives that included same-law passages and trained for one epoch at 320 tokens. It lost to the retrieval it was reranking; removing those negatives is what made the difference.

Evaluation

DharaBench test split: 150 questions phrased with new-code numbers whose answer is a judgment citing only the old code. Right judgment in the top 10:

Setup	Recall@10
BM25, question as typed	0.27
Dense only (bge-m3)	0.31
BM25 + citation graph + bge-m3, rank fusion (citation graph x2)	0.38
LLM reranker over the fusion	0.37
First fine-tuned version over the fusion	0.36
This model over the fusion	0.43

On the dev split it scores 0.49, against 0.47 for the weighted fusion. With 150 questions per split, figures carry about ±0.04 of noise. It ranks better from the even fusion than from one that already weights the citation graph, and it does not improve statute ranking. Full tables: eval/results.

Dhara also uses its top score over the retrieved sections to decline questions outside criminal law. With a threshold fitted on dev to answer 95% of in-scope questions, it declines about half of the out-of-scope ones on test (civil, tax, family law).

Limits

• Trained on Supreme Court criminal judgments and the six criminal codes only; other courts and areas of law are out of its domain.
• English only.
• Questions were generated from headnotes, so they are cleaner than real queries.
• A ranking aid for research, not legal advice.