Numbers hide in prose
Capital ratios, margin requirements and filing deadlines appear as words inside clauses, not cells inside tables. Extraction requires precision reading, then arithmetic.
A research benchmark & open dataset · EMNLP 2026 submission
Can language models read India’s financial law?
IndiaFinBench is the first evaluation benchmark for large language models on Indian financial regulatory text — 406 expert-annotated questions drawn from 192 SEBI and RBI documents spanning thirty-four years of the regulatory record.
0expert-annotated questions
0RBI & SEBI documents
0frontier models evaluated
0Recall@5, hybrid retrieval
0best model accuracy
0human expert baseline
The Problem
Benchmarks read English. Regulation reads differently.
Financial NLP benchmarks — FinQA, TAT-QA, FiQA — are built almost entirely on Western corporate filings. None test whether a model can navigate the regulatory apparatus governing the world’s most populous market. Indian regulation poses three difficulties that existing benchmarks never measure.
Numerical thresholds are buried in dense statutory prose, often written out in words. Circulars supersede one another in long chains, so the operative rule at a given date is a temporal-reasoning problem, not a lookup. And the vocabulary — LODR, PMLA, AIF, FEMA, SFB — is jurisdiction-specific, thinly represented in Western training corpora.
Circulars supersede circulars
A 2019 master circular may amend a 2014 directive that replaced a 1998 notification. Answering “what rule applied in 2016?” means untangling the chain.
The vocabulary is sovereign
LODR is not a typo and an AIF is not a hedge fund. Jurisdiction-specific terms of art carry exact legal meanings that general-purpose corpora rarely teach.
The listed entity shall submit quarterly and year-to-date standalone financial results to the stock exchange within forty-five days of end of each quarter, other than the last quarter, as per the requirements of Regulation 33.
The Corpus
Thirty-four years of primary sources, read in full.
Every question in IndiaFinBench traces to a primary-source document published by the Securities and Exchange Board of India or the Reserve Bank of India between 1992 and 2026 — circulars, master directions, notifications and regulations, collected with full source URLs and parsed into a queryable corpus.
- SEBI documents
- 92
- RBI documents
- 100
- Document span
- 1992–2026
- Indexed chunks
- 4,347
- Chunk size
- 1,600 chars
- License
- Public domain (GoI)
The Benchmark
From circular to question: a dual-validated construction.
- Collect192 primary documents, 1992–2026, with source URLs
- AuthorQA pairs drafted against exact passages, four task types
- Validatemodel check on answerability — 90.7% agreement, κ = 0.918 (CON)
- Adjudicatehuman IAA on 180 items across 3 rounds — 77.2% agreement, κ = 0.645 (CON)
- Release406 items, CC BY 4.0, with per-item difficulty labels
Difficulty assigned at authoring time by reasoning depth — hard means multi-instrument tracking or compound arithmetic.
The Evaluation
Twelve models. Zero shots. One human baseline to beat.
Dashed rule marks the human expert baseline for the selected metric (n = 100). Paired bootstrap over all 66 model pairs resolves three statistically distinct tiers.
| # | Model | Params | Access | REG | NUM | CON | TMP | Overall | 95% CI |
|---|
| Model | Easy n=160 | Medium n=182 | Hard n=64 | Δ |
|---|
The Findings
What 4,872 graded answers say about regulatory reasoning.
3 tiers
Performance is tiered, and the tiers are real
Paired bootstrap (10,000 resamples, all 66 pairs) separates frontier API models (81–90%), mid-tier open-weight models (75–79%), and a small-model floor at 70%. Most cross-tier gaps are significant at p < 0.05.
17B ≈ 70B
Scale alone does not buy regulatory reasoning
Llama 4 Scout 17B statistically matches LLaMA-3.3-70B (p = 0.79) with a quarter of the parameters — and GPT-OSS 120B is indistinguishable from GPT-OSS 20B (p = 0.91, Δ = +0.3 pp).
35.9 pp
Numerical reasoning is the discriminator
NUM shows the widest spread of any task — from 84.8% (Gemini 2.5 Flash) down to 48.9% (Gemini 2.5 Pro). If you want to tell models apart, ask them to do arithmetic inside statute.
48.9% vs 89.7%
Capability dissociates within a single model
Gemini 2.5 Pro ranks first on regulatory interpretation yet last on numerical reasoning — task-type performance can split wide open inside the same weights.
11th / 12
Reasoning-specialised ≠ timeline-capable
DeepSeek R1 70B, built for chain-of-thought, ranks 11th overall and manages only 70.5% on temporal reasoning — general deliberation does not transfer to supersession chains.
12 / 12 > human
Every model beats the human baseline
Human experts score 69.0% (n = 100, CI [59.4, 77.2]). All twelve models exceed it — yet the best still miss one answer in ten, in a domain where the answer is a legal obligation.
The Retrieval
The benchmark closes the book. The retrieval system opens it.
IndiaFinBench evaluates closed-book reading. Its open-book counterpart is a hybrid retrieval system over the same 192-document corpus: dense semantic search and sparse lexical search run in parallel, fused by reciprocal rank — because regulatory text, saturated with citation identifiers, rewards exact matching as much as meaning.
| Configuration | Recall@5 | MRR | p50 latency |
|---|---|---|---|
| Dense only B0 | 0.688 | 0.542 | 48 ms |
| BM25 only B1 | 0.764 | 0.674 | 30 ms |
| Hybrid RRF B2 selected | 0.785 | 0.640 | 77 ms |
| Small chunks, 800 chars B3 | 0.583 | 0.493 | 138 ms |
| Large chunks, 2,400 chars B4 | 0.542 | 0.410 | 71 ms |
| Hybrid, k = 10 B5 | 0.785 | 0.640 | 78 ms |
Retrieving from corpus…
The Access
An open dataset, an open leaderboard, an open invitation.
Context
Question
Gold answer
{% raw %}@article{pall2026indiafinbench,
title = {{IndiaFinBench}: An Evaluation Benchmark
for LLM Performance on Indian Financial
Regulatory Text},
author = {Pall, Rajveer Singh},
journal = {Proceedings of EMNLP},
year = {2026},
url = {https://github.com/Rajveer-code/IndiaFinBench}
}{% endraw %}