Papers
arxiv:2611.18130

JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams

Published on Nov 13
· Submitted by
Issa Sugiura
on Nov 16
Authors:
,

Abstract

Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.

Community

Paper author Paper submitter

JFinBench is a Japanese financial benchmark comprising 321 examples from six certification exams held in 2026. Building benchmarks from public exam PDFs is challenging due to heterogeneous layouts, complex tables and figures, and diverse question formats. We address this with a human-in-the-loop agentic pipeline: a vision-language model converts PDFs into layout-preserving Markdown, a coding agent generates exam-specific conversion code and review interfaces, and humans verify all examples and provide feedback for corrections. This reduces verification to roughly two hours per exam, enabling practical benchmark updates as new exams are released.

Evaluating 12 LLMs, we find that leading proprietary models exceed 93%, outperforming the best open-weight model by 12 percentage points, mainly due to gaps in specialized knowledge. Web and terminal access improves all models and narrows the gap, but a 4-point difference remains, with rankings largely unchanged (Spearman ρ = 0.87). These findings highlight the importance of knowledge encoded in model parameters, even when external tools are available.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.18130
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.18130 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.18130 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.