What they found
The authors built JFinBench from 321 examples taken from the 2026 editions of six Japanese certification exams. The exams cover accounting, tax, financial planning, and banking. A coding agent writes the conversion code and a review page, and a human checks every example and sends corrections back to the agent. The authors tested 12 recent models. Claude Fable 5 and GPT-6 Astra scored above 93% overall, and the best open-weight model, Kimi K3.1 Pro, trailed Claude Fable 5 by 12 percentage points. When models could also use a terminal and web access, every model improved. About 4 points of gap remained, and the ranking stayed largely the same.
Why it matters
Finance rules differ by country, and many benchmarks cover them poorly. Test questions that are published often leak into training data, which makes scores look better than they should. A benchmark that can be rebuilt from each new exam edition helps keep scores honest. The authors say the pipeline lowers the effort of adding new exams. The abstract does not give details on how much effort it saves.
How it changes the race
This is an incremental but useful tool for measuring SI models in one country's financial system. It suggests that knowledge stored inside a model still matters even when the model can search the web. It also shows a clear gap between the top closed models and the best open-weight model on this test.
Safety and control
The paper is about evaluation, not safety. It does show a way to keep tests fresh against data contamination, which makes measured SI capability more trustworthy. A human still checks every example.
The abstract, by the authors
Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.
Quoted from arXiv:2611.18130. The research and its words belong to its authors.
