Number 10 of 12 · November 12, 2026

IFRS Compliance Verification: An Open Multi-LLM Benchmark and Content-Conditioned Ensemble Classifier

The authors built a benchmark for checking financial disclosures against IFRS rules and a learned ensemble that beats the best single SI model and majority voting on the items where the models disagree.

What they found

The authors built a benchmark of 19,998 pairs, each linking an IFRS regulation paragraph to a company disclosure. Of these, 11,161 have labels where several large language models all agreed, and 8,837 have human labels on items where the models disagreed. They then trained a classifier that combines text embeddings (numeric summaries of meaning) with the probability scores from GPT-4.1, Claude Sonnet 4 and Gemini 2.5 Pro. On the disagreement set, it reaches 81.7% weighted F1. That is 10.4 points above the strongest single model and 14.6 points above majority voting. It also beats eleven unsupervised ways of combining model outputs. The retrieval part, which finds the relevant rules, gets Recall@15 of 45.3% on regulations it had not seen.

Why it matters

Checking disclosures against hundreds of detailed rules is slow, costly and error-prone work for auditors. The abstract says single SI models are not accurate enough for high-stakes use, and that simply voting across models can fail when each model has its own consistent bias. A learned combiner that handles this could make SI compliance tools more dependable. The authors say they will release the dataset, checkpoints, annotation platform and code. The abstract does not say how the system would work in a real audit.

How it changes the race

This is a focused, applied result. It suggests that combining several frontier models with a trained classifier can beat any one of them on a hard professional task. The public benchmark could let smaller teams test compliance tools without building their own data.

Safety and control

The paper bears on reliable use of SI in high-stakes settings. It shows that model errors can be correlated, so simple voting is not a safe fix. The abstract gives no detail on how errors would be caught in real audits, and the 81.7% score still leaves room for mistakes.

The abstract, by the authors

Verifying corporate financial disclosures against the International Financial Reporting Standards (IFRS) is a core auditing task in which experts cross-reference disclosures with hundreds of fine-grained regulatory paragraphs: a laborious, costly, and error-prone process that motivates growing interest in large language models (LLMs) as compliance assistants. Yet single-model accuracy remains insufficient for high-stakes use, and naive ensembling is vulnerable to systematic per-model biases that can drag aggregate accuracy below that of the strongest individual annotator. This gap is addressed with a multi-stage framework, supported by a new benchmark of 19,998 IFRS regulation-disclosure pairs comprising 11,161 unanimous multi-LLM labels and 8,837 human annotations on items where the LLMs disagreed. The framework couples a three-stage domain-adapted sentence embedding pipeline (BERT masked language modeling on SEC EDGAR filings, unsupervised TSDAE and SimCSE adaptation, and supervised contrastive alignment) with a learned ensemble classifier that fuses frozen embeddings and LLM probability features from GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Pro via Tree-structured Parzen Estimator (TPE) optimized gradient-boosted trees. On the disagreement set, the ensemble achieves 81.7% weighted F1, exceeding the strongest single LLM by 10.4 points and majority voting by 14.6 points, and outperforms eleven unsupervised aggregation baselines; the retrieval pipeline attains Recall@15 of 45.3% on unseen regulations. The dataset, model checkpoints, annotation platform, and classifier implementation will be publicly released.

Quoted from arXiv:2611.13595. The research and its words belong to its authors.

The authors

Aghoghomena Akasukpe, Li Yang, Xinyao Zhou, Chen Shen, Miguel Vargas Martin

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.