What they found
The authors build a framework that looks at what happens inside a model when it works through a benchmark. It combines attention signals, activation signals and measures of internal representations. The goal is to place benchmarks on a range from retrieval-dominant (recalling facts) to reasoning-dominant, and to reveal failure modes that a single score hides. They test 16 LLMs on 24 diverse benchmarks and run deeper internal-state checks on chosen models and tasks. They report systematic differences in processing between knowledge-centric and reasoning-centric tasks. The abstract gives no specific numbers on how large these differences are.
Why it matters
Most model tests report one score per task. That shows what a model achieves but not how it got there. This work asks what kind of processing sits behind a score. If it holds, people could choose benchmarks more carefully and pick models with more insight into what each one is good at. The authors describe these benefits as potential, not proven. The code is open source.
How it changes the race
This is mainly an evaluation tool, so it is incremental for capability. It could help labs and users judge which benchmarks truly test reasoning and which mostly test recall.
Safety and control
By surfacing failure modes that task-level scores conceal, this kind of analysis could help people understand SI model behavior better. The abstract does not make direct safety claims.
The abstract, by the authors
Current evaluations of Large Language Models (LLMs) report performance at the dataset or task level, revealing what a model achieves but offering little insight into the internal processing associated with its performance. We propose a task-centric framework for characterising the model-internal processing elicited by a benchmark. It combines attention- and activation-based signals with representation-based measures to characterise benchmarks spanning a continuum from retrieval-dominant to reasoning-dominant and to surface failure modes that task-level scores conceal. We evaluate 16 LLMs across 24 diverse benchmarks and conduct targeted internal-state analyses on representative models and tasks, observing systematic differences in processing between knowledge-centric and reasoning-centric tasks. We shift evaluation from asking what score a model achieves toward characterising the processing patterns associated with that score, potentially supporting more targeted benchmark selection and more informed model choice. We open source our code at https://github.com/extraordinarylab/trace-kr.
Quoted from arXiv:2611.17461. The research and its words belong to its authors.
