What they found
The authors built MLIRBench, a test set for how well language models reason about MLIR programs. MLIR is a kind of intermediate code that compilers use. A multi-agent pipeline makes small changes to programs, runs them to check what really happens, and builds tasks from torch and linalg program pairs. The dataset has 1,768 validated pairs (888 linalg and 880 torch), each labeled EQUIV, DIVERGE, or ERROR. Nine models were tested, five open-weight and four proprietary. Gemini 3.1 Pro did best, with 92.65% hard-label accuracy across all 1,768 pairs, counting unsuccessful predictions as wrong. Its aggregate score was 74.18, and a judge gave its result-behavior predictions 5.04/10.
Why it matters
It is hard to tell whether SI models understand what a program does or just repeat patterns from training. This benchmark checks answers against real execution, so the labels are not guesses. The gap between high classification accuracy and a middling behavior score suggests that picking the right label does not mean a model can predict detailed execution. The abstract does not say how the other eight models did.
How it changes the race
This is a measurement tool, not a new capability. If it holds up, it gives labs a reproducible way to test SI reasoning over compiler code, which matters for code generation and hardware-aware optimization.
The abstract, by the authors
In recent years, large language models (LLMs) have become increasingly popular in software engineering and have performed well on different programming tasks. However, it is still difficult to know whether these models truly understand program behavior or whether they mainly rely on patterns learned from their training data. This distinction is particularly important for intermediate representations (IRs), where operations, dataflow, and execution semantics are explicitly represented. In this paper, we introduce MLIRBench, a benchmark for evaluating how LLMs reason about MLIR programs. MLIRBench uses a multi-agent pipeline to generate program mutations, validate their behavior through execution, and construct reasoning tasks from torch and linalg program pairs. The dataset contains 1,768 validated pairs, including 888 linalg pairs and 880 torch pairs, labeled as EQUIV, DIVERGE, or ERROR. We evaluated nine models, including five open-weight and four proprietary models. Gemini 3.1 Pro achieved the highest hard-label accuracy of 92.65% across all 1,768 pairs, counting unsuccessful predictions as incorrect. Across its 1,768 scored predictions, it achieved an aggregate score of 74.18 and a judge-assigned result-behavior score of 5.04/10. These results show that strong semantic classification does not necessarily imply accurate prediction of detailed execution behavior, highlighting a gap in current LLM reasoning over compiler IRs. By exposing this gap through execution-validated program pairs, MLIRBench provides a reproducible foundation for studying semantic reasoning over compiler IRs and related tasks such as program repair and semantic classification.
Quoted from arXiv:2611.14881. The research and its words belong to its authors.
