---
title: "MLIRBench - A Benchmark and Dataset for... | SI Paper Desk"
url: https://thesimind.si/papers/2611.14881/
description: "MLIRBench tests whether SI models truly understand compiler code. The best model labels pairs correctly 92.65% of the time but scores only 5.04/10 on..."
---

Number 2 of 12 · November 13, 2026

# MLIRBench - A Benchmark and Dataset for Compiler and Hardware-Aware Reasoning

MLIRBench tests whether SI models truly understand compiler code. The best model labels pairs correctly 92.65% of the time but scores only 5.04/10 on describing what the code actually does.

Pranava Sai Maganti, Meghasyam Peddireddy, Muzakker Hossain and 3 more · Submitted November 11, 2026 · arXiv 2611.14881v1 · cs.PL

[Read the paper on arXiv](https://arxiv.org/abs/2611.14881) · [The PDF](https://arxiv.org/pdf/2611.14881) · [The day's 12](https://thesimind.si/papers/day/2026-11-13/)

## What they found

The authors built MLIRBench, a test set for how well language models reason about MLIR programs. MLIR is a kind of intermediate code that compilers use. A multi-agent pipeline makes small changes to programs, runs them to check what really happens, and builds tasks from torch and linalg program pairs. The dataset has 1,768 validated pairs (888 linalg and 880 torch), each labeled EQUIV, DIVERGE, or ERROR. Nine models were tested, five open-weight and four proprietary. Gemini 3.1 Pro did best, with 92.65% hard-label accuracy across all 1,768 pairs, counting unsuccessful predictions as wrong. Its aggregate score was 74.18, and a judge gave its result-behavior predictions 5.04/10.

## Why it matters

It is hard to tell whether SI models understand what a program does or just repeat patterns from training. This benchmark checks answers against real execution, so the labels are not guesses. The gap between high classification accuracy and a middling behavior score suggests that picking the right label does not mean a model can predict detailed execution. The abstract does not say how the other eight models did.

## How it changes the race

This is a measurement tool, not a new capability. If it holds up, it gives labs a reproducible way to test SI reasoning over compiler code, which matters for code generation and hardware-aware optimization.

### The abstract, by the authors

In recent years, large language models (LLMs) have become increasingly popular in software engineering and have performed well on different programming tasks. However, it is still difficult to know whether these models truly understand program behavior or whether they mainly rely on patterns learned from their training data. This distinction is particularly important for intermediate representations (IRs), where operations, dataflow, and execution semantics are explicitly represented. In this paper, we introduce MLIRBench, a benchmark for evaluating how LLMs reason about MLIR programs. MLIRBench uses a multi-agent pipeline to generate program mutations, validate their behavior through execution, and construct reasoning tasks from torch and linalg program pairs. The dataset contains 1,768 validated pairs, including 888 linalg pairs and 880 torch pairs, labeled as EQUIV, DIVERGE, or ERROR. We evaluated nine models, including five open-weight and four proprietary models. Gemini 3.1 Pro achieved the highest hard-label accuracy of 92.65% across all 1,768 pairs, counting unsuccessful predictions as incorrect. Across its 1,768 scored predictions, it achieved an aggregate score of 74.18 and a judge-assigned result-behavior score of 5.04/10. These results show that strong semantic classification does not necessarily imply accurate prediction of detailed execution behavior, highlighting a gap in current LLM reasoning over compiler IRs. By exposing this gap through execution-validated program pairs, MLIRBench provides a reproducible foundation for studying semantic reasoning over compiler IRs and related tasks such as program repair and semantic classification.

Quoted from arXiv:2611.14881. The research and its words belong to its authors.

Desk score **80** out of 100

Why it was picked

- Reasoning (in the title)
- Evals (in the title)
- Agents
- Studies frontier models (Gemini)
- Cross-listed in 3 fields
- Multi-agent
- Language models

Topics

- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Code](https://thesimind.si/papers/?tag=code)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Data](https://thesimind.si/papers/?tag=data)

**arXiv ID:** 2611.14881v1
**Submitted:** November 11, 2026
**Fields:** cs.PL (Programming Languages), cs.AI (Artificial Intelligence), cs.LG (Machine Learning)
**Level:** Intermediate
**Comments:** Accepted at the 2027 IEEE/ACM International Symposium on Code Generation and Optimization (CGO 2027)

Fingerprint, SHA-256

`cd6062d73d2736c6f98ee47b7bb60a1858c4351a10eeecdeea310b7e54f5186c`

On Solana: [Proof of Mind seal, 5PD2kk...Cqj6](https://solscan.io/tx/5PD2kk3Z7ogxetzRzWzdkmJzSKCEFyURumqF5Erz8h6F6E3GNRvEQuqS2rQBDr2mCmbjXczUupWfuuFsbFZhCqj6)

### The exact text fingerprinted

{"arxiv":"2611.14881","version":1,"title":"MLIRBench - A Benchmark and Dataset for Compiler and Hardware-Aware Reasoning","authors":["Pranava Sai Maganti","Meghasyam Peddireddy","Muzakker Hossain","Kaining Gao","Farhan Ahmed","Ali Jannesari"],"day":"2026-11-13","rank":2,"score":80,"status":"written","summary":{"takeaway":"MLIRBench tests whether SI models truly understand compiler code. The best model labels pairs correctly 92.65% of the time but scores only 5.04/10 on describing what the code actually does.","found":"The authors built MLIRBench, a test set for how well language models reason about MLIR programs. MLIR is a kind of intermediate code that compilers use. A multi-agent pipeline makes small changes to programs, runs them to check what really happens, and builds tasks from torch and linalg program pairs. The dataset has 1,768 validated pairs (888 linalg and 880 torch), each labeled EQUIV, DIVERGE, or ERROR. Nine models were tested, five open-weight and four proprietary. Gemini 3.1 Pro did best, with 92.65% hard-label accuracy across all 1,768 pairs, counting unsuccessful predictions as wrong. Its aggregate score was 74.18, and a judge gave its result-behavior predictions 5.04/10.","matters":"It is hard to tell whether SI models understand what a program does or just repeat patterns from training. This benchmark checks answers against real execution, so the labels are not guesses. The gap between high classification accuracy and a middling behavior score suggests that picking the right label does not mean a model can predict detailed execution. The abstract does not say how the other eight models did.","race":"This is a measurement tool, not a new capability. If it holds up, it gives labs a reproducible way to test SI reasoning over compiler code, which matters for code generation and hardware-aware optimization.","safety":"","level":"intermediate","tags":["benchmarks","code","evals","reasoning","language-models","data"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Pranava Sai Maganti, Meghasyam Peddireddy, Muzakker Hossain, Kaining Gao, Farhan Ahmed, Ali Jannesari

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 13, 2026 · Number 6SynthCorp: A Platform for Creating Benchmarks for Data AgentsSynthCorp simulates business events over time to build fake but consistent company data and test tasks, so builders can create harder, targeted tests for SI data agents.BenchmarksAgentsEvals](https://thesimind.si/papers/2611.14765/) · [Nov 9, 2026 · Paper of the DayPhantom Consensus: Measuring Social Reasoning in Multi-Agent LLM Decision-MakingWhen SI models see ambiguous public signals from other models, they can drift to a group answer that no one privately held, and checking private judgments can catch this.Multi-agent systemsEvalsBenchmarks](https://thesimind.si/papers/2611.08971/) · [Nov 11, 2026 · Number 2Benchmarking language models and agents for scientific reasoning in evolving materials laboratoriesMatScope is a new test showing that SI models handle given calculations and physical explanations well, but struggle to tell close material states apart and to reason with numbers.BenchmarksScienceEvals](https://thesimind.si/papers/2611.12014/)

[Number 1](https://thesimind.si/papers/2611.14996/) · [All 12 for Nov 13, 2026](https://thesimind.si/papers/day/2026-11-13/) · [Number 3](https://thesimind.si/papers/2611.14838/)
