---
title: "Publications as benchmarks: auditing AI... | SI Paper Desk"
url: https://thesimind.si/papers/2611.11796/
description: "The authors turn published papers into auditable tests for SI agents, and on spatial transcriptomics alignment the best agents matched or beat all baseline..."
---

Number 3 of 12 · November 11, 2026

# Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment

The authors turn published papers into auditable tests for SI agents, and on spatial transcriptomics alignment the best agents matched or beat all baseline methods on 80% of 121 tasks.

Yiqun T. Chen, Jan Matthias, Jianing Yao and 2 more · Submitted November 9, 2026 · arXiv 2611.11796v1 · q-bio.GN

[Read the paper on arXiv](https://arxiv.org/abs/2611.11796) · [The PDF](https://arxiv.org/pdf/2611.11796) · [The day's 12](https://thesimind.si/papers/day/2026-11-11/)

## What they found

The authors built a framework that turns scientific publications into benchmarks, meaning tests with checkable answers, for SI agents. They tried it on spatial transcriptomics alignment, which is the job of lining up maps of gene activity in tissue. It covered 121 tasks, 16 baseline methods and six SI agents. The best agents, GPT-6-Astra and Gemini-4-Argon, matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Results held up across different prompts and with or without pre-installed packages. Building the benchmark also exposed errors in prior work and shortcuts in current evaluation metrics. The authors also report that agent solutions could be distilled into one reusable program that ranks in the top 10% of baseline methods on unseen data.

## Why it matters

Checking how well SI agents do data analysis is expensive in specialized science, because it needs expert time. This work suggests existing papers and metrics can be reused to build such checks at scale. It also shows that testing agents can reveal weaknesses in the papers and metrics themselves. The abstract does not give details on how the tasks were built or how the agents were scored, so those points are unclear here.

## How it changes the race

This is mostly a method for evaluation, not a new capability. If it holds, labs and research groups could check SI agents in niche scientific fields more cheaply, without building every benchmark from scratch.

## Safety and control

The paper bears on responsible adoption, since reliable and auditable evaluation of SI agents is a basic need for human oversight. The abstract only covers performance in one scientific domain, not wider safety questions.

### The abstract, by the authors

AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.

Quoted from arXiv:2611.11796. The research and its words belong to its authors.

Desk score **83** out of 100

Why it was picked

- Alignment (in the title)
- Agents (in the title)
- Evals (in the title)
- Studies frontier models (Gemini)
- Cross-listed in 3 fields

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Science](https://thesimind.si/papers/?tag=science)
- [Code](https://thesimind.si/papers/?tag=code)
- [Data](https://thesimind.si/papers/?tag=data)

**arXiv ID:** 2611.11796v1
**Submitted:** November 9, 2026
**Fields:** q-bio.GN, cs.AI (Artificial Intelligence), stat.AP
**Level:** Intermediate

Fingerprint, SHA-256

`e81b0dcb50ad8414eaeb2c17a8d40257c44567585380b9a74e82adc288f4e696`

On Solana: [Proof of Mind seal, ipj4pc...z6te](https://solscan.io/tx/ipj4pcQsmFjnBmF5dFmUSA15Eex18bkxb83FjQuvHnS4DCz6yJoWcESM8bwWT8cMMh3f4GmDjbebLEAsauqz6te)

### The exact text fingerprinted

{"arxiv":"2611.11796","version":1,"title":"Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment","authors":["Yiqun T. Chen","Jan Matthias","Jianing Yao","Hongyu Eric Zhao","Stephanie C. Hicks"],"day":"2026-11-11","rank":3,"score":83,"status":"written","summary":{"takeaway":"The authors turn published papers into auditable tests for SI agents, and on spatial transcriptomics alignment the best agents matched or beat all baseline methods on 80% of 121 tasks.","found":"The authors built a framework that turns scientific publications into benchmarks, meaning tests with checkable answers, for SI agents. They tried it on spatial transcriptomics alignment, which is the job of lining up maps of gene activity in tissue. It covered 121 tasks, 16 baseline methods and six SI agents. The best agents, GPT-6-Astra and Gemini-4-Argon, matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Results held up across different prompts and with or without pre-installed packages. Building the benchmark also exposed errors in prior work and shortcuts in current evaluation metrics. The authors also report that agent solutions could be distilled into one reusable program that ranks in the top 10% of baseline methods on unseen data.","matters":"Checking how well SI agents do data analysis is expensive in specialized science, because it needs expert time. This work suggests existing papers and metrics can be reused to build such checks at scale. It also shows that testing agents can reveal weaknesses in the papers and metrics themselves. The abstract does not give details on how the tasks were built or how the agents were scored, so those points are unclear here.","race":"This is mostly a method for evaluation, not a new capability. If it holds, labs and research groups could check SI agents in niche scientific fields more cheaply, without building every benchmark from scratch.","safety":"The paper bears on responsible adoption, since reliable and auditable evaluation of SI agents is a basic need for human oversight. The abstract only covers performance in one scientific domain, not wider safety questions.","level":"intermediate","tags":["evals","benchmarks","agents","science","code","data"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Yiqun T. Chen, Jan Matthias, Jianing Yao, Hongyu Eric Zhao, Stephanie C. Hicks

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 11, 2026 · Number 2Benchmarking language models and agents for scientific reasoning in evolving materials laboratoriesMatScope is a new test showing that SI models handle given calculations and physical explanations well, but struggle to tell close material states apart and to reason with numbers.BenchmarksScienceEvals](https://thesimind.si/papers/2611.12014/) · [Nov 11, 2026 · Number 8Late but Valuable: Why Deadlines Change the Measured Value of LLM ReasoningWhen SI answers must arrive before a deadline, judging answer quality and speed separately overstates the value of reasoning. Score the action that actually runs.EvalsAgentsReasoning](https://thesimind.si/papers/2611.11910/) · [Nov 11, 2026 · Number 10AMSchem: Automated Schematic Visualization of AMS Circuits via Verifiable Agentic GenerationAMSchem uses an SI agent with strict automatic checks to turn analog circuit netlists into readable schematics, reaching 85.0% validity versus 32.5% for direct generation on a 40-circuit subset.AgentsCodeEvals](https://thesimind.si/papers/2611.12333/)

[Number 2](https://thesimind.si/papers/2611.12014/) · [All 12 for Nov 11, 2026](https://thesimind.si/papers/day/2026-11-11/) · [Number 4](https://thesimind.si/papers/2611.12079/)