---
title: "Comparative Analysis of Agent Evaluation... | SI Paper Desk"
url: https://thesimind.si/papers/2611.14493/
description: "In a controlled test of three ways to check enterprise analytics agents, the method that sees the agent's full step-by-step trace caught failures best, and..."
---

Paper of the Day · November 12, 2026

# Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents

In a controlled test of three ways to check enterprise analytics agents, the method that sees the agent's full step-by-step trace caught failures best, and security red-teaming found different problems than functional testing.

Howard Zhang, Chuck Arvin, Logan Brown and 3 more · Submitted November 10, 2026 · arXiv 2611.14493v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.14493) · [The PDF](https://arxiv.org/pdf/2611.14493) · [The day's 12](https://thesimind.si/papers/day/2026-11-12/)

## What they found

The authors compared three evaluation frameworks on two analytics agents, using frozen execution traces (saved records of what the agents did). The three were trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics (fixed rules). In Phase 1, 97% of configurations met the stability threshold, and stability patterns did not depend on the agent. In Phase 2, on 72 ground-truth test cases, the trace-based framework had the best balance of precision and recall (F1=0.90). Text-only evaluation matched it on recall but produced 5x more false alarms. In Phase 3, the study found 24 novel failures, with zero overlap between functional testing and security red-teaming.

## Why it matters

Companies that put SI agents into multi-tool workflows need ways to catch failures before users do. This study suggests that what the evaluator can see matters a lot: access to the full trace beat text alone. It also suggests that functional checks and security checks find different problems, so one is not a substitute for the other. The authors also report that LLM-based evaluators found subtle errors that human reviewers missed. The abstract describes a controlled research setting with two agents, so how far this carries to other settings is not stated.

## How it changes the race

This is practical, incremental guidance for teams deploying SI agents. It points to richer trace access and a split between functional and security testing as a way to cut missed failures and false alarms.

## Safety and control

The finding that functional testing and security red-teaming found entirely different failures suggests that relying on only one leaves gaps. The authors recommend using both: trace-based evaluation for function and adversarial red-teaming for security.

### The abstract, by the authors

Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.

Quoted from arXiv:2611.14493. The research and its words belong to its authors.

Desk score **80** out of 100

Why it was picked

- Agents (in the title)
- Evals (in the title)
- Safety
- Cross-listed in 4 fields
- Accepted at KDD
- Language models

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Security](https://thesimind.si/papers/?tag=security)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Code](https://thesimind.si/papers/?tag=code)

**arXiv ID:** 2611.14493v1
**Submitted:** November 10, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.CL (Computation and Language), cs.LG (Machine Learning), cs.SE (Software Engineering)
**Level:** Intermediate
**Comments:** Accepted to KDD 2026, Jeju, Korea. 9 pages, 3 figures

Fingerprint, SHA-256

`1472a6505a3ed0917119bfc2d7775cf15635f7f8e9484a4dd50298b0b8390bef`

On Solana: [Proof of Mind seal, 66zHq3...ugfh](https://solscan.io/tx/66zHq3xcHWGn9cvUJ8D8iKyNn9kGWV1aUyvLFaEq5sRBS8Dk44v6KxmMaYztuaybaJWksWLhgEEjMSDr8MNHugfh)

### The exact text fingerprinted

{"arxiv":"2611.14493","version":1,"title":"Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents","authors":["Howard Zhang","Chuck Arvin","Logan Brown","Mihir Bhatia","Neelu Choudhary","Lily Miao"],"day":"2026-11-12","rank":1,"score":80,"status":"written","summary":{"takeaway":"In a controlled test of three ways to check enterprise analytics agents, the method that sees the agent's full step-by-step trace caught failures best, and security red-teaming found different problems than functional testing.","found":"The authors compared three evaluation frameworks on two analytics agents, using frozen execution traces (saved records of what the agents did). The three were trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics (fixed rules). In Phase 1, 97% of configurations met the stability threshold, and stability patterns did not depend on the agent. In Phase 2, on 72 ground-truth test cases, the trace-based framework had the best balance of precision and recall (F1=0.90). Text-only evaluation matched it on recall but produced 5x more false alarms. In Phase 3, the study found 24 novel failures, with zero overlap between functional testing and security red-teaming.","matters":"Companies that put SI agents into multi-tool workflows need ways to catch failures before users do. This study suggests that what the evaluator can see matters a lot: access to the full trace beat text alone. It also suggests that functional checks and security checks find different problems, so one is not a substitute for the other. The authors also report that LLM-based evaluators found subtle errors that human reviewers missed. The abstract describes a controlled research setting with two agents, so how far this carries to other settings is not stated.","race":"This is practical, incremental guidance for teams deploying SI agents. It points to richer trace access and a split between functional and security testing as a way to cut missed failures and false alarms.","safety":"The finding that functional testing and security red-teaming found entirely different failures suggests that relying on only one leaves gaps. The authors recommend using both: trace-based evaluation for function and adversarial red-teaming for security.","level":"intermediate","tags":["evals","agents","benchmarks","security","language-models","code"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 14, 2026 · Number 4Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its ExperimentsA local test kit for small coding agents shows how noisy single scores are: several apparent gains did not hold up when tested again on new tasks.EvalsCodeAgents](https://thesimind.si/papers/2611.16485/) · [Nov 19, 2026 · Number 4Stopping Is Not the Limit: Understanding and Eliciting Agent Capability in Long-Horizon Vulnerability ExploitationSI agents that stop while trying to exploit software flaws often have more capability left, and a context-management method called SCoRe can draw it out.SecurityEvalsAgents](https://thesimind.si/papers/2611.22888/) · [Nov 15, 2026 · Paper of the DayLook Here, Then Decide: Small Decision Models in a Harness Find Memory-Safety Vulnerabilities at Scale on Commodity GPUsA small 4B-parameter open model is weak at judging code alone, but it ranks candidate functions well enough that a harness on cheap GPUs confirmed 187 memory-safety findings.SecurityCodeEfficiency](https://thesimind.si/papers/2611.16565/)

[All 12 for Nov 12, 2026](https://thesimind.si/papers/day/2026-11-12/) · [Number 2](https://thesimind.si/papers/2611.13183/)
