---
title: "AgentScript-Eval: Benchmarking LLMs and... | SI Paper Desk"
url: https://thesimind.si/papers/2611.13183/
description: "On a new benchmark for a private Salesforce agent language, SI models mostly fail zero-shot, and the setup around the model, like a compile-and-edit loop..."
---

Number 2 of 12 · November 12, 2026

# AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

On a new benchmark for a private Salesforce agent language, SI models mostly fail zero-shot, and the setup around the model, like a compile-and-edit loop and clear guidance, mattered more than model size.

Dipin Khati, Shubham Mehrotra, Bin Bi and 5 more · Submitted November 9, 2026 · arXiv 2611.13183v1 · cs.SE

[Read the paper on arXiv](https://arxiv.org/abs/2611.13183) · [The PDF](https://arxiv.org/pdf/2611.13183) · [The day's 12](https://thesimind.si/papers/day/2026-11-12/)

## What they found

The authors built AgentScript-Eval, a benchmark of 100 real tasks for AgentScript, the declarative language Salesforce uses in Agentforce to build conversational agents. Tasks cover writing an agent from a specification, fixing a broken one, and improving a working one. Five models were tested zero-shot, meaning with no examples. The best model reached only 25.0% pass@3, and every model scored 0% on authoring from scratch. A 31B open-weight model had the highest compilation rate (48.0%) and matched the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding a compile-and-edit loop roughly doubled partial-credit accuracy. The same skill pack scored 28% as plain documentation but 14% as a live plugin, and rewriting one skill in the plugin raised it to 37%, the best result.

## Why it matters

Many companies use private, niche languages that SI models saw little or nothing of in training. This benchmark shows how weak models can be there, even though they write common languages well. It also suggests that teams may get more from better tooling and better-worded guidance than from picking a bigger or pricier model. The abstract does not say how the tasks were chosen beyond calling them real, and it covers one language and five models.

## How it changes the race

If the results hold, the edge in enterprise settings may come less from the largest model and more from the harness and guidance around it. That could lower costs and let smaller or open-weight models compete on this kind of work.

### The abstract, by the authors

Domain-specific languages (DSLs) are widely used in enterprise software, giving a compact interface to complex tasks, from data access to infrastructure automation. Platforms increasingly extend them to agents, where a developer declares the topics, permitted actions, and guardrails of an agent rather than writing general-purpose code. Large language models (LLMs) write general-purpose code well, but that ability has been measured almost entirely on languages abundant in public pretraining data, and enterprise DSLs are neither abundant nor public. We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents. Tasks span authoring an agent from a specification, debugging a broken definition, and optimizing a working one. Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch. We further find that model scale and price are weak predictors of success - a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy. Beyond that gain, the wording of domain guidance matters more than its packaging. The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%, our best result, with catalog and tooling untouched. Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.

Quoted from arXiv:2611.13183. The research and its words belong to its authors.

Desk score **80** out of 100

Why it was picked

- Agents (in the title)
- Evals (in the title)
- Safety
- Language models (in the title)
- Accepted at NeurIPS
- Cross-listed in 2 fields

Topics

- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Code](https://thesimind.si/papers/?tag=code)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Efficiency](https://thesimind.si/papers/?tag=efficiency)

**arXiv ID:** 2611.13183v1
**Submitted:** November 9, 2026
**Fields:** cs.SE (Software Engineering), cs.AI (Artificial Intelligence)
**Level:** Intermediate
**Comments:** Accepted to 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: 2nd Workshop on Agentic AI Benchmarks and Applications for Enterprise Tasks (AABA4ET), NeurIPS 2026

Fingerprint, SHA-256

`203c241f12380f6b343519e245d22c009e083fe3c4df090a79d0b6a847f72172`

On Solana: [Proof of Mind seal, 66zHq3...ugfh](https://solscan.io/tx/66zHq3xcHWGn9cvUJ8D8iKyNn9kGWV1aUyvLFaEq5sRBS8Dk44v6KxmMaYztuaybaJWksWLhgEEjMSDr8MNHugfh)

### The exact text fingerprinted

{"arxiv":"2611.13183","version":1,"title":"AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL","authors":["Dipin Khati","Shubham Mehrotra","Bin Bi","Zhou Yu","Denys Poshyvanyk","James Zhu","Sitaram Asur","Phil L. Mui"],"day":"2026-11-12","rank":2,"score":80,"status":"written","summary":{"takeaway":"On a new benchmark for a private Salesforce agent language, SI models mostly fail zero-shot, and the setup around the model, like a compile-and-edit loop and clear guidance, mattered more than model size.","found":"The authors built AgentScript-Eval, a benchmark of 100 real tasks for AgentScript, the declarative language Salesforce uses in Agentforce to build conversational agents. Tasks cover writing an agent from a specification, fixing a broken one, and improving a working one. Five models were tested zero-shot, meaning with no examples. The best model reached only 25.0% pass@3, and every model scored 0% on authoring from scratch. A 31B open-weight model had the highest compilation rate (48.0%) and matched the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding a compile-and-edit loop roughly doubled partial-credit accuracy. The same skill pack scored 28% as plain documentation but 14% as a live plugin, and rewriting one skill in the plugin raised it to 37%, the best result.","matters":"Many companies use private, niche languages that SI models saw little or nothing of in training. This benchmark shows how weak models can be there, even though they write common languages well. It also suggests that teams may get more from better tooling and better-worded guidance than from picking a bigger or pricier model. The abstract does not say how the tasks were chosen beyond calling them real, and it covers one language and five models.","race":"If the results hold, the edge in enterprise settings may come less from the largest model and more from the harness and guidance around it. That could lower costs and let smaller or open-weight models compete on this kind of work.","safety":"","level":"intermediate","tags":["benchmarks","code","agents","evals","language-models","efficiency"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 11, 2026 · Number 10AMSchem: Automated Schematic Visualization of AMS Circuits via Verifiable Agentic GenerationAMSchem uses an SI agent with strict automatic checks to turn analog circuit netlists into readable schematics, reaching 85.0% validity versus 32.5% for direct generation on a 40-circuit subset.AgentsCodeEvals](https://thesimind.si/papers/2611.12333/) · [Nov 18, 2026 · Paper of the DayGRACE: Parallel Best-of-N Multi-Agent Analysis of Closed-Form Questions over Dataframes with Guardrail Selection and Adaptive RetryGRACE runs several SI agents in parallel on spreadsheet-style questions, judges their executed work instead of voting, and can reject all answers and retry, lifting answer match from 39% to 71% on one benchmark.AgentsMulti-agent systemsCode](https://thesimind.si/papers/2611.20983/) · [Nov 10, 2026 · Number 3PACE: Keeping the Right Tokens in the KV Cache Is Not Enough for Long-Horizon LLM AgentsKeeping the needed tokens in a SI agent's memory cache is not enough. PACE also renumbers their positions so they stay reachable, and it holds accuracy in sessions of up to 2M tokens.AgentsEfficiencyMemory](https://thesimind.si/papers/2611.10240/)

[Number 1](https://thesimind.si/papers/2611.14493/) · [All 12 for Nov 12, 2026](https://thesimind.si/papers/day/2026-11-12/) · [Number 3](https://thesimind.si/papers/2611.13926/)
