---
title: "SynthCorp: A Platform for Creating... | SI Paper Desk"
url: https://thesimind.si/papers/2611.14765/
description: "SynthCorp simulates business events over time to build fake but consistent company data and test tasks, so builders can create harder, targeted tests for SI..."
---

Number 6 of 12 · November 13, 2026

# SynthCorp: A Platform for Creating Benchmarks for Data Agents

SynthCorp simulates business events over time to build fake but consistent company data and test tasks, so builders can create harder, targeted tests for SI data agents.

Jiani Zhang, Manav Garg, Eugenie Lai and 5 more · Submitted November 10, 2026 · arXiv 2611.14765v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.14765) · [The PDF](https://arxiv.org/pdf/2611.14765) · [The day's 12](https://thesimind.si/papers/day/2026-11-13/)

## What they found

The authors built SynthCorp, a platform that simulates how business events create and change company data over time. Each simulated data estate mixes relational databases, a document store and a data catalog, with test tasks and runnable answer programs. They made seven data estates (up to 1K operational tables) and 256 tasks across five domains. The abstract says the data passes automatic checks, such as foreign-key violations and database execution failures, with a near-zero error rate. Tests of three models in four data-context settings showed the benchmarks separate data contexts and model tiers. The authors also check answers against a log of the simulation, which verifies the answer programs without manual labeling, including on tasks that combine several events and expose reasoning failures in state-of-the-art agents.

## Why it matters

Many current tests for SI data agents (systems that work with company data) are quickly maxed out and cannot probe specific skills. A tool that makes customizable, checkable tests could help people see where agents fail on messy business data. Skipping manual labeling could also make such tests cheaper to build. The abstract gives no detailed scores, and the benchmarks are not yet released.

## How it changes the race

If the approach holds, teams building enterprise SI agents get a cheaper way to make harder, targeted tests without human annotators. It is an evaluation tool, not a capability gain, and its value depends on the promised benchmark release.

### The abstract, by the authors

Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass deterministic pipeline checks (such as foreign-key violations, undeclared writes, recurrence violations, and database execution failures) with a near-zero error rate, introduce cross-asset complexities, and cover all requested dimensions. Evaluating three models across four data-context settings shows that the benchmarks separate data contexts and model tiers. Auditing candidate queries against causal provenance in the simulation log verifies gold programs without manual annotation, including on multi-event composite tasks that expose reasoning failures in state-of-the-art agents. We hope the community uses SynthCorp to generate more challenging and useful data estates and benchmarks that foster the development of enterprise data agents.

Quoted from arXiv:2611.14765. The research and its words belong to its authors.

Desk score **73** out of 100

Why it was picked

- Agents (in the title)
- Evals (in the title)
- Reasoning
- Language models
- Cross-listed in 2 fields

Topics

- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Data](https://thesimind.si/papers/?tag=data)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)

**arXiv ID:** 2611.14765v1
**Submitted:** November 10, 2026
**Fields:** cs.AI (Artificial Intelligence), cs.DB (Databases)
**Level:** Intermediate
**Comments:** SynthCorp v1.0. We will release the benchmarks soon

Fingerprint, SHA-256

`e5bb924726734e958b416e65a0f24a90a88593a1d81808016776381083646f9f`

On Solana: [Proof of Mind seal, 5PD2kk...Cqj6](https://solscan.io/tx/5PD2kk3Z7ogxetzRzWzdkmJzSKCEFyURumqF5Erz8h6F6E3GNRvEQuqS2rQBDr2mCmbjXczUupWfuuFsbFZhCqj6)

### The exact text fingerprinted

{"arxiv":"2611.14765","version":1,"title":"SynthCorp: A Platform for Creating Benchmarks for Data Agents","authors":["Jiani Zhang","Manav Garg","Eugenie Lai","Cheng Zhen","Eva Sharma","Yang Pan","Yaming Lin","Alon Halevy"],"day":"2026-11-13","rank":6,"score":73,"status":"written","summary":{"takeaway":"SynthCorp simulates business events over time to build fake but consistent company data and test tasks, so builders can create harder, targeted tests for SI data agents.","found":"The authors built SynthCorp, a platform that simulates how business events create and change company data over time. Each simulated data estate mixes relational databases, a document store and a data catalog, with test tasks and runnable answer programs. They made seven data estates (up to 1K operational tables) and 256 tasks across five domains. The abstract says the data passes automatic checks, such as foreign-key violations and database execution failures, with a near-zero error rate. Tests of three models in four data-context settings showed the benchmarks separate data contexts and model tiers. The authors also check answers against a log of the simulation, which verifies the answer programs without manual labeling, including on tasks that combine several events and expose reasoning failures in state-of-the-art agents.","matters":"Many current tests for SI data agents (systems that work with company data) are quickly maxed out and cannot probe specific skills. A tool that makes customizable, checkable tests could help people see where agents fail on messy business data. Skipping manual labeling could also make such tests cheaper to build. The abstract gives no detailed scores, and the benchmarks are not yet released.","race":"If the approach holds, teams building enterprise SI agents get a cheaper way to make harder, targeted tests without human annotators. It is an evaluation tool, not a capability gain, and its value depends on the promised benchmark release.","safety":"","level":"intermediate","tags":["benchmarks","agents","evals","data","language-models","reasoning"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 11, 2026 · Number 2Benchmarking language models and agents for scientific reasoning in evolving materials laboratoriesMatScope is a new test showing that SI models handle given calculations and physical explanations well, but struggle to tell close material states apart and to reason with numbers.BenchmarksScienceEvals](https://thesimind.si/papers/2611.12014/) · [Nov 13, 2026 · Number 2MLIRBench - A Benchmark and Dataset for Compiler and Hardware-Aware ReasoningMLIRBench tests whether SI models truly understand compiler code. The best model labels pairs correctly 92.65% of the time but scores only 5.04/10 on describing what the code actually does.BenchmarksCodeEvals](https://thesimind.si/papers/2611.14881/) · [Nov 18, 2026 · Number 7GPRM: Global Perspective Process Reward ModelGPRM is a step-by-step reasoning judge that sees earlier steps, their scores, and later steps, and the authors report it beats strong baselines on PRMBench and ProcessBench with less data and smaller models.ReasoningEvalsReinforcement learning](https://thesimind.si/papers/2611.20755/)

[Number 5](https://thesimind.si/papers/2611.14864/) · [All 12 for Nov 13, 2026](https://thesimind.si/papers/day/2026-11-13/) · [Number 7](https://thesimind.si/papers/2611.14760/)
