What they found
The NovaSilicon Team built AnalogLayoutWorld, an open-source benchmark with 40 expert-reviewed tasks. Each task asks an SI agent to produce a chip layout from a fixed transistor-level netlist (a list of parts and how they connect). The benchmark gives agents a design environment and checks results with independent physical and post-layout electrical verification. It also uses layout quality metrics made with analog layout experts. Across 13 model-harness configurations, 91.1% of attempts produced GDS layouts (a standard chip layout file), but only 23.0% met the acceptance criteria. The authors also test Novara, their own internal agent system, which got the highest reported aggregate score. They link performance gaps to test-time scaling under fixed budgets, and suggest that reasoning efficiency and good use of verification feedback help quality hold up as tasks get harder.
Why it matters
Coding agents do well at software, but real engineering work follows physical rules and needs many steps with tools. This benchmark gives a way to measure that gap. The large drop from producing a file to producing a correct layout shows that looking finished is not the same as being right. Independent verification is what exposes it.
How it changes the race
It gives frontier SI builders an open test for long, physics-bound engineering work, beyond software tasks. The authors' own system leads the table, so that comparison deserves care until others run it.
Safety and control
The gap between 91.1% of attempts producing layouts and 23.0% passing acceptance shows why independent checks matter before trusting SI output in engineering work.
The abstract, by the authors
Coding agents have demonstrated strong capabilities in software development, yet their ability to tackle long-horizon engineering tasks governed by physical constraints remains less understood. We study this question through analog circuit layout, which demands domain knowledge, geometric reasoning, and iterative interaction with design and verification tools. We introduce AnalogLayoutWorld, an open-source agent benchmark with 40 expert-reviewed tasks for end-to-end layout generation from fixed transistor-level netlists. It combines an integrated design environment with independent physical and post-layout electrical verification, as well as objective layout quality metrics co-designed with analog layout experts. Across 13 model-harness configurations, 91.1% of attempts produce GDS layouts, but only 23.0% satisfy the benchmark's acceptance criteria. We also evaluate Novara, an internally developed agent system, which achieves the highest reported aggregate score. Our analysis links performance differences to test-time scaling under fixed budgets, suggesting that reasoning efficiency and effective use of verification feedback help sustain design quality as task complexity grows.
Quoted from arXiv:2611.16991. The research and its words belong to its authors.
