---
title: "SoK: Assessing the Evolution of LLM... | SI Paper Desk"
url: https://thesimind.si/papers/2611.26601/
description: "This survey rates how well SI agents find, check and fix software security bugs, and reports that disclosed vulnerabilities more than doubled in 2026 while..."
---

Number 10 of 12 · November 21, 2026

# SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research

This survey rates how well SI agents find, check and fix software security bugs, and reports that disclosed vulnerabilities more than doubled in 2026 while fix and attack timelines have not clearly changed.

Amine Lbath, Manan Suri, Aurelien Delaitre and 3 more · Submitted November 19, 2026 · arXiv 2611.26601v1 · cs.CR

[Read the paper on arXiv](https://arxiv.org/abs/2611.26601) · [The PDF](https://arxiv.org/pdf/2611.26601) · [The day's 12](https://thesimind.si/papers/day/2026-11-21/)

## What they found

The authors build a framework that rates SI security skills on an autonomy scale modeled on the levels of driving automation. Each claim is also qualified by its correctness, its availability and the strength of its evidence. Using it, they find that agents found, validated and patched vulnerabilities in real software without human help before the restricted frontier models of 2026. They also find that open-weight models later reached comparable autonomy for discovery and validation. In a decade of public vulnerability records, disclosed vulnerabilities more than doubled in 2026, with more rated critical or high. Severe bugs that had survived a decade or more in widely used code were reported several times more often than before. The time from report to fix, and from disclosure to attack, has not shifted noticeably so far.

## Why it matters

Claims about SI and cybersecurity are spread across papers, vendor reports and disclosures, and are hard to compare. A common scale and evidence grading helps readers judge which claims to trust. The data suggests SI is changing how many bugs are found before it changes how fast they are fixed or exploited. The authors also grade proposed mitigations and give guidance to each group of stakeholders. The abstract does not say what that guidance is.

## How it changes the race

If the findings hold, strong bug-finding skill is no longer limited to restricted frontier models, since open-weight models reached comparable autonomy for discovery and validation. That would widen who can do this work, for defenders and attackers alike.

## Safety and control

The paper bears directly on security and misuse. Open-weight models with comparable discovery autonomy are harder to restrict, and the authors report a surge in severe disclosures. They found no clear change yet in fix or attack timing, and the abstract does not say what the graded mitigations are.

### The abstract, by the authors

LLM-based agents are becoming increasingly capable at discovering, validating and patching vulnerabilities in real software, and vendors and researchers report that this is already changing how vulnerabilities are found and fixed. These claims, however, are scattered across papers, vendor reports and disclosures and are rarely comparable. We introduce a framework that rates these capabilities on an autonomy scale modeled on the levels of driving automation, and qualifies each claim by its correctness, its availability and the strength of its evidence. We use this framework to systematize the published evidence on LLM capability in vulnerability research. We find that agents discovered, validated, and patched vulnerabilities in real-world software without human intervention before the restricted frontier models of 2026, and that open-weight models subsequently attained comparable autonomy for discovery and validation. We then measure the consequences of evolving LLM security capabilities in a decade of public vulnerability records, advisories and exploitation data. In 2026 the number of disclosed vulnerabilities more than doubled, with an increasing share rated critical or high, and severe bugs that had survived a decade or more in widely used code were reported several times more often than before. Yet the time between the stages of a vulnerability's life, from report to fix and from disclosure to attack, has not shifted noticeably so far. We grade the mitigations proposed in the literature with an explicit rubric and draw guidance for each group of stakeholders from the best-supported ones.

Quoted from arXiv:2611.26601. The research and its words belong to its authors.

Desk score **66** out of 100

Why it was picked

- Security (in the title)
- Agents
- Evals
- Language models (in the title)
- Cross-listed in 3 fields

Topics

- [Security](https://thesimind.si/papers/?tag=security)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Code](https://thesimind.si/papers/?tag=code)
- [Policy and law](https://thesimind.si/papers/?tag=policy)

**arXiv ID:** 2611.26601v1
**Submitted:** November 19, 2026
**Fields:** cs.CR (Cryptography and Security), cs.AI (Artificial Intelligence), cs.SE (Software Engineering)
**Level:** Intermediate

Fingerprint, SHA-256

`e28fa9a56a40eef3766ab71fad0cda5163606430e770719c5834e5588c8ae454`

On Solana: [Proof of Mind seal, beudvj...We9f](https://solscan.io/tx/beudvjGVbBTGVZsMaL2MgMJjuMy9dK2a2K3ETkw5AFwjoARs9PQedxMxfrBaiJ5MetjT2AHxuhfTCyP34eQWe9f)

### The exact text fingerprinted

{"arxiv":"2611.26601","version":1,"title":"SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research","authors":["Amine Lbath","Manan Suri","Aurelien Delaitre","Vadim Okun","Massih-Reza Amini","Dinesh Manocha"],"day":"2026-11-21","rank":10,"score":66,"status":"written","summary":{"takeaway":"This survey rates how well SI agents find, check and fix software security bugs, and reports that disclosed vulnerabilities more than doubled in 2026 while fix and attack timelines have not clearly changed.","found":"The authors build a framework that rates SI security skills on an autonomy scale modeled on the levels of driving automation. Each claim is also qualified by its correctness, its availability and the strength of its evidence. Using it, they find that agents found, validated and patched vulnerabilities in real software without human help before the restricted frontier models of 2026. They also find that open-weight models later reached comparable autonomy for discovery and validation. In a decade of public vulnerability records, disclosed vulnerabilities more than doubled in 2026, with more rated critical or high. Severe bugs that had survived a decade or more in widely used code were reported several times more often than before. The time from report to fix, and from disclosure to attack, has not shifted noticeably so far.","matters":"Claims about SI and cybersecurity are spread across papers, vendor reports and disclosures, and are hard to compare. A common scale and evidence grading helps readers judge which claims to trust. The data suggests SI is changing how many bugs are found before it changes how fast they are fixed or exploited. The authors also grade proposed mitigations and give guidance to each group of stakeholders. The abstract does not say what that guidance is.","race":"If the findings hold, strong bug-finding skill is no longer limited to restricted frontier models, since open-weight models reached comparable autonomy for discovery and validation. That would widen who can do this work, for defenders and attackers alike.","safety":"The paper bears directly on security and misuse. Open-weight models with comparable discovery autonomy are harder to restrict, and the authors report a surge in severe disclosures. They found no clear change yet in fix or attack timing, and the abstract does not say what the graded mitigations are.","level":"intermediate","tags":["security","agents","evals","language-models","code","policy"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Amine Lbath, Manan Suri, Aurelien Delaitre, Vadim Okun, Massih-Reza Amini, Dinesh Manocha

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 15, 2026 · Paper of the DayLook Here, Then Decide: Small Decision Models in a Harness Find Memory-Safety Vulnerabilities at Scale on Commodity GPUsA small 4B-parameter open model is weak at judging code alone, but it ranks candidate functions well enough that a harness on cheap GPUs confirmed 187 memory-safety findings.SecurityCodeEfficiency](https://thesimind.si/papers/2611.16565/) · [Nov 11, 2026 · Number 7Treating the Model as an Untrusted Component: Deterministic Policy Enforcement, Human Approval and Taint Tracking Against Prompt Injection in Tool-Using Agents, with Validation on Open-Weights ModelsRules enforced outside the model blocked far more prompt injection attacks than prompt-based defences did, though the human approval and taint tracking results come only from simulation.SecurityAgentsSafety](https://thesimind.si/papers/2611.11571/) · [Nov 12, 2026 · Paper of the DayComparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics AgentsIn a controlled test of three ways to check enterprise analytics agents, the method that sees the agent's full step-by-step trace caught failures best, and security red-teaming found different problems than functional testing.EvalsAgentsBenchmarks](https://thesimind.si/papers/2611.14493/)

[Number 9](https://thesimind.si/papers/2611.26335/) · [All 12 for Nov 21, 2026](https://thesimind.si/papers/day/2026-11-21/) · [Number 11](https://thesimind.si/papers/2611.26307/)
