---
title: "Can LLMs Write Backdoors? An Empirical... | SI Paper Desk"
url: https://thesimind.si/papers/2611.17395/
description: "The authors show that general-purpose SI language models can plant backdoors in other neural networks by editing weights from plain-language instructions..."
---

Number 9 of 12 · November 16, 2026

# Can LLMs Write Backdoors? An Empirical Security Study of Weight-Space Programming

The authors show that general-purpose SI language models can plant backdoors in other neural networks by editing weights from plain-language instructions, sometimes changing very few parameters.

Zhaojia Wang, Jiucui Lu and Zheng Li · Submitted November 12, 2026 · arXiv 2611.17395v1 · cs.CR

[Read the paper on arXiv](https://arxiv.org/abs/2611.17395) · [The PDF](https://arxiv.org/pdf/2611.17395) · [The day's 12](https://thesimind.si/papers/day/2026-11-16/)

## What they found

The authors tested whether an SI language model can edit another model's weights to add a backdoor. A backdoor is a hidden trigger that makes a model misbehave on purpose. The editor had no task data, ran no forward passes and computed no gradients. They tested five LLMs as editors on three targets: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. Backdoors were injected across different architectures while clean performance was largely preserved. Some attacks needed very few changes: only six parameters for the MLP and 0.02% of the weights for a fine-tuned Qwen model. All five LLMs backdoored the MLP, but only GPT-6 Astra succeeded on all three targets.

## Why it matters

Weight-space programming means an SI model reads and edits another model's weights directly, without normal training. It may help improve models. This study suggests the same skill can be used to attack them. The abstract does not give attack success rates or say how the attacks were detected, so the details are unclear. The results show that the risk grows with editor skill and differs by target.

## How it changes the race

If the results hold, stronger SI models become more capable of tampering with other models, so model supply chains need more scrutiny. It also shows that a backdoor attack no longer needs training data or compute for training. This is a security finding, not a capability gain.

## Safety and control

The paper bears directly on security and misuse. It points to a risk that SI models could be used to plant hidden backdoors in other models, and the authors call for safeguarding LLM-driven weight editing.

### The abstract, by the authors

Recent advances in weight-space programming (WSP) have demonstrated that large language models (LLMs) can directly inspect and modify neural network weights without conventional gradient-based training. While this capability opens new possibilities for model improvement, it also raises an important security question: Can LLMs exploit weight-space programming to inject backdoors into neural networks? To answer this question, we present an empirical security study of LLM-driven weight-space programming for backdoor injection. Specifically, we investigate whether general-purpose LLMs can directly modify the weights of target models according to natural-language instructions, without accessing task-specific data, performing forward passes, or computing gradients. We evaluate five LLMs as weight editors across three target architectures: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. Our experiments reveal three key findings. First, LLM-driven weight editing can successfully inject backdoors across different architectures while largely preserving clean performance. Second, successful attacks can require remarkably few parameter modifications, including only six parameters for the MLP and 0.02% of the weights for a fine-tuned Qwen model. Third, editing capabilities vary substantially across LLMs and target architectures: although all five LLMs successfully backdoor the MLP, only GPT-6 Astra succeeds across all three targets. These findings highlight an emerging security risk of weight-space programming and underscore the importance of safeguarding LLM-driven weight editing.

Quoted from arXiv:2611.17395. The research and its words belong to its authors.

Desk score **70** out of 100

Why it was picked

- Safety (in the title)
- Language models (in the title)
- Security
- Studies frontier models (Qwen2.5)
- Cross-listed in 3 fields

Topics

- [Security](https://thesimind.si/papers/?tag=security)
- [Safety](https://thesimind.si/papers/?tag=safety)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Agents](https://thesimind.si/papers/?tag=agents)

**arXiv ID:** 2611.17395v1
**Submitted:** November 12, 2026
**Fields:** cs.CR (Cryptography and Security), cs.AI (Artificial Intelligence), cs.LG (Machine Learning)
**Level:** Intermediate

Fingerprint, SHA-256

`85712b14fb525951c7e24ef15d1e4ce81422fb50ee646cc99f24ffac41a18b9f`

On Solana: [Proof of Mind seal, 3mXUMW...GSim](https://solscan.io/tx/3mXUMW5MkdFiyPBNPYd5DfwHhma5EdjhUFEw9jZaDuuPqgszJtoDYttSrD9kyWUFdMCzn7RnFNd4nQhwAUgrGSim)

### The exact text fingerprinted

{"arxiv":"2611.17395","version":1,"title":"Can LLMs Write Backdoors? An Empirical Security Study of Weight-Space Programming","authors":["Zhaojia Wang","Jiucui Lu","Zheng Li"],"day":"2026-11-16","rank":9,"score":70,"status":"written","summary":{"takeaway":"The authors show that general-purpose SI language models can plant backdoors in other neural networks by editing weights from plain-language instructions, sometimes changing very few parameters.","found":"The authors tested whether an SI language model can edit another model's weights to add a backdoor. A backdoor is a hidden trigger that makes a model misbehave on purpose. The editor had no task data, ran no forward passes and computed no gradients. They tested five LLMs as editors on three targets: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. Backdoors were injected across different architectures while clean performance was largely preserved. Some attacks needed very few changes: only six parameters for the MLP and 0.02% of the weights for a fine-tuned Qwen model. All five LLMs backdoored the MLP, but only GPT-6 Astra succeeded on all three targets.","matters":"Weight-space programming means an SI model reads and edits another model's weights directly, without normal training. It may help improve models. This study suggests the same skill can be used to attack them. The abstract does not give attack success rates or say how the attacks were detected, so the details are unclear. The results show that the risk grows with editor skill and differs by target.","race":"If the results hold, stronger SI models become more capable of tampering with other models, so model supply chains need more scrutiny. It also shows that a backdoor attack no longer needs training data or compute for training. This is a security finding, not a capability gain.","safety":"The paper bears directly on security and misuse. It points to a risk that SI models could be used to plant hidden backdoors in other models, and the authors call for safeguarding LLM-driven weight editing.","level":"intermediate","tags":["security","safety","language-models","evals","agents"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Zhaojia Wang, Jiucui Lu, Zheng Li

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 16, 2026 · Paper of the DayOnly as Safe as the Weakest Agent: A Break That Appears Only When the System Is Multi-AgentWhen a strong SI model hands work to a cheaper sub-agent, the whole system can be only as safe as the weaker agent, even if the strong model alone resists the attack.SafetyMulti-agent systemsSecurity](https://thesimind.si/papers/2611.17520/) · [Nov 20, 2026 · Number 2AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM AgentsAutoPI is a red teaming tool that tests SI agents for prompt injection across many attack types, not just how often attacks work, and it reports higher success and coverage than recent rivals.SecurityAgentsEvals](https://thesimind.si/papers/2611.24313/) · [Nov 16, 2026 · Number 2Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM AgentsSoteria checks an SI agent's plan against written rules before any tool runs, then enforces those rules during execution. The authors report more completed tasks and no policy violations.AgentsSafetyReasoning](https://thesimind.si/papers/2611.17880/)

[Number 8](https://thesimind.si/papers/2611.17620/) · [All 12 for Nov 16, 2026](https://thesimind.si/papers/day/2026-11-16/) · [Number 10](https://thesimind.si/papers/2611.17481/)
