What they found
The authors tested whether an SI language model can edit another model's weights to add a backdoor. A backdoor is a hidden trigger that makes a model misbehave on purpose. The editor had no task data, ran no forward passes and computed no gradients. They tested five LLMs as editors on three targets: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. Backdoors were injected across different architectures while clean performance was largely preserved. Some attacks needed very few changes: only six parameters for the MLP and 0.02% of the weights for a fine-tuned Qwen model. All five LLMs backdoored the MLP, but only GPT-6 Astra succeeded on all three targets.
Why it matters
Weight-space programming means an SI model reads and edits another model's weights directly, without normal training. It may help improve models. This study suggests the same skill can be used to attack them. The abstract does not give attack success rates or say how the attacks were detected, so the details are unclear. The results show that the risk grows with editor skill and differs by target.
How it changes the race
If the results hold, stronger SI models become more capable of tampering with other models, so model supply chains need more scrutiny. It also shows that a backdoor attack no longer needs training data or compute for training. This is a security finding, not a capability gain.
Safety and control
The paper bears directly on security and misuse. It points to a risk that SI models could be used to plant hidden backdoors in other models, and the authors call for safeguarding LLM-driven weight editing.
The abstract, by the authors
Recent advances in weight-space programming (WSP) have demonstrated that large language models (LLMs) can directly inspect and modify neural network weights without conventional gradient-based training. While this capability opens new possibilities for model improvement, it also raises an important security question: Can LLMs exploit weight-space programming to inject backdoors into neural networks? To answer this question, we present an empirical security study of LLM-driven weight-space programming for backdoor injection. Specifically, we investigate whether general-purpose LLMs can directly modify the weights of target models according to natural-language instructions, without accessing task-specific data, performing forward passes, or computing gradients. We evaluate five LLMs as weight editors across three target architectures: an MLP on MNIST, ResNet-18 on CIFAR-10, and Qwen2.5-0.5B-Instruct on AG News. Our experiments reveal three key findings. First, LLM-driven weight editing can successfully inject backdoors across different architectures while largely preserving clean performance. Second, successful attacks can require remarkably few parameter modifications, including only six parameters for the MLP and 0.02% of the weights for a fine-tuned Qwen model. Third, editing capabilities vary substantially across LLMs and target architectures: although all five LLMs successfully backdoor the MLP, only GPT-6 Astra succeeds across all three targets. These findings highlight an emerging security risk of weight-space programming and underscore the importance of safeguarding LLM-driven weight editing.
Quoted from arXiv:2611.17395. The research and its words belong to its authors.
