Number 10 of 12 · November 17, 2026

Adversarially Attacking Symbolic Vocabulary Vulnerabilities In LLM Planners

Changing the names of actions and objects in a planning task, using an automated attacker called Symbolic Swapper, can cut LLM planning performance by over 70%, even for strong reasoning models.

What they found

The authors built an attack model called Symbolic Swapper. It is LLM-based and automatically rewrites planning tasks so that the names of actions, predicates and objects are non-standard. The rewritten tasks keep the same structure but remove the everyday word cues that models lean on. The authors report that these attacks can lower the planning performance of base models, and even of robust planning frameworks, by over 70%. Their analysis says that after successful obfuscation, models seem to reason much worse, even though they still show some ability to understand the domain. The authors describe this as a gap between perceiving a task and being able to plan within it.

Why it matters

SI agents are being used for planning, where they must follow strict rules about actions and objects. This work suggests that strong planning scores can depend partly on familiar word names rather than pure planning skill. Symbolic planners, which work on the structure alone, do not have this weakness, so the paper adds to the case that LLM planners are less reliable. The abstract does not give details on which models or benchmarks were tested, so the scope of the result is unclear from it.

How it changes the race

If the result holds, builders of SI agents will need to test planning with obfuscated vocabularies, not just standard ones. Gains from recent reasoning models may be less robust than they look on regular benchmarks.

Safety and control

The paper bears on reliability and security of SI agents. An attacker could in principle craft unusual task wording that breaks an agent's planning. The abstract frames this as a way to reveal weaknesses, and it does not describe any defense.

The abstract, by the authors

As large language models (LLMs) are increasingly deployed as autonomous agents, their abilities in navigating automated planning tasks have become essential. However, despite their strengths, LLMs' planning ability is not robust, particularly when they are not allowed to exploit commonsense cues in the symbolic vocabulary (e.g., names of actions, predicates, and objects) used to define planning tasks. This makes LLMs unreliable compared to symbolic planners. With recent reasoning models showing improved planning performance on tasks with obfuscated symbolic vocabularies, we utilize an adversarial attack framework to reveal to what extent the crucial weaknesses of LLM agents at joint symbolic interpretation and planning remain and whether they can reason properly when presented with severe cases. As such, our main contribution is devising an LLM-based attack model, which we call Symbolic Swapper, to automatically generate obfuscated planning domains with non-standard symbolic vocabularies that systematically break strong reasoning models' ability to plan. In doing so, we reveal how much LLMs' planning abilities can be further degraded under more advanced obfuscation schemes, allowing us to ascertain their success when having to rely on their pure planning ability. We find that attacks can decrease the planning performance of base models and even robust planning frameworks by over 70%. We further analyze the factors behind how LLM planning abilities break down and find that successful obfuscation causes models to appear to significantly degrade in reasoning quality despite them showing some inherent ability to understand the domain, revealing a gap in the perception of a task domain and being able to plan under it.

Quoted from arXiv:2611.17910. The research and its words belong to its authors.

The authors

Stephen Obadinma, Xiaodan Zhu

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.