---
title: "ShadowBench: Exposing Lexical Anchoring... | SI Paper Desk"
url: https://thesimind.si/papers/2611.15925/
description: "ShadowBench suggests SI models know many facts but struggle to recall them without a name, and that unlearning methods may hide a fact's name without..."
---

Number 3 of 12 · November 15, 2026

# ShadowBench: Exposing Lexical Anchoring and the Illusion of Forgetting in Large Language Models

ShadowBench suggests SI models know many facts but struggle to recall them without a name, and that unlearning methods may hide a fact's name without erasing the knowledge.

Sujan Maharjan and Lu Liu · Submitted November 11, 2026 · arXiv 2611.15925v1 · cs.CL

[Read the paper on arXiv](https://arxiv.org/abs/2611.15925) · [The PDF](https://arxiv.org/pdf/2611.15925) · [The day's 12](https://thesimind.si/papers/day/2026-11-15/)

## What they found

The authors built ShadowBench, a test of whether a model can identify an entity from two described traits instead of its name. They call this task Dual-Trait Association. They report a "Shadow Gap" across all model scales, up to GPT-5.4 and Claude-Sonnet-4.6, with performance drops of over 20% when the name is removed. They then tested machine unlearning methods such as Gradient Difference and NPO. Unlearned models failed direct questions. But for popular entities, the models rebuilt the "forgotten" entity in their internal reasoning traces in over 74% of cases, as measured by the authors' Latent Entity Leakage Rate (LELR). For niche entities, shadow accuracy stayed at 60-90% even though LELR was 0%.

## Why it matters

Most tests of SI knowledge ask about things by name. This work argues that such tests may miss what a model really knows, and may overstate how well a fact has been removed. If the results hold, people who rely on unlearning to remove private or harmful data should not trust name-based checks alone. The abstract does not give details on how the benchmark was built or checked, so those claims need the full paper.

## How it changes the race

It gives evaluators a harder, name-free way to measure what frontier SI models know. It also raises the bar for any claim that unlearning truly removes knowledge.

## Safety and control

The authors say current unlearning methods act as superficial output filters and are not yet reliable for true erasure of knowledge from model weights. That matters for privacy, copyright and removing dangerous knowledge, since "forgotten" facts may still be used indirectly.

### The abstract, by the authors

Large Language Models (LLMs) have emerged as primary interfaces for factual retrieval, yet current evaluation paradigms rely almost exclusively on explicit entity names. In this work, we demonstrate that LLM knowledge is fundamentally lexically anchored: models possess extensive factual information but find it significantly difficult to retrieve without explicit name tokens. To quantify this limitation, we introduce ShadowBench, a rigorously hardened, shortcut-resistant benchmark designed to evaluate Latent Entity Association via a novel Dual-Trait Association (DTA) task. Our evaluation reveals a pervasive "Shadow Gap" across all model scalesunicodex2013up to the frontier models GPT-5.4 and Claude-Sonnet-4.6unicodex2013where removing lexical anchors causes performance drops of over 20%. Furthermore, we demonstrate that this lexical dependency exposes a critical vulnerability in AI safety. Applying ShadowBench to Machine Unlearning, we find that state-of-the-art algorithms (e.g., Gradient Difference, NPO) achieve only superficial lexical erasure. While unlearned models fail on direct queries, our novel Latent Entity Leakage Rate (LELR) metric reveals that for popular entities, models explicitly reconstruct the "forgotten" entity in their internal reasoning traces in over 74% of cases, utilizing residual shadow knowledge to solve associative tasks. Conversely, for niche entities, models retain high shadow accuracy (60-90%) via implicit attribute correlation despite 0% LELR. Ultimately, ShadowBench indicates that current unlearning paradigms create an "Illusion of Forgetting," demonstrating that existing methods act as superficial output filters and are not yet reliable for achieving true parametric erasure.

Quoted from arXiv:2611.15925. The research and its words belong to its authors.

Desk score **77** out of 100

Why it was picked

- Safety
- Reasoning
- Evals
- Language models (in the title)
- Security
- Studies frontier models (GPT-5.4)
- Accepted at NeurIPS

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Safety](https://thesimind.si/papers/?tag=safety)
- [Privacy](https://thesimind.si/papers/?tag=privacy)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)

**arXiv ID:** 2611.15925v1
**Submitted:** November 11, 2026
**Fields:** cs.CL (Computation and Language), cs.LG (Machine Learning)
**Level:** Intermediate
**Comments:** Accepted at NeurIPS 2026 Evaluations and Datasets Track

Fingerprint, SHA-256

`0c12deca99b2f60182419e86aae22a6de8841c7b972d578694c257c870f4db17`

On Solana: [Proof of Mind seal, Szprsf...FGjr](https://solscan.io/tx/SzprsfAMG4vXvYD7hrurYFce76smMcrTbLnANy8xS96bh14DoynCdjEEMZf7RPtRUgYeejS8HGnxjqtD1XuFGjr)

### The exact text fingerprinted

{"arxiv":"2611.15925","version":1,"title":"ShadowBench: Exposing Lexical Anchoring and the Illusion of Forgetting in Large Language Models","authors":["Sujan Maharjan","Lu Liu"],"day":"2026-11-15","rank":3,"score":77,"status":"written","summary":{"takeaway":"ShadowBench suggests SI models know many facts but struggle to recall them without a name, and that unlearning methods may hide a fact's name without erasing the knowledge.","found":"The authors built ShadowBench, a test of whether a model can identify an entity from two described traits instead of its name. They call this task Dual-Trait Association. They report a \"Shadow Gap\" across all model scales, up to GPT-5.4 and Claude-Sonnet-4.6, with performance drops of over 20% when the name is removed. They then tested machine unlearning methods such as Gradient Difference and NPO. Unlearned models failed direct questions. But for popular entities, the models rebuilt the \"forgotten\" entity in their internal reasoning traces in over 74% of cases, as measured by the authors' Latent Entity Leakage Rate (LELR). For niche entities, shadow accuracy stayed at 60-90% even though LELR was 0%.","matters":"Most tests of SI knowledge ask about things by name. This work argues that such tests may miss what a model really knows, and may overstate how well a fact has been removed. If the results hold, people who rely on unlearning to remove private or harmful data should not trust name-based checks alone. The abstract does not give details on how the benchmark was built or checked, so those claims need the full paper.","race":"It gives evaluators a harder, name-free way to measure what frontier SI models know. It also raises the bar for any claim that unlearning truly removes knowledge.","safety":"The authors say current unlearning methods act as superficial output filters and are not yet reliable for true erasure of knowledge from model weights. That matters for privacy, copyright and removing dangerous knowledge, since \"forgotten\" facts may still be used indirectly.","level":"intermediate","tags":["evals","benchmarks","safety","privacy","language-models","reasoning"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Sujan Maharjan, Lu Liu

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 9, 2026 · Paper of the DayPhantom Consensus: Measuring Social Reasoning in Multi-Agent LLM Decision-MakingWhen SI models see ambiguous public signals from other models, they can drift to a group answer that no one privately held, and checking private judgments can catch this.Multi-agent systemsEvalsBenchmarks](https://thesimind.si/papers/2611.08971/) · [Nov 12, 2026 · Number 4Evaluating LLM-Generated World Models for Neuro-Symbolic ReasoningSI models can write world models that run without errors, but those models often differ a lot in structure from a human-made reference, so "it runs" does not mean "it is right."EvalsBenchmarksLanguage models](https://thesimind.si/papers/2611.14419/) · [Nov 20, 2026 · Number 2AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM AgentsAutoPI is a red teaming tool that tests SI agents for prompt injection across many attack types, not just how often attacks work, and it reports higher success and coverage than recent rivals.SecurityAgentsEvals](https://thesimind.si/papers/2611.24313/)

[Number 2](https://thesimind.si/papers/2611.16378/) · [All 12 for Nov 15, 2026](https://thesimind.si/papers/day/2026-11-15/) · [Number 4](https://thesimind.si/papers/2611.16433/)
