What they found
The authors say current tests for knowledge graph question answering (KGQA) use questions whose words reveal the relations on the correct path. Real users often do not know the schema. So the authors built HiddenPathQA, a manually verified set of 1,055 questions spanning 2-6 hops and multiple domains, where the wording does not name the path's relations. They also propose Holistic Path Selection, which compares a question to whole candidate relation chains in the local subgraph instead of scoring one hop at a time. They built two versions, TieredHolistic and FlatHolistic. It beats six baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings.
Why it matters
A knowledge graph is a store of facts linked by named relations. SI systems that answer questions from it may look strong on tests only because the question hints at the route. This paper tests that assumption directly. The pseudoword stress test swaps entity names for meaningless placeholders. The method's advantage holds there, so the authors argue memorization of entities does not explain the gains alone. The abstract gives no detail on failure cases or limits.
How it changes the race
This is a benchmark and method result for retrieval over structured knowledge. If it holds, builders of SI systems that rely on knowledge graphs get a harder, more realistic test, and a hint that choosing whole chains beats hop-by-hop choices. It is a focused advance, not a change in general capability.
Safety and control
The abstract does not address safety directly. It does show that systems may rely on hidden shortcuts in evaluation, which matters for trusting test scores. The example it gives is a medical-style question.
The abstract, by the authors
Knowledge graph question answering (KGQA) systems are typically evaluated on questions whose surface tokens lexically expose the gold KG path. Users, however, may ask questions without knowing the underlying schema: "Which condition could Andrew have due to family history?" compresses a 4-hop chain "Andrew -[father]-> parent -[father]-> grandparent -[medical condition]-> disease -[subclass of]-> hereditary disorder" into a single phrase (family history) that names none of those relations. We identify this collapse of path-question alignment as a blind spot in existing KGQA evaluation. Three major LLM-KG paradigms, hop-by-hop exploration, plan-then-retrieve, and similarity-based retrieval, implicitly rely on such lexical cues, an assumption rarely tested in practice. We make three contributions. (i) We introduce HiddenPathQA, a manually verified benchmark of 1,055 questions spanning 2-6 hops and multiple domains, constructed so that question wording does not reveal the gold path's relations. (ii) We propose Holistic Path Selection, which compares questions against candidate relation chains in the local subgraph as complete units, replacing irreversible per-hop relation scoring with globally informed chain-level reasoning. We develop two implementations, TieredHolistic and FlatHolistic. (iii) We conduct a pseudoword stress test that replaces entity names with meaningless placeholders to distinguish KG-grounded reasoning from entity-level memorization. Holistic Path Selection outperforms six strong KGQA baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings. Its advantage under pseudoword replacement shows that the gains cannot be explained by entity-level memorization alone.
Quoted from arXiv:2611.11802. The research and its words belong to its authors.