---
title: "HiddenPathQA: A Benchmark for Knowledge... | SI Paper Desk"
url: https://thesimind.si/papers/2611.11802/
description: "Many SI question answering systems over knowledge graphs lean on wording that gives away the path to the answer. HiddenPathQA tests them without that help..."
---

Number 12 of 12 · November 11, 2026

# HiddenPathQA: A Benchmark for Knowledge Graph Question Answering When Questions Hide Their Paths

Many SI question answering systems over knowledge graphs lean on wording that gives away the path to the answer. HiddenPathQA tests them without that help, and a chain-level method does better.

Sumin Jo and Edward Choi · Submitted November 9, 2026 · arXiv 2611.11802v1 · cs.CL

[Read the paper on arXiv](https://arxiv.org/abs/2611.11802) · [The PDF](https://arxiv.org/pdf/2611.11802) · [The day's 12](https://thesimind.si/papers/day/2026-11-11/)

## What they found

The authors say current tests for knowledge graph question answering (KGQA) use questions whose words reveal the relations on the correct path. Real users often do not know the schema. So the authors built HiddenPathQA, a manually verified set of 1,055 questions spanning 2-6 hops and multiple domains, where the wording does not name the path's relations. They also propose Holistic Path Selection, which compares a question to whole candidate relation chains in the local subgraph instead of scoring one hop at a time. They built two versions, TieredHolistic and FlatHolistic. It beats six baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings.

## Why it matters

A knowledge graph is a store of facts linked by named relations. SI systems that answer questions from it may look strong on tests only because the question hints at the route. This paper tests that assumption directly. The pseudoword stress test swaps entity names for meaningless placeholders. The method's advantage holds there, so the authors argue memorization of entities does not explain the gains alone. The abstract gives no detail on failure cases or limits.

## How it changes the race

This is a benchmark and method result for retrieval over structured knowledge. If it holds, builders of SI systems that rely on knowledge graphs get a harder, more realistic test, and a hint that choosing whole chains beats hop-by-hop choices. It is a focused advance, not a change in general capability.

## Safety and control

The abstract does not address safety directly. It does show that systems may rely on hidden shortcuts in evaluation, which matters for trusting test scores. The example it gives is a medical-style question.

### The abstract, by the authors

Knowledge graph question answering (KGQA) systems are typically evaluated on questions whose surface tokens lexically expose the gold KG path. Users, however, may ask questions without knowing the underlying schema: "Which condition could Andrew have due to family history?" compresses a 4-hop chain "Andrew -[father]-> parent -[father]-> grandparent -[medical condition]-> disease -[subclass of]-> hereditary disorder" into a single phrase (family history) that names none of those relations. We identify this collapse of path-question alignment as a blind spot in existing KGQA evaluation. Three major LLM-KG paradigms, hop-by-hop exploration, plan-then-retrieve, and similarity-based retrieval, implicitly rely on such lexical cues, an assumption rarely tested in practice. We make three contributions. (i) We introduce HiddenPathQA, a manually verified benchmark of 1,055 questions spanning 2-6 hops and multiple domains, constructed so that question wording does not reveal the gold path's relations. (ii) We propose Holistic Path Selection, which compares questions against candidate relation chains in the local subgraph as complete units, replacing irreversible per-hop relation scoring with globally informed chain-level reasoning. We develop two implementations, TieredHolistic and FlatHolistic. (iii) We conduct a pseudoword stress test that replaces entity names with meaningless placeholders to distinguish KG-grounded reasoning from entity-level memorization. Holistic Path Selection outperforms six strong KGQA baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings. Its advantage under pseudoword replacement shows that the gains cannot be explained by entity-level memorization alone.

Quoted from arXiv:2611.11802. The research and its words belong to its authors.

Desk score **67** out of 100

Why it was picked

- Evals (in the title)
- Alignment
- Reasoning
- Language models
- Cross-listed in 2 fields

Topics

- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Retrieval](https://thesimind.si/papers/?tag=retrieval)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)

**arXiv ID:** 2611.11802v1
**Submitted:** November 9, 2026
**Fields:** cs.CL (Computation and Language), cs.AI (Artificial Intelligence)
**Level:** Intermediate

Fingerprint, SHA-256

`ef1e0943936f8f8a34e9d93011d4ccdb82692dc17a04259adc09e2e2d4bd9e6b`

On Solana: [Proof of Mind seal, ipj4pc...z6te](https://solscan.io/tx/ipj4pcQsmFjnBmF5dFmUSA15Eex18bkxb83FjQuvHnS4DCz6yJoWcESM8bwWT8cMMh3f4GmDjbebLEAsauqz6te)

### The exact text fingerprinted

{"arxiv":"2611.11802","version":1,"title":"HiddenPathQA: A Benchmark for Knowledge Graph Question Answering When Questions Hide Their Paths","authors":["Sumin Jo","Edward Choi"],"day":"2026-11-11","rank":12,"score":67,"status":"written","summary":{"takeaway":"Many SI question answering systems over knowledge graphs lean on wording that gives away the path to the answer. HiddenPathQA tests them without that help, and a chain-level method does better.","found":"The authors say current tests for knowledge graph question answering (KGQA) use questions whose words reveal the relations on the correct path. Real users often do not know the schema. So the authors built HiddenPathQA, a manually verified set of 1,055 questions spanning 2-6 hops and multiple domains, where the wording does not name the path's relations. They also propose Holistic Path Selection, which compares a question to whole candidate relation chains in the local subgraph instead of scoring one hop at a time. They built two versions, TieredHolistic and FlatHolistic. It beats six baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings.","matters":"A knowledge graph is a store of facts linked by named relations. SI systems that answer questions from it may look strong on tests only because the question hints at the route. This paper tests that assumption directly. The pseudoword stress test swaps entity names for meaningless placeholders. The method's advantage holds there, so the authors argue memorization of entities does not explain the gains alone. The abstract gives no detail on failure cases or limits.","race":"This is a benchmark and method result for retrieval over structured knowledge. If it holds, builders of SI systems that rely on knowledge graphs get a harder, more realistic test, and a hint that choosing whole chains beats hop-by-hop choices. It is a focused advance, not a change in general capability.","safety":"The abstract does not address safety directly. It does show that systems may rely on hidden shortcuts in evaluation, which matters for trusting test scores. The example it gives is a medical-style question.","level":"intermediate","tags":["benchmarks","retrieval","evals","language-models","reasoning"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Sumin Jo, Edward Choi

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 9, 2026 · Paper of the DayPhantom Consensus: Measuring Social Reasoning in Multi-Agent LLM Decision-MakingWhen SI models see ambiguous public signals from other models, they can drift to a group answer that no one privately held, and checking private judgments can catch this.Multi-agent systemsEvalsBenchmarks](https://thesimind.si/papers/2611.08971/) · [Nov 11, 2026 · Number 2Benchmarking language models and agents for scientific reasoning in evolving materials laboratoriesMatScope is a new test showing that SI models handle given calculations and physical explanations well, but struggle to tell close material states apart and to reason with numbers.BenchmarksScienceEvals](https://thesimind.si/papers/2611.12014/) · [Nov 9, 2026 · Number 2Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDPIn multiple-choice safety tests, forcing a model to pick a letter seems to crowd out its refusal, so these scores may measure knowledge more than safety.SafetyInterpretabilityEvals](https://thesimind.si/papers/2611.08639/)

[Number 11](https://thesimind.si/papers/2611.11724/) · [All 12 for Nov 11, 2026](https://thesimind.si/papers/day/2026-11-11/)