---
title: "Reasoning-Trace Analysis Using Large... | SI Paper Desk"
url: https://thesimind.si/papers/2611.26422/
description: "Asking an SI model to show its reasoning for each Q-matrix entry lets people inspect why it made each choice, though these matrices matched the reference..."
---

Number 5 of 12 · November 21, 2026

# Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction

Asking an SI model to show its reasoning for each Q-matrix entry lets people inspect why it made each choice, though these matrices matched the reference less well than directly generated ones.

Jihong Zhang, Zijing Jin and Mingfeng Xue · Submitted November 19, 2026 · arXiv 2611.26422v1 · cs.AI

[Read the paper on arXiv](https://arxiv.org/abs/2611.26422) · [The PDF](https://arxiv.org/pdf/2611.26422) · [The day's 12](https://thesimind.si/papers/day/2026-11-21/)

## What they found

The authors propose Reasoning-Trace Analysis (RTA). An LLM (large language model) writes out step-by-step reasoning, with each step labeled by the skill it uses. Those steps are then analyzed to rebuild and compare each item-skill assignment in a Q-matrix, which is a table linking test items to the skills they require. In a simulation with perturbed fraction subtraction items, four LLMs recovered the required attributes and withheld attributes the items did not require. In an empirical study with 11 items and 536 examinees, Q-matrices built from reasoning steps overlapped with the reference matrix at a mean of .809. Directly generated matrices scored .877, and the reasoning-based ones varied more across replications. Reading the traces tied disagreements to item-type coding, labeling rules, and how operations were written as steps. Matrices with similar overlap fit examinee responses differently.

## Why it matters

Test designers may use SI to draft Q-matrices, but a finished matrix does not show why each skill was assigned. This work gives a way to inspect those reasons. It also shows that agreement with an expert matrix and fit to student data can tell different stories, so both checks are needed. The abstract says RTA is meant to supplement expert review and empirical validation, not replace them.

## How it changes the race

This is a narrow result for educational measurement, not frontier SI capability. It is an incremental step toward making SI-generated outputs easier to audit in a specific field.

## Safety and control

The paper speaks to responsible use of SI in measurement by making each decision inspectable. The abstract does not say whether the written reasoning faithfully reflects how the model reached its answer, so the traces should be treated as evidence to check, not proof.

### The abstract, by the authors

Large language models (LLMs) can generate Q-matrices for diagnostic classification models that approximate expert specifications, but evaluations have focused on the final matrices, leaving the basis of individual item-attribute assignments unexamined. This opacity limits the responsible use of AI in measurement. We propose Reasoning-Trace Analysis (RTA), a framework in which an LLM generates attribute-labeled reasoning steps through chain-of-thought (CoT) prompting and these steps are analyzed to reconstruct, compare, and inspect each assignment. RTA is intended to supplement expert review and empirical validation. In a simulation study with perturbed fraction subtraction items, four LLMs recovered the required attributes and withheld attributes the items did not require. In an empirical study with 11 fraction subtraction items, responses from 536 examinees, four LLMs, up to three temperatures, and 100 replications per condition, CoT-derived Q-matrices overlapped substantially with the reference Q-matrix (mean overlap .809) but less than directly generated Q-matrices (.877), and they varied more across replications. Inspection of the traces linked disagreements to item-type coding, labeling rules, and how operations were expressed as steps. Q-matrices with similar overlap differed in fit to examinee responses. RTA provides inspectable evidence that matrix-level agreement and fit do not.

Quoted from arXiv:2611.26422. The research and its words belong to its authors.

Desk score **75** out of 100

Why it was picked

- Interpretability (in the title)
- Reasoning (in the title)
- Evals
- Language models (in the title)

Topics

- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Education](https://thesimind.si/papers/?tag=education)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)

**arXiv ID:** 2611.26422v1
**Submitted:** November 19, 2026
**Fields:** cs.AI (Artificial Intelligence)
**Level:** Intermediate
**Comments:** 5 figures

Fingerprint, SHA-256

`38ee66ec7e48c9c732258e0bd3fdde40ecc5909ad47b5525386285ca3e391e2c`

On Solana: [Proof of Mind seal, beudvj...We9f](https://solscan.io/tx/beudvjGVbBTGVZsMaL2MgMJjuMy9dK2a2K3ETkw5AFwjoARs9PQedxMxfrBaiJ5MetjT2AHxuhfTCyP34eQWe9f)

### The exact text fingerprinted

{"arxiv":"2611.26422","version":1,"title":"Reasoning-Trace Analysis Using Large Language Models for Interpretable Q-Matrix Construction","authors":["Jihong Zhang","Zijing Jin","Mingfeng Xue"],"day":"2026-11-21","rank":5,"score":75,"status":"written","summary":{"takeaway":"Asking an SI model to show its reasoning for each Q-matrix entry lets people inspect why it made each choice, though these matrices matched the reference less well than directly generated ones.","found":"The authors propose Reasoning-Trace Analysis (RTA). An LLM (large language model) writes out step-by-step reasoning, with each step labeled by the skill it uses. Those steps are then analyzed to rebuild and compare each item-skill assignment in a Q-matrix, which is a table linking test items to the skills they require. In a simulation with perturbed fraction subtraction items, four LLMs recovered the required attributes and withheld attributes the items did not require. In an empirical study with 11 items and 536 examinees, Q-matrices built from reasoning steps overlapped with the reference matrix at a mean of .809. Directly generated matrices scored .877, and the reasoning-based ones varied more across replications. Reading the traces tied disagreements to item-type coding, labeling rules, and how operations were written as steps. Matrices with similar overlap fit examinee responses differently.","matters":"Test designers may use SI to draft Q-matrices, but a finished matrix does not show why each skill was assigned. This work gives a way to inspect those reasons. It also shows that agreement with an expert matrix and fit to student data can tell different stories, so both checks are needed. The abstract says RTA is meant to supplement expert review and empirical validation, not replace them.","race":"This is a narrow result for educational measurement, not frontier SI capability. It is an incremental step toward making SI-generated outputs easier to audit in a specific field.","safety":"The paper speaks to responsible use of SI in measurement by making each decision inspectable. The abstract does not say whether the written reasoning faithfully reflects how the model reached its answer, so the traces should be treated as evidence to check, not proof.","level":"intermediate","tags":["interpretability","education","language-models","evals","reasoning"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Jihong Zhang, Zijing Jin, Mingfeng Xue

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 16, 2026 · Number 11Tracing Knowledge and Reasoning in LLMYang Wang and colleagues propose a way to look inside language models while they take benchmarks, to show whether a task needs recall of facts or real reasoning, not just a score.EvalsInterpretabilityBenchmarks](https://thesimind.si/papers/2611.17461/) · [Nov 16, 2026 · Number 2Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM AgentsSoteria checks an SI agent's plan against written rules before any tool runs, then enforces those rules during execution. The authors report more completed tasks and no policy violations.AgentsSafetyReasoning](https://thesimind.si/papers/2611.17880/) · [Nov 18, 2026 · Number 3Abstention Direction in Large Language Models: Mechanistic Interpretability of Abstention Across Query Ambiguity, Knowledge Limits, and Safety AlignmentThe authors find directions inside language models that control abstention (declining to answer) for unclear questions, unknown facts and harmful requests, and these directions partly overlap.InterpretabilityAlignmentSafety](https://thesimind.si/papers/2611.20441/)

[Number 4](https://thesimind.si/papers/2611.26545/) · [All 12 for Nov 21, 2026](https://thesimind.si/papers/day/2026-11-21/) · [Number 6](https://thesimind.si/papers/2611.26590/)
