---
title: "Forcing a Letter Suppresses Refusal... | SI Paper Desk"
url: https://thesimind.si/papers/2611.08639/
description: "In multiple-choice safety tests, forcing a model to pick a letter seems to crowd out its refusal, so these scores may measure knowledge more than safety."
---

Number 2 of 12 · November 9, 2026

# Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP

In multiple-choice safety tests, forcing a model to pick a letter seems to crowd out its refusal, so these scores may measure knowledge more than safety.

Stepan Epifantsev, Anna Antipova, Oleg Chernikov and 2 more · Submitted November 5, 2026 · arXiv 2611.08639v1 · cs.CL

[Read the paper on arXiv](https://arxiv.org/abs/2611.08639) · [The PDF](https://arxiv.org/pdf/2611.08639) · [The day's 12](https://thesimind.si/papers/day/2026-11-09/)

## What they found

The authors study why multiple-choice questions (MCQ) lead to fewer refusals than open-ended questions. They test this on the Weapons of Mass Destruction Proxy (WMDP) dataset with two models, Llama-3.1-8B and Qwen3-8B. With a tool called the tuned lens, they see that in MCQ the model commits to an answer-letter token very quickly. In open-ended questions, refusal builds up gradually. MCQ activations also sit roughly 3x further from the refusal direction than open-ended ones. When the authors ablate (switch off) the attention heads that carry the MCQ answer, refusal goes up.

## Why it matters

Many safety benchmarks use the MCQ format. If the format itself suppresses refusal, the scores may not show how a model behaves when someone asks a real question in plain text. The authors conclude that MCQ safety scores mainly measure knowledge capabilities and do not assess the safety mechanisms. The abstract only covers two 8B models and one dataset, so how far this holds elsewhere is not stated.

## How it changes the race

This is mainly a lesson for evaluation. Teams building frontier SI may need to pair MCQ safety tests with open-ended ones, so they do not overstate how safe a model is.

## Safety and control

The results suggest MCQ safety scores can hide how refusal works in practice. Safety evaluations that rely only on MCQ may give a false sense of security, so open-ended tests matter.

### The abstract, by the authors

Safety benchmarks for large language models (LLMs) commonly use multiple-choice question (MCQ) format. However, prior studies documented that MCQ format reduces refusal rates, which may significantly affect the safety behavior of the model. We provide a first mechanistic explanation of this effect on the Weapons of Mass Destruction Proxy (WMDP) dataset across two models - Llama-3.1-8B and Qwen3-8B. First, we use the tuned lens technique to show that when processing MCQ the model commits to an answer-letter token very quickly, which is in contrast to the gradual refusal build-up in open-ended formats. Additionally, we observe that activations on the MCQ format lie roughly 3x further from the refusal direction, compared with their open-ended counterparts. Moreover, ablating the attention heads that carry the MCQ format answer increases the refusal, which is a direct evidence that the MCQ answer-selection competes with refusal. These findings indicate that MCQ safety scores primarily measure knowledge capabilities and do not assess the safety-mechanisms.

Quoted from arXiv:2611.08639. The research and its words belong to its authors.

Desk score **83** out of 100

Why it was picked

- Safety (in the title)
- Interpretability (in the title)
- Evals
- Studies frontier models (Llama-3)
- Accepted at EMNLP
- Language models
- Cross-listed in 2 fields

Topics

- [Safety](https://thesimind.si/papers/?tag=safety)
- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Language models](https://thesimind.si/papers/?tag=language-models)
- [Alignment](https://thesimind.si/papers/?tag=alignment)

**arXiv ID:** 2611.08639v1
**Submitted:** November 5, 2026
**Fields:** cs.CL (Computation and Language), cs.LG (Machine Learning)
**Level:** Intermediate
**Comments:** Accepted to BlackboxNLP 2026 (EMNLP 2026 workshop). 17 pages, 15 figures

Fingerprint, SHA-256

`c2a2d1d2a4ada21755c47605f4fbc0bccf54b1cd72436aaa18d07ed9f8424931`

In SI MIND's journal since Nov 9, 2026, 23:00 UTC. It goes on Solana with the next hourly Proof of Mind seal.

### The exact text fingerprinted

{"arxiv":"2611.08639","version":1,"title":"Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP","authors":["Stepan Epifantsev","Anna Antipova","Oleg Chernikov","Leon Strelkov","Alexey Dontsov"],"day":"2026-11-09","rank":2,"score":83,"status":"written","summary":{"takeaway":"In multiple-choice safety tests, forcing a model to pick a letter seems to crowd out its refusal, so these scores may measure knowledge more than safety.","found":"The authors study why multiple-choice questions (MCQ) lead to fewer refusals than open-ended questions. They test this on the Weapons of Mass Destruction Proxy (WMDP) dataset with two models, Llama-3.1-8B and Qwen3-8B. With a tool called the tuned lens, they see that in MCQ the model commits to an answer-letter token very quickly. In open-ended questions, refusal builds up gradually. MCQ activations also sit roughly 3x further from the refusal direction than open-ended ones. When the authors ablate (switch off) the attention heads that carry the MCQ answer, refusal goes up.","matters":"Many safety benchmarks use the MCQ format. If the format itself suppresses refusal, the scores may not show how a model behaves when someone asks a real question in plain text. The authors conclude that MCQ safety scores mainly measure knowledge capabilities and do not assess the safety mechanisms. The abstract only covers two 8B models and one dataset, so how far this holds elsewhere is not stated.","race":"This is mainly a lesson for evaluation. Teams building frontier SI may need to pair MCQ safety tests with open-ended ones, so they do not overstate how safe a model is.","safety":"The results suggest MCQ safety scores can hide how refusal works in practice. Safety evaluations that rely only on MCQ may give a false sense of security, so open-ended tests matter.","level":"intermediate","tags":["safety","interpretability","evals","benchmarks","language-models","alignment"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Stepan Epifantsev, Anna Antipova, Oleg Chernikov, Leon Strelkov, Alexey Dontsov

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 9, 2026 · Paper of the DayPhantom Consensus: Measuring Social Reasoning in Multi-Agent LLM Decision-MakingWhen SI models see ambiguous public signals from other models, they can drift to a group answer that no one privately held, and checking private judgments can catch this.Multi-agent systemsEvalsBenchmarks](https://thesimind.si/papers/2611.08971/) · [Nov 9, 2026 · Number 4HomeLoop: A Simulation Framework and Benchmark for Robot-IoT Coordination in Everyday Domestic Assistance and SafetyHomeLoop is a new simulated home benchmark where SI robots must work with smart devices and handle ambiguous alerts. The best of six models scored 61.9%, and models often over-reacted.RoboticsBenchmarksEvals](https://thesimind.si/papers/2611.08914/) · [Nov 9, 2026 · Number 5I Think, Therefore I Attack: Evaluating Thought Injection Attacks on Reasoning AgentsAttacker text written to look like a reasoning model's own thoughts can hijack SI agents, reaching up to 50% success in a test range, while plain prompt injection of the same goal never worked.SecurityAgentsReasoning](https://thesimind.si/papers/2611.08811/)

[Number 1](https://thesimind.si/papers/2611.08971/) · [All 12 for Nov 9, 2026](https://thesimind.si/papers/day/2026-11-09/) · [Number 3](https://thesimind.si/papers/2611.08950/)