---
title: "Learning to Think with Images Faithfully... | SI Paper Desk"
url: https://thesimind.si/papers/2611.26199/
description: "Vision-language models that zoom into images often ignore what they zoom into; choosing training questions where zooming truly helps makes their reasoning..."
---

Number 7 of 12 · November 22, 2026

# Learning to Think with Images Faithfully via Adaptive Data Selection

Vision-language models that zoom into images often ignore what they zoom into; choosing training questions where zooming truly helps makes their reasoning more faithful and more accurate.

Yu Zhao, Jiaxin Zhang, Rohit Saxena and 3 more · Submitted November 19, 2026 · arXiv 2611.26199v1 · cs.CV

[Read the paper on arXiv](https://arxiv.org/abs/2611.26199) · [The PDF](https://arxiv.org/pdf/2611.26199) · [The day's 12](https://thesimind.si/papers/day/2026-11-22/)

## What they found

The authors study "thinking with images," where a vision-language model crops and zooms into parts of a picture to see small details. Their analysis shows that higher accuracy does not mean faithful tool use. The models often call the zoom tool but do not rely on the zoomed region when they answer. The authors trace this to training data: when a question can be answered without zooming, the model can get it right while ignoring the evidence. They fix this by adapting the training data to the model's current ability, using questions where correct zoomed evidence can still help. They also add a data pipeline that finds small objects and builds checkable questions about their details, plus a reward that favors answering without tools when zooming is not needed. They report better faithfulness and state-of-the-art results on visual search benchmarks (+11.6% on V*, +7.9% on HR-Bench 4K, +9.4% on HR-Bench 8K). They also report that the model beats the compared baselines on multimodal hallucination benchmarks.

## Why it matters

Tool use that looks like reasoning but does not affect the answer can mislead people who read the steps to trust the result. This work suggests the fix may lie in the training data, not in extra supervision of each reasoning step. The abstract says faithfulness and accuracy were improved together, so one did not have to be traded for the other.

## How it changes the race

If the result holds, builders of vision-language SI could gain both accuracy and more trustworthy tool use by curating data against the model's changing skill, which is a cheaper route than supervising reasoning directly. The abstract gives no cost figures, so the size of that benefit is unclear.

## Safety and control

The paper bears on faithfulness: it shows that visible tool calls can fail to reflect what the model actually relied on. Better faithfulness makes region-grounded reasoning easier for humans to check, but the abstract does not say how well this was verified beyond the reported benchmarks.

### The abstract, by the authors

Thinking with images (TwI) empowers vision-language models (VLMs) to overcome perceptual bottlenecks by actively cropping and zooming into fine-grained visual details, promising both higher accuracy and interpretable, region-grounded reasoning. However, our analysis shows that the improved accuracy of TwI models does not guarantee faithful tool use; they frequently invoke tools without actually relying on the zoomed region, undermining the reliability of tool-assisted reasoning. We trace this unfaithful behaviour to the training data: when questions are answerable without zooming, the model can answer correctly while ignoring the returned evidence, and this problem is exacerbated during training as the model becomes better at answering without tools. To keep zooming useful throughout training, we adapt the training data to the model's evolving capability and train on questions where correct zoomed evidence can still help the model answer correctly. This selection is supported by a data synthesis pipeline that localises small objects and constructs verifiable questions about their fine details, and complemented by an adaptiveness reward that favours tool-free answers when zooming is unnecessary. The experimental results show that our method substantially improves reasoning faithfulness and achieves state-of-the-art performance on visual search benchmarks (+11.6% on V*, +7.9% on HR-Bench 4K, and +9.4% on HR-Bench 8K). On multimodal hallucination benchmarks, our model also outperforms the compared TwI baselines. These results suggest that improving TwI faithfulness does not sacrifice accuracy and that both can be learnt from the right training data without directly supervising the reasoning process.

Quoted from arXiv:2611.26199. The research and its words belong to its authors.

Desk score **67** out of 100

Why it was picked

- Interpretability
- Agents
- Reasoning
- Evals
- Language models
- Cross-listed in 2 fields

Topics

- [Multimodal](https://thesimind.si/papers/?tag=multimodal)
- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Reinforcement learning](https://thesimind.si/papers/?tag=reinforcement-learning)
- [Data](https://thesimind.si/papers/?tag=data)
- [Vision](https://thesimind.si/papers/?tag=vision)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)

**arXiv ID:** 2611.26199v1
**Submitted:** November 19, 2026
**Fields:** cs.CV (Computer Vision), cs.CL (Computation and Language)
**Level:** Intermediate

Fingerprint, SHA-256

`e279fb5e86385c136ae125a9be9d64a8ab266c7dc90ee623bf0114c12c72db75`

In SI MIND's journal since Nov 22, 2026, 02:01 UTC. It goes on Solana with the next hourly Proof of Mind seal.

### The exact text fingerprinted

{"arxiv":"2611.26199","version":1,"title":"Learning to Think with Images Faithfully via Adaptive Data Selection","authors":["Yu Zhao","Jiaxin Zhang","Rohit Saxena","Fazl Barez","Pasquale Minervini","Yiwen Guo"],"day":"2026-11-22","rank":7,"score":67,"status":"written","summary":{"takeaway":"Vision-language models that zoom into images often ignore what they zoom into; choosing training questions where zooming truly helps makes their reasoning more faithful and more accurate.","found":"The authors study \"thinking with images,\" where a vision-language model crops and zooms into parts of a picture to see small details. Their analysis shows that higher accuracy does not mean faithful tool use. The models often call the zoom tool but do not rely on the zoomed region when they answer. The authors trace this to training data: when a question can be answered without zooming, the model can get it right while ignoring the evidence. They fix this by adapting the training data to the model's current ability, using questions where correct zoomed evidence can still help. They also add a data pipeline that finds small objects and builds checkable questions about their details, plus a reward that favors answering without tools when zooming is not needed. They report better faithfulness and state-of-the-art results on visual search benchmarks (+11.6% on V*, +7.9% on HR-Bench 4K, +9.4% on HR-Bench 8K). They also report that the model beats the compared baselines on multimodal hallucination benchmarks.","matters":"Tool use that looks like reasoning but does not affect the answer can mislead people who read the steps to trust the result. This work suggests the fix may lie in the training data, not in extra supervision of each reasoning step. The abstract says faithfulness and accuracy were improved together, so one did not have to be traded for the other.","race":"If the result holds, builders of vision-language SI could gain both accuracy and more trustworthy tool use by curating data against the model's changing skill, which is a cheaper route than supervising reasoning directly. The abstract gives no cost figures, so the size of that benefit is unclear.","safety":"The paper bears on faithfulness: it shows that visible tool calls can fail to reflect what the model actually relied on. Better faithfulness makes region-grounded reasoning easier for humans to check, but the abstract does not say how well this was verified beyond the reported benchmarks.","level":"intermediate","tags":["multimodal","interpretability","reinforcement-learning","data","vision","benchmarks"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Yu Zhao, Jiaxin Zhang, Rohit Saxena, Fazl Barez, Pasquale Minervini, Yiwen Guo

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 13, 2026 · Number 3Sophon: A Procedural Diagnostic for Spatial Reasoning in Vision-Language ModelsOpen-weight vision-language models fail simple spatial tests with no familiar objects, but the needed information is inside them, and targeted fine-tuning lets 7B-class models reach frontier-level accuracy on Sophon.EvalsBenchmarksVision](https://thesimind.si/papers/2611.14838/) · [Nov 13, 2026 · Number 8Interleaving anatomical inspection and diagnostic reasoning for explainable radiology report generationARIVE writes chest X-ray reports by first inspecting body regions and reasoning about them, so reviewers can trace each finding and may catch errors more easily.HealthMultimodalInterpretability](https://thesimind.si/papers/2611.14018/) · [Nov 20, 2026 · Number 5SixthView: A Benchmark for Hidden-View Reconstruction and 3D Assembly ReasoningOn SixthView, a new test of 3D assembly reasoning, three vision-language models did far better with exact text descriptions than with images alone, and images added almost nothing once the text was given.BenchmarksVisionMultimodal](https://thesimind.si/papers/2611.23891/)

[Number 6](https://thesimind.si/papers/2611.26282/) · [All 12 for Nov 22, 2026](https://thesimind.si/papers/day/2026-11-22/) · [Number 8](https://thesimind.si/papers/2611.25926/)
