---
title: "Sophon: A Procedural Diagnostic for... | SI Paper Desk"
url: https://thesimind.si/papers/2611.14838/
description: "Open-weight vision-language models fail simple spatial tests with no familiar objects, but the needed information is inside them, and targeted fine-tuning..."
---

Number 3 of 12 · November 13, 2026

# Sophon: A Procedural Diagnostic for Spatial Reasoning in Vision-Language Models

Open-weight vision-language models fail simple spatial tests with no familiar objects, but the needed information is inside them, and targeted fine-tuning lets 7B-class models reach frontier-level accuracy on Sophon.

J. Zach Gazak, Ryan Swindle and Justin Fletcher · Submitted November 10, 2026 · arXiv 2611.14838v1 · cs.CV

[Read the paper on arXiv](https://arxiv.org/abs/2611.14838) · [The PDF](https://arxiv.org/pdf/2611.14838) · [The day's 12](https://thesimind.si/papers/day/2026-11-13/)

## What they found

The authors built Sophon, a generated test using colored point clouds in three coordinate systems (1D linear, 2D Cartesian, 2D polar) and four question types. Four open models (LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Idefics3) scored 47-52%, while Opus 4.6, Gemini 2.5 Pro and GPT-5.4 scored 77-83% and humans 84-94%. These same open models score 85-94% on spatial benchmarks such as BLINK Spatial and What's Up CLEVR. Linear probes (simple tests of what a layer encodes) found the spatial information in the vision encoder at 83-85% and in the language decoder at 80-86%. At the output stage, models raised the right answer tokens but chose among them randomly. LoRA fine-tuning of a mid-stack part of the language tower brought 7B-class models to frontier aggregate accuracy on Sophon.

## Why it matters

Strong scores on common spatial benchmarks may reflect object and scene shortcuts rather than real geometric reasoning. Sophon strips those cues away, so it can test this more directly. The authors argue the problem is routing, meaning the signal is present but not used well, rather than a missing ability. The abstract does not say how well the fix carries over to other tasks or to real images.

## How it changes the race

If this holds, builders could fix some spatial weaknesses in smaller open models with cheap targeted fine-tuning instead of larger training runs. It also suggests that benchmark scores alone can overstate what open models can do.

## Safety and control

The work is partly interpretability: it traces where in the model the failure happens. It also shows that high benchmark scores can hide real weaknesses, which matters for trusting evaluations.

### The abstract, by the authors

Open-weight Vision-Language Models (VLMs) score 85-94% on spatial benchmarks (BLINK Spatial, What's Up CLEVR) yet collapse to chance on simple geometric questions when object, scene, and texture priors are stripped away. We introduce and open-source Sophon, a procedurally generated diagnostic of paired colored point-cloud stimuli across three coordinate systems (1D linear, 2D Cartesian, 2D polar) and four question types (binary, categorical, comparative, quantitative). Across four open architectures (LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Idefics3), accuracy clusters at 47-52% while frontier models (Opus 4.6, Gemini 2.5 Pro, GPT-5.4) clear 77-83% and humans reach 84-94%. We use linear probes and low-rank adaptation (LoRA) to trace the open-weight failures. Probes of the vision encoder (ViT) and language decoder (LLM) recover Sophon spatial information at 83-85% and 80-86% accuracy, respectively, and the language tower's reasoning circuit answers text-only controls (point clouds as sets of coordinates) at +40 pp over visual stimuli. Language tower per-layer attention attribution localizes visual integration to mid-stack layers. At readout, models elevate the correct answer-set tokens but select among them randomly. We localize the mechanism by tuning individual and grouped components across every architecture's stack, including the vision tower (ViT workspace and reader), the projector, and the language tower (LLM workspace and reader). These experiments demonstrate that this mid-stack workspace holds circuitry capable of organizing the spatial signal so that the frozen reader can decode it. LoRA fine-tuning adjusts that circuitry such that 7B-class VLMs answer Sophon spatial reasoning questions at frontier aggregate accuracy. This work suggests that these models do not have a capability gap for spatial reasoning but suffer from a routing bottleneck that targeted fine-tuning repairs.

Quoted from arXiv:2611.14838. The research and its words belong to its authors.

Desk score **79** out of 100

Why it was picked

- Reasoning (in the title)
- Interpretability
- Evals
- Language models (in the title)
- Studies frontier models (Qwen2.5)
- Accepted at NeurIPS
- Cross-listed in 2 fields

Topics

- [Evals](https://thesimind.si/papers/?tag=evals)
- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Vision](https://thesimind.si/papers/?tag=vision)
- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Multimodal](https://thesimind.si/papers/?tag=multimodal)
- [Training](https://thesimind.si/papers/?tag=training)

**arXiv ID:** 2611.14838v1
**Submitted:** November 10, 2026
**Fields:** cs.CV (Computer Vision), cs.LG (Machine Learning)
**Level:** Intermediate
**Comments:** NeurIPS 2026 (Evaluations & Datasets Track). 10 page main text, 44 pages total. Code and data: [https://github.com/zgazak/sophon](https://github.com/zgazak/sophon)

Fingerprint, SHA-256

`5be190e871fbc9fca2c1cd7204bbb06526c9ffc2793333901fec33b4143d3e9f`

On Solana: [Proof of Mind seal, 5PD2kk...Cqj6](https://solscan.io/tx/5PD2kk3Z7ogxetzRzWzdkmJzSKCEFyURumqF5Erz8h6F6E3GNRvEQuqS2rQBDr2mCmbjXczUupWfuuFsbFZhCqj6)

### The exact text fingerprinted

{"arxiv":"2611.14838","version":1,"title":"Sophon: A Procedural Diagnostic for Spatial Reasoning in Vision-Language Models","authors":["J. Zach Gazak","Ryan Swindle","Justin Fletcher"],"day":"2026-11-13","rank":3,"score":79,"status":"written","summary":{"takeaway":"Open-weight vision-language models fail simple spatial tests with no familiar objects, but the needed information is inside them, and targeted fine-tuning lets 7B-class models reach frontier-level accuracy on Sophon.","found":"The authors built Sophon, a generated test using colored point clouds in three coordinate systems (1D linear, 2D Cartesian, 2D polar) and four question types. Four open models (LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Idefics3) scored 47-52%, while Opus 4.6, Gemini 2.5 Pro and GPT-5.4 scored 77-83% and humans 84-94%. These same open models score 85-94% on spatial benchmarks such as BLINK Spatial and What's Up CLEVR. Linear probes (simple tests of what a layer encodes) found the spatial information in the vision encoder at 83-85% and in the language decoder at 80-86%. At the output stage, models raised the right answer tokens but chose among them randomly. LoRA fine-tuning of a mid-stack part of the language tower brought 7B-class models to frontier aggregate accuracy on Sophon.","matters":"Strong scores on common spatial benchmarks may reflect object and scene shortcuts rather than real geometric reasoning. Sophon strips those cues away, so it can test this more directly. The authors argue the problem is routing, meaning the signal is present but not used well, rather than a missing ability. The abstract does not say how well the fix carries over to other tasks or to real images.","race":"If this holds, builders could fix some spatial weaknesses in smaller open models with cheap targeted fine-tuning instead of larger training runs. It also suggests that benchmark scores alone can overstate what open models can do.","safety":"The work is partly interpretability: it traces where in the model the failure happens. It also shows that high benchmark scores can hide real weaknesses, which matters for trusting evaluations.","level":"intermediate","tags":["evals","benchmarks","vision","interpretability","multimodal","training"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

J. Zach Gazak, Ryan Swindle, Justin Fletcher

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 10, 2026 · Number 9Mental Imagery That Matters: Curing Latent Collapse in Multimodal ReasoningSome SI models that reason with internal visual states barely use them. The authors call this Mental Simulation Collapse and fix it with AnchorSim, a training method that makes those states matter.MultimodalReasoningInterpretability](https://thesimind.si/papers/2611.10423/) · [Nov 11, 2026 · Number 5How a Model's Architecture and Performance Influences Explainability: A Study of Grad-CAM and CRAFT in CNN-Based Image ClassificationIn this study of image models, explanation quality changed with both training progress and architecture: ResNet50 and ConvNeXt-Tiny gave more faithful, steadier explanations over training, while VGG16 gave simpler but less faithful ones.InterpretabilityEvalsVision](https://thesimind.si/papers/2611.12075/) · [Nov 20, 2026 · Number 5SixthView: A Benchmark for Hidden-View Reconstruction and 3D Assembly ReasoningOn SixthView, a new test of 3D assembly reasoning, three vision-language models did far better with exact text descriptions than with images alone, and images added almost nothing once the text was given.BenchmarksVisionMultimodal](https://thesimind.si/papers/2611.23891/)

[Number 2](https://thesimind.si/papers/2611.14881/) · [All 12 for Nov 13, 2026](https://thesimind.si/papers/day/2026-11-13/) · [Number 4](https://thesimind.si/papers/2611.14009/)
