What they found
The authors built Sophon, a generated test using colored point clouds in three coordinate systems (1D linear, 2D Cartesian, 2D polar) and four question types. Four open models (LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Idefics3) scored 47-52%, while Opus 4.6, Gemini 2.5 Pro and GPT-5.4 scored 77-83% and humans 84-94%. These same open models score 85-94% on spatial benchmarks such as BLINK Spatial and What's Up CLEVR. Linear probes (simple tests of what a layer encodes) found the spatial information in the vision encoder at 83-85% and in the language decoder at 80-86%. At the output stage, models raised the right answer tokens but chose among them randomly. LoRA fine-tuning of a mid-stack part of the language tower brought 7B-class models to frontier aggregate accuracy on Sophon.
Why it matters
Strong scores on common spatial benchmarks may reflect object and scene shortcuts rather than real geometric reasoning. Sophon strips those cues away, so it can test this more directly. The authors argue the problem is routing, meaning the signal is present but not used well, rather than a missing ability. The abstract does not say how well the fix carries over to other tasks or to real images.
How it changes the race
If this holds, builders could fix some spatial weaknesses in smaller open models with cheap targeted fine-tuning instead of larger training runs. It also suggests that benchmark scores alone can overstate what open models can do.
Safety and control
The work is partly interpretability: it traces where in the model the failure happens. It also shows that high benchmark scores can hide real weaknesses, which matters for trusting evaluations.
The abstract, by the authors
Open-weight Vision-Language Models (VLMs) score 85-94% on spatial benchmarks (BLINK Spatial, What's Up CLEVR) yet collapse to chance on simple geometric questions when object, scene, and texture priors are stripped away. We introduce and open-source Sophon, a procedurally generated diagnostic of paired colored point-cloud stimuli across three coordinate systems (1D linear, 2D Cartesian, 2D polar) and four question types (binary, categorical, comparative, quantitative). Across four open architectures (LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Idefics3), accuracy clusters at 47-52% while frontier models (Opus 4.6, Gemini 2.5 Pro, GPT-5.4) clear 77-83% and humans reach 84-94%. We use linear probes and low-rank adaptation (LoRA) to trace the open-weight failures. Probes of the vision encoder (ViT) and language decoder (LLM) recover Sophon spatial information at 83-85% and 80-86% accuracy, respectively, and the language tower's reasoning circuit answers text-only controls (point clouds as sets of coordinates) at +40 pp over visual stimuli. Language tower per-layer attention attribution localizes visual integration to mid-stack layers. At readout, models elevate the correct answer-set tokens but select among them randomly. We localize the mechanism by tuning individual and grouped components across every architecture's stack, including the vision tower (ViT workspace and reader), the projector, and the language tower (LLM workspace and reader). These experiments demonstrate that this mid-stack workspace holds circuitry capable of organizing the spatial signal so that the frozen reader can decode it. LoRA fine-tuning adjusts that circuitry such that 7B-class VLMs answer Sophon spatial reasoning questions at frontier aggregate accuracy. This work suggests that these models do not have a capability gap for spatial reasoning but suffer from a routing bottleneck that targeted fine-tuning repairs.
Quoted from arXiv:2611.14838. The research and its words belong to its authors.
