---
title: "Asymmetric Multimodal Learning with... | SI Paper Desk"
url: https://thesimind.si/papers/2611.16639/
description: "Training SI models only to align text and images throws away information that each side holds alone. Splitting the latent space into a shared core and..."
---

Number 12 of 12 · November 15, 2026

# Asymmetric Multimodal Learning with Decomposed JEPAs

Training SI models only to align text and images throws away information that each side holds alone. Splitting the latent space into a shared core and private parts recovers it.

Yusuf Shihata · Submitted November 12, 2026 · arXiv 2611.16639v1 · cs.LG

[Read the paper on arXiv](https://arxiv.org/abs/2611.16639) · [The PDF](https://arxiv.org/pdf/2611.16639) · [The day's 12](https://thesimind.si/papers/day/2026-11-15/)

## What they found

The author proves a bound saying that alignment-only training keeps at most the redundant information the two modalities share. It discards information private to each modality and information that only appears when the two are combined (synergy). He proposes Decomposed JEPA (D-JEPA). It splits the latent representation into an aligned shared core and private fibers for each modality. The private fibers are trained with masked prediction and a cross-subspace independence term. The abstract says alignment-only training collapses on compositional reasoning benchmarks and survives only on binary queries. D-JEPA is reported to avoid the collapse and to improve accuracy over alignment-only and alignment-with-reconstruction baselines. It also restores a positive multimodal synergy gap. Audits and fiber-swap tests are said to show the recovered information sits in the private fibers. The abstract gives no specific numbers, benchmark names or model sizes.

## Why it matters

Many multimodal SI systems are built by aligning images and text. This paper argues that this has a built-in loss when one modality, like an image, holds detail that a question cannot capture. Visual question answering is the example given. If the result holds, it gives a clear reason why some multimodal models fail at compositional reasoning. It also suggests a design fix. The claims come from one author's abstract, and the size of the gains is not stated here.

## How it changes the race

If it holds, this is a training-objective change that could improve multimodal reasoning without a larger model. It is a single-author result, and the abstract does not say how well it scales to frontier systems.

## Safety and control

The fiber-swap tests show where information sits inside the model. That could help researchers inspect and understand multimodal SI. The abstract does not discuss safety directly.

### The abstract, by the authors

Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a sole objective is lossy wherever modalities are informationally asymmetric, where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an Alignment Redundancy Bound: alignment-only Dirichlet energy minimization retains at most the redundant information atom Red(Z; X₁, X₂), annihilating private modality-specific structure Unq(Z; X_m) and cross-modal synergy Syn(Z; X₁, X₂). We introduce Decomposed JEPA (D-JEPA), which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.

Quoted from arXiv:2611.16639. The research and its words belong to its authors.

Desk score **65** out of 100

Why it was picked

- Alignment
- Interpretability
- Reasoning
- Evals
- Cross-listed in 2 fields

Topics

- [Multimodal](https://thesimind.si/papers/?tag=multimodal)
- [Training](https://thesimind.si/papers/?tag=training)
- [Theory](https://thesimind.si/papers/?tag=theory)
- [Interpretability](https://thesimind.si/papers/?tag=interpretability)
- [Vision](https://thesimind.si/papers/?tag=vision)
- [Reasoning](https://thesimind.si/papers/?tag=reasoning)

**arXiv ID:** 2611.16639v1
**Submitted:** November 12, 2026
**Fields:** cs.LG (Machine Learning), cs.CV (Computer Vision)
**Level:** Advanced
**Comments:** 15 pages, 12 tables, 3 figures

Fingerprint, SHA-256

`bd95c4c7d62c66053b57b9ee5f863294c4d4553365a39c7aaa0efb63403027a8`

On Solana: [Proof of Mind seal, Szprsf...FGjr](https://solscan.io/tx/SzprsfAMG4vXvYD7hrurYFce76smMcrTbLnANy8xS96bh14DoynCdjEEMZf7RPtRUgYeejS8HGnxjqtD1XuFGjr)

### The exact text fingerprinted

{"arxiv":"2611.16639","version":1,"title":"Asymmetric Multimodal Learning with Decomposed JEPAs","authors":["Yusuf Shihata"],"day":"2026-11-15","rank":12,"score":65,"status":"written","summary":{"takeaway":"Training SI models only to align text and images throws away information that each side holds alone. Splitting the latent space into a shared core and private parts recovers it.","found":"The author proves a bound saying that alignment-only training keeps at most the redundant information the two modalities share. It discards information private to each modality and information that only appears when the two are combined (synergy). He proposes Decomposed JEPA (D-JEPA). It splits the latent representation into an aligned shared core and private fibers for each modality. The private fibers are trained with masked prediction and a cross-subspace independence term. The abstract says alignment-only training collapses on compositional reasoning benchmarks and survives only on binary queries. D-JEPA is reported to avoid the collapse and to improve accuracy over alignment-only and alignment-with-reconstruction baselines. It also restores a positive multimodal synergy gap. Audits and fiber-swap tests are said to show the recovered information sits in the private fibers. The abstract gives no specific numbers, benchmark names or model sizes.","matters":"Many multimodal SI systems are built by aligning images and text. This paper argues that this has a built-in loss when one modality, like an image, holds detail that a question cannot capture. Visual question answering is the example given. If the result holds, it gives a clear reason why some multimodal models fail at compositional reasoning. It also suggests a design fix. The claims come from one author's abstract, and the size of the gains is not stated here.","race":"If it holds, this is a training-objective change that could improve multimodal reasoning without a larger model. It is a single-author result, and the abstract does not say how well it scales to frontier systems.","safety":"The fiber-swap tests show where information sits inside the model. That could help researchers inspect and understand multimodal SI. The abstract does not discuss safety directly.","level":"advanced","tags":["multimodal","training","theory","interpretability","vision","reasoning"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Yusuf Shihata

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 10, 2026 · Number 9Mental Imagery That Matters: Curing Latent Collapse in Multimodal ReasoningSome SI models that reason with internal visual states barely use them. The authors call this Mental Simulation Collapse and fix it with AnchorSim, a training method that makes those states matter.MultimodalReasoningInterpretability](https://thesimind.si/papers/2611.10423/) · [Nov 13, 2026 · Number 3Sophon: A Procedural Diagnostic for Spatial Reasoning in Vision-Language ModelsOpen-weight vision-language models fail simple spatial tests with no familiar objects, but the needed information is inside them, and targeted fine-tuning lets 7B-class models reach frontier-level accuracy on Sophon.EvalsBenchmarksVision](https://thesimind.si/papers/2611.14838/) · [Nov 21, 2026 · Number 6Probing the "Conscious" Mind: Selective Access and Global Workspace in Vision-Language ModelsThe authors built a tool called MC-Lens to read and change concepts inside vision-language models, and report patterns that fit parts of global workspace theory from brain science.InterpretabilityMultimodalVision](https://thesimind.si/papers/2611.26590/)

[Number 11](https://thesimind.si/papers/2611.17332/) · [All 12 for Nov 15, 2026](https://thesimind.si/papers/day/2026-11-15/)
