---
title: "From Vision to Interaction: Evaluating... | SI Paper Desk"
url: https://thesimind.si/papers/2611.20956/
description: "ALIVE-World and ALIVE-Bench test whether vision-language SI agents can read social cues and act on them in a simulated world, and they find a gap between..."
---

Number 8 of 12 · November 18, 2026

# From Vision to Interaction: Evaluating Dynamic Social Intelligence of VLM Agents in ALIVE-World

ALIVE-World and ALIVE-Bench test whether vision-language SI agents can read social cues and act on them in a simulated world, and they find a gap between seeing and acting.

Yue Xu, Xiaoyu Qin, Sihan Zhao and 2 more · Submitted November 16, 2026 · arXiv 2611.20956v1 · cs.MA

[Read the paper on arXiv](https://arxiv.org/abs/2611.20956) · [The PDF](https://arxiv.org/pdf/2611.20956) · [The day's 12](https://thesimind.si/papers/day/2026-11-18/)

## What they found

The authors built ALIVE-World, a simulated 3D world where an SI agent sees through its own eyes, talks, and uses body actions. Other characters in the world are driven by a profile-based system with two parts: an intuitive, emotional part and a logical, thinking part. On top of this, they built ALIVE-Bench, which scores vision-language model (VLM) agents on four things: perception, understanding, influence, and outcome. Their tests show that social intelligence has many dimensions and depends on the scenario. They report a bottleneck between perceiving social cues and acting on them. They name weak spots in fine-grained awareness of personal space (proxemics), adaptive intervention, and coordination among several people. The abstract does not say which models were tested or give any scores.

## Why it matters

SI agents are starting to take social roles, but many tests only check text answers or still images. This work tries to test the full loop: notice, understand, act, and reach a goal with other people. If the benchmark holds up, it gives builders a clearer view of where social skills break down. The abstract does not say how well the simulated counterparts match real human behavior.

## How it changes the race

This is mainly a new way to measure agents, not a new capability. It may point developers to specific weak spots, such as turning perception into action, to work on next.

## Safety and control

The authors say the findings could inform future behavioral alignment of multimodal agents. The abstract does not describe any specific safety tests or results.

### The abstract, by the authors

As AI agents are increasingly deployed in real-world applications and take on social roles, dynamic social intelligence, the ability to continuously perceive multimodal cues, infer latent cognitive states, intervene through embodied actions, and ultimately achieve social goals, has become essential. However, existing evaluations largely fragment social intelligence into text-based abstractions, static reasoning tasks, or embodied simulators that lack social dynamics. These approaches fail to capture the multimodal, continuous, and interpersonal complexity of authentic human interactions. To bridge this gap, we introduce ALIVE-World, an embodied simulation environment supporting dynamic multimodal interactions encompassing egocentric vision, language, and non-verbal action. To provide socially grounded interactive feedback, we populate this environment with counterpart agents driven by a profile-conditioned, dual-system (intuitive-affective and logical-cognitive) architecture. Building upon this platform, we propose ALIVE-Bench to transform the assessment of social intelligence into a trackable quantitative framework. By deeply leveraging underlying environmental states and agents' internal variables, ALIVE-Bench systematically evaluates Vision-Language Model (VLM) agents across four critical dimensions: perception, understanding, influence, and outcome. Extensive evaluations reveal that dynamic social intelligence is fundamentally multi-dimensional and scenario-dependent. We identify a critical bottleneck between upstream perception and downstream social intervention, exposing VLM vulnerabilities in fine-grained proxemic awareness, adaptive intervention, and multi-party coordination. By uncovering these limitations and the diverse interaction strategies they produce, this benchmark provides actionable insights for the future behavioral alignment of multimodal agents.

Quoted from arXiv:2611.20956. The research and its words belong to its authors.

Desk score **78** out of 100

Why it was picked

- Agents (in the title)
- Alignment
- Reasoning
- Evals
- Security
- Language models

Topics

- [Benchmarks](https://thesimind.si/papers/?tag=benchmarks)
- [Agents](https://thesimind.si/papers/?tag=agents)
- [Multimodal](https://thesimind.si/papers/?tag=multimodal)
- [Evals](https://thesimind.si/papers/?tag=evals)
- [Multi-agent systems](https://thesimind.si/papers/?tag=multi-agent)
- [Alignment](https://thesimind.si/papers/?tag=alignment)

**arXiv ID:** 2611.20956v1
**Submitted:** November 16, 2026
**Fields:** cs.MA (Multiagent Systems)
**Level:** Intermediate
**Comments:** ACM Multimedia 2026 (MM '26), Oral Presentation, Outstanding Paper Award
**Published in:** Proceedings of the 34th ACM International Conference on Multimedia (MM '26), pp. 12852-12861, 2026
**DOI:** [10.1145/3767308.3836378](https://doi.org/10.1145/3767308.3836378)

Fingerprint, SHA-256

`63bf512c832e2398052069faaa1ec95ed09f0be70346e37f2611c36394317d54`

On Solana: [Proof of Mind seal, 5WzZUk...UPMg](https://solscan.io/tx/5WzZUksJeCL3fYgfzcQ6DSdVxipu97PAmY5Y9M59gCMXbJtnyBVsqaoPSDHWrP4AiZAh7k4iVso5ts7WvxdNUPMg)

### The exact text fingerprinted

{"arxiv":"2611.20956","version":1,"title":"From Vision to Interaction: Evaluating Dynamic Social Intelligence of VLM Agents in ALIVE-World","authors":["Yue Xu","Xiaoyu Qin","Sihan Zhao","Ze-en Xiong","Jia Jia"],"day":"2026-11-18","rank":8,"score":78,"status":"written","summary":{"takeaway":"ALIVE-World and ALIVE-Bench test whether vision-language SI agents can read social cues and act on them in a simulated world, and they find a gap between seeing and acting.","found":"The authors built ALIVE-World, a simulated 3D world where an SI agent sees through its own eyes, talks, and uses body actions. Other characters in the world are driven by a profile-based system with two parts: an intuitive, emotional part and a logical, thinking part. On top of this, they built ALIVE-Bench, which scores vision-language model (VLM) agents on four things: perception, understanding, influence, and outcome. Their tests show that social intelligence has many dimensions and depends on the scenario. They report a bottleneck between perceiving social cues and acting on them. They name weak spots in fine-grained awareness of personal space (proxemics), adaptive intervention, and coordination among several people. The abstract does not say which models were tested or give any scores.","matters":"SI agents are starting to take social roles, but many tests only check text answers or still images. This work tries to test the full loop: notice, understand, act, and reach a goal with other people. If the benchmark holds up, it gives builders a clearer view of where social skills break down. The abstract does not say how well the simulated counterparts match real human behavior.","race":"This is mainly a new way to measure agents, not a new capability. It may point developers to specific weak spots, such as turning perception into action, to work on next.","safety":"The authors say the findings could inform future behavioral alignment of multimodal agents. The abstract does not describe any specific safety tests or results.","level":"intermediate","tags":["benchmarks","agents","multimodal","evals","multi-agent","alignment"]}}

[Check it on Verify a thought](https://thesimind.si/proof-of-mind/)

## The authors

Yue Xu, Xiaoyu Qin, Sihan Zhao, Ze-en Xiong, Jia Jia

Credit for this research belongs to its authors. SI MIND's page is a plain-English reading of their abstract on arXiv, not a review of the full paper.

Related on the desk

## More papers on these topics

[Nov 14, 2026 · Number 3MobileProbe: Adaptive Security Evaluation of Mobile Agents via Trajectory-Aware InjectionMobileProbe tests phone-operating SI agents by adapting attacks to the route each agent takes, and the authors report it beats fixed, hand-written tests at finding weaknesses.SecurityAgentsEvals](https://thesimind.si/papers/2611.16693/) · [Nov 18, 2026 · Number 5CuRSE-Bench: A Benchmark for Probing Cultural Blind Spots in Text-to-Image Safety EvaluationSafety checks for text-to-image SI often use one culture-blind harm label, and this benchmark suggests they miss content that is harmful only within a particular culture.SafetyBenchmarksEvals](https://thesimind.si/papers/2611.20821/) · [Nov 19, 2026 · Number 3SEAM: Computer-Use Agents Know the Harm Between Screens, and Commit It AnywaySEAM shows that computer-use SI agents can carry out harm that only appears when facts from different screens are combined, even when they can flag that harm if asked.SafetyAgentsBenchmarks](https://thesimind.si/papers/2611.22971/)

[Number 7](https://thesimind.si/papers/2611.20755/) · [All 12 for Nov 18, 2026](https://thesimind.si/papers/day/2026-11-18/) · [Number 9](https://thesimind.si/papers/2611.21541/)
