R^3-VQA: "Read the Room" by Video Social Reasoning
Journal:
arXiv
Published Date:
May 7, 2025
Abstract
"Read the room" is a significant social reasoning capability in human daily
life. Humans can infer others' mental states from subtle social cues. Previous
social reasoning tasks and datasets lack complexity (e.g., simple scenes, basic
interactions, incomplete mental state variables, single-step reasoning, etc.)
and fall far short of the challenges present in real-life social interactions.
In this paper, we contribute a valuable, high-quality, and comprehensive video
dataset named R^3-VQA with precise and fine-grained annotations of social
events and mental states (i.e., belief, intent, desire, and emotion) as well as
corresponding social causal chains in complex social scenarios. Moreover, we
include human-annotated and model-generated QAs. Our task R^3-VQA includes
three aspects: Social Event Understanding, Mental State Estimation, and Social
Causal Reasoning. As a benchmark, we comprehensively evaluate the social
reasoning capabilities and consistencies of current state-of-the-art large
vision-language models (LVLMs). Comprehensive experiments show that (i) LVLMs
are still far from human-level consistent social reasoning in complex social
scenarios; (ii) Theory of Mind (ToM) prompting can help LVLMs perform better on
social reasoning tasks. We provide some of our dataset and codes in
supplementary material and will release our full dataset and codes upon
acceptance.
