Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT
Journal:
arXiv
Published Date:
May 30, 2025
Abstract
Understanding the physical world - governed by laws of motion, spatial
relations, and causality - poses a fundamental challenge for multimodal large
language models (MLLMs). While recent advances such as OpenAI o3 and GPT-4o
demonstrate impressive perceptual and reasoning capabilities, our investigation
reveals these models struggle profoundly with visual physical reasoning,
failing to grasp basic physical laws, spatial interactions, and causal effects
in complex scenes. More importantly, they often fail to follow coherent
reasoning chains grounded in visual evidence, especially when multiple steps
are needed to arrive at the correct answer. To rigorously evaluate this
capability, we introduce MVPBench, a curated benchmark designed to rigorously
evaluate visual physical reasoning through the lens of visual chain-of-thought
(CoT). Each example features interleaved multi-image inputs and demands not
only the correct final answer but also a coherent, step-by-step reasoning path
grounded in evolving visual cues. This setup mirrors how humans reason through
real-world physical processes over time. To ensure fine-grained evaluation, we
introduce a graph-based CoT consistency metric that verifies whether the
reasoning path of model adheres to valid physical logic. Additionally, we
minimize shortcut exploitation from text priors, encouraging models to rely on
visual understanding. Experimental results reveal a concerning trend: even
cutting-edge MLLMs exhibit poor visual reasoning accuracy and weak image-text
alignment in physical domains. Surprisingly, RL-based post-training alignment -
commonly believed to improve visual reasoning performance - often harms spatial
reasoning, suggesting a need to rethink current fine-tuning practices.
