VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?
Journal:
arXiv
Published Date:
May 29, 2025
Abstract
Recent studies have shown that long chain-of-thought (CoT) reasoning can
significantly enhance the performance of large language models (LLMs) on
complex tasks. However, this benefit is yet to be demonstrated in the domain of
video understanding, since most existing benchmarks lack the reasoning depth
required to demonstrate the advantages of extended CoT chains. While recent
efforts have proposed benchmarks aimed at video reasoning, the tasks are often
knowledge-driven and do not rely heavily on visual content. To bridge this gap,
we introduce VideoReasonBench, a benchmark designed to evaluate vision-centric,
complex video reasoning. To ensure visual richness and high reasoning
complexity, each video in VideoReasonBench depicts a sequence of fine-grained
operations on a latent state that is only visible in part of the video. The
questions evaluate three escalating levels of video reasoning skills: recalling
observed visual information, inferring the content of latent states, and
predicting information beyond the video. Under such task setting, models have
to precisely recall multiple operations in the video, and perform step-by-step
reasoning to get correct final answers for these questions. Using
VideoReasonBench, we comprehensively evaluate 18 state-of-the-art multimodal
LLMs (MLLMs), finding that most perform poorly on complex video reasoning,
e.g., GPT-4o achieves only 6.9% accuracy, while the thinking-enhanced
Gemini-2.5-Pro significantly outperforms others with 56.0% accuracy. Our
investigations on "test-time scaling" further reveal that extended thinking
budget, while offering none or minimal benefits on existing video benchmarks,
is essential for improving the performance on VideoReasonBench.
