V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models
Journal:
arXiv
Published Date:
Apr 8, 2025
Abstract
Recent advancements in Multimodal Large Language Models (MLLMs) have
demonstrated impressive capabilities in visual-text processing. However,
existing static image-text benchmarks are insufficient for evaluating their
dynamic perception and interactive reasoning abilities. We introduce
Vision-centric Multiple Abilities Game Evaluation(V-MAGE), a novel game-based
evaluation framework designed to systematically assess MLLMs' visual reasoning
in interactive, continuous-space environments. V-MAGE features five distinct
video games comprising over 30 carefully constructed evaluation scenarios.
These scenarios are set in free-form, visually complex environments that
require models to interpret dynamic game states and make decisions based solely
on visual input, thereby closely reflecting the conditions encountered by human
players. To ensure robust and interpretable comparisons across models, V-MAGE
employs a dynamic Elo-based ranking system that accounts for varying difficulty
levels and task diversity. Benchmarking state-of-the-art MLLMs against human
baselines reveals that while leading models approach human-level performance in
simple tasks, their performance drops significantly in complex scenarios
requiring advanced reasoning and task orchestration. This persistent
performance gap highlights fundamental limitations in current MLLMs' ability to
perform real-time, vision-grounded interactions. Through extensive analyses, we
demonstrate the utility of V-MAGE in uncovering these limitations and providing
actionable insights for improving the visual and reasoning capabilities of
MLLMs in dynamic, interactive settings. Code is publicly available at
https://github.com/CSU-JPG/V-MAGE.
