MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoning
Journal:
arXiv
Published Date:
Jun 5, 2025
Abstract
Despite rapid advances in vision-language models (VLMs), current benchmarks
for multimodal reasoning fall short in three key dimensions. First, they
overwhelmingly rely on static images, failing to capture the temporal
complexity of real-world environments. Second, they narrowly focus on
mathematical problem-solving, neglecting the broader spectrum of reasoning
skills -- including abstract, physical, planning, spatial, and temporal
capabilities -- required for robust multimodal intelligence. Third, many
benchmarks quickly saturate, offering limited headroom for diagnosing failure
modes or measuring continued progress. We introduce MORSE-500 (Multimodal
Reasoning Stress-test Environment), a video benchmark composed of 500 fully
scripted clips with embedded questions spanning six complementary reasoning
categories. Each instance is programmatically generated using deterministic
Python scripts (via Manim, Matplotlib, MoviePy), generative video models, and
curated real footage. This script-driven design allows fine-grained control
over visual complexity, distractor density, and temporal dynamics -- enabling
difficulty to be scaled systematically as models improve. Unlike static
benchmarks that become obsolete once saturated, MORSE-500 is built to evolve:
its controllable generation pipeline supports the creation of arbitrarily
challenging new instances, making it ideally suited for stress-testing
next-generation models. Initial experiments with state-of-the-art systems --
including various Gemini 2.5 Pro and OpenAI o3 which represent the strongest
available at the time, alongside strong open-source models -- reveal
substantial performance gaps across all categories, with particularly large
deficits in abstract and planning tasks. We release the full dataset,
generation scripts, and evaluation harness to support transparent,
reproducible, and forward-looking multimodal reasoning research.
