Evaluating Visual Mathematics in Multimodal LLMs: A Multilingual Benchmark Based on the Kangaroo Tests
Journal:
arXiv
Published Date:
Jun 9, 2025
Abstract
Multimodal Large Language Models (MLLMs) promise advanced vision language
capabilities, yet their effectiveness in visually presented mathematics remains
underexplored. This paper analyzes the development and evaluation of MLLMs for
mathematical problem solving, focusing on diagrams, multilingual text, and
symbolic notation. We then assess several models, including GPT 4o, Pixtral,
Qwen VL, Llama 3.2 Vision variants, and Gemini 2.0 Flash in a multilingual
Kangaroo style benchmark spanning English, French, Spanish, and Catalan. Our
experiments reveal four key findings. First, overall precision remains moderate
across geometry, visual algebra, logic, patterns, and combinatorics: no single
model excels in every topic. Second, while most models see improved accuracy
with questions that do not have images, the gain is often limited; performance
for some remains nearly unchanged without visual input, indicating
underutilization of diagrammatic information. Third, substantial variation
exists across languages and difficulty levels: models frequently handle easier
items but struggle with advanced geometry and combinatorial reasoning. Notably,
Gemini 2.0 Flash achieves the highest precision on image based tasks, followed
by Qwen VL 2.5 72B and GPT 4o, though none approach human level performance.
Fourth, a complementary analysis aimed at distinguishing whether models reason
or simply recite reveals that Gemini and GPT 4o stand out for their structured
reasoning and consistent accuracy. In contrast, Pixtral and Llama exhibit less
consistent reasoning, often defaulting to heuristics or randomness when unable
to align their outputs with the given answer options.
