MLLMs are Deeply Affected by Modality Bias
Journal:
arXiv
Published Date:
May 24, 2025
Abstract
Recent advances in Multimodal Large Language Models (MLLMs) have shown
promising results in integrating diverse modalities such as texts and images.
MLLMs are heavily influenced by modality bias, often relying on language while
under-utilizing other modalities like visual inputs. This position paper argues
that MLLMs are deeply affected by modality bias. Firstly, we diagnose the
current state of modality bias, highlighting its manifestations across various
tasks. Secondly, we propose a systematic research road-map related to modality
bias in MLLMs. Thirdly, we identify key factors of modality bias in MLLMs and
offer actionable suggestions for future research to mitigate it. To
substantiate these findings, we conduct experiments that demonstrate the
influence of each factor: 1. Data Characteristics: Language data is compact and
abstract, while visual data is redundant and complex, creating an inherent
imbalance in learning dynamics. 2. Imbalanced Backbone Capabilities: The
dominance of pretrained language models in MLLMs leads to overreliance on
language and neglect of visual information. 3. Training Objectives: Current
objectives often fail to promote balanced cross-modal alignment, resulting in
shortcut learning biased toward language. These findings highlight the need for
balanced training strategies and model architectures to better integrate
multiple modalities in MLLMs. We call for interdisciplinary efforts to tackle
these challenges and drive innovation in MLLM research. Our work provides a
fresh perspective on modality bias in MLLMs and offers insights for developing
more robust and generalizable multimodal systems-advancing progress toward
Artificial General Intelligence.
