Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
Journal:
arXiv
Published Date:
Apr 28, 2025
Abstract
Multi-Modal Entity Alignment (MMEA) aims to retrieve equivalent entities from
different Multi-Modal Knowledge Graphs (MMKGs), a critical information
retrieval task. Existing studies have explored various fusion paradigms and
consistency constraints to improve the alignment of equivalent entities, while
overlooking that the visual modality may not always contribute positively.
Empirically, entities with low-similarity images usually generate
unsatisfactory performance, highlighting the limitation of overly relying on
visual features. We believe the model can be biased toward the visual modality,
leading to a shortcut image-matching task. To address this, we propose a
counterfactual debiasing framework for MMEA, termed CDMEA, which investigates
visual modality bias from a causal perspective. Our approach aims to leverage
both visual and graph modalities to enhance MMEA while suppressing the direct
causal effect of the visual modality on model predictions. By estimating the
Total Effect (TE) of both modalities and excluding the Natural Direct Effect
(NDE) of the visual modality, we ensure that the model predicts based on the
Total Indirect Effect (TIE), effectively utilizing both modalities and reducing
visual modality bias. Extensive experiments on 9 benchmark datasets show that
CDMEA outperforms 14 state-of-the-art methods, especially in low-similarity,
high-noise, and low-resource data scenarios.
