Enhancing Multimodal Entity Linking with Jaccard Distance-based Conditional Contrastive Learning and Contextual Visual Augmentation
Journal:
arXiv
Published Date:
Jan 24, 2025
Abstract
Previous research on multimodal entity linking (MEL) has primarily employed
contrastive learning as the primary objective. However, using the rest of the
batch as negative samples without careful consideration, these studies risk
leveraging easy features and potentially overlook essential details that make
entities unique. In this work, we propose JD-CCL (Jaccard Distance-based
Conditional Contrastive Learning), a novel approach designed to enhance the
ability to match multimodal entity linking models. JD-CCL leverages
meta-information to select negative samples with similar attributes, making the
linking task more challenging and robust. Additionally, to address the
limitations caused by the variations within the visual modality among mentions
and entities, we introduce a novel method, CVaCPT (Contextual Visual-aid
Controllable Patch Transform). It enhances visual representations by
incorporating multi-view synthetic images and contextual textual
representations to scale and shift patch representations. Experimental results
on benchmark MEL datasets demonstrate the strong effectiveness of our approach.
