MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings
Journal:
arXiv
Published Date:
Jun 29, 2025
Abstract
Multimodal embedding models, built upon causal Vision Language Models (VLMs),
have shown promise in various tasks. However, current approaches face three key
limitations: the use of causal attention in VLM backbones is suboptimal for
embedding tasks; scalability issues due to reliance on high-quality labeled
paired data for contrastive learning; and limited diversity in training
objectives and data. To address these issues, we propose MoCa, a two-stage
framework for transforming pre-trained VLMs into effective bidirectional
multimodal embedding models. The first stage, Modality-aware Continual
Pre-training, introduces a joint reconstruction objective that simultaneously
denoises interleaved text and image inputs, enhancing bidirectional
context-aware reasoning. The second stage, Heterogeneous Contrastive
Fine-tuning, leverages diverse, semantically rich multimodal data beyond simple
image-caption pairs to enhance generalization and alignment. Our method
addresses the stated limitations by introducing bidirectional attention through
continual pre-training, scaling effectively with massive unlabeled datasets via
joint reconstruction objectives, and utilizing diverse multimodal data for
enhanced representation robustness. Experiments demonstrate that MoCa
consistently improves performance across MMEB and ViDoRe-v2 benchmarks,
achieving new state-of-the-art results, and exhibits strong scalability with
both model size and training data on MMEB.
