Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Journal:
arXiv
Published Date:
Jun 3, 2025
Abstract
Vision-language models, such as CLIP, have achieved significant success in
aligning visual and textual representations, becoming essential components of
many multi-modal large language models (MLLMs) like LLaVA and OpenFlamingo.
However, numerous studies have identified CLIP's limited fine-grained
perception as a critical drawback, leading to substantial failures in
downstream MLLMs. In contrast, vision-centric foundation models like DINOv2
demonstrate remarkable capabilities in capturing fine details from images. In
this work, we propose a novel kernel-based method to align CLIP's visual
representation with that of DINOv2, ensuring that the resulting embeddings
maintain compatibility with text embeddings while enhancing perceptual
capabilities. Our alignment objective is designed for efficient stochastic
optimization. Following this image-only alignment fine-tuning, the visual
encoder retains compatibility with the frozen text encoder and exhibits
significant improvements in zero-shot object recognition, fine-grained spatial
reasoning, and localization. By integrating the aligned visual encoder,
downstream MLLMs also demonstrate enhanced performance.
