Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP
Journal:
arXiv
Published Date:
Dec 27, 2024
Abstract
The application of Contrastive Language-Image Pre-training (CLIP) in Weakly
Supervised Semantic Segmentation (WSSS) research powerful cross-modal semantic
understanding capabilities. Existing methods attempt to optimize input text
prompts for improved alignment of images and text, by finely adjusting text
prototypes to facilitate semantic matching. Nevertheless, given the modality
gap between text and vision spaces, the text prototypes employed by these
methods have not effectively established a close correspondence with
pixel-level vision features. In this work, our theoretical analysis indicates
that the inherent modality gap results in misalignment of text and region
features, and that this gap cannot be sufficiently reduced by minimizing
contrast loss in CLIP. To mitigate the impact of the modality gap, we propose a
Vision Prototype Learning (VPL) framework, by introducing more representative
vision prototypes. The core of this framework is to learn class-specific vision
prototypes in vision space with the help of text prototypes, for capturing
high-quality localization maps. Moreover, we propose a regional semantic
contrast module that contrasts regions embedding with corresponding prototypes,
leading to more comprehensive and robust feature learning. Experimental results
show that our proposed framework achieves state-of-the-art performance on two
benchmark datasets.
