Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation
Journal:
arXiv
Published Date:
Mar 26, 2025
Abstract
Weakly Supervised Semantic Segmentation (WSSS) with image-level labels aims
to achieve pixel-level predictions using Class Activation Maps (CAMs).
Recently, Contrastive Language-Image Pre-training (CLIP) has been introduced in
WSSS. However, recent methods primarily focus on image-text alignment for CAM
generation, while CLIP's potential in patch-text alignment remains unexplored.
In this work, we propose ExCEL to explore CLIP's dense knowledge via a novel
patch-text alignment paradigm for WSSS. Specifically, we propose Text Semantic
Enrichment (TSE) and Visual Calibration (VC) modules to improve the dense
alignment across both text and vision modalities. To make text embeddings
semantically informative, our TSE module applies Large Language Models (LLMs)
to build a dataset-wide knowledge base and enriches the text representations
with an implicit attribute-hunting process. To mine fine-grained knowledge from
visual features, our VC module first proposes Static Visual Calibration (SVC)
to propagate fine-grained knowledge in a non-parametric manner. Then Learnable
Visual Calibration (LVC) is further proposed to dynamically shift the frozen
features towards distributions with diverse semantics. With these enhancements,
ExCEL not only retains CLIP's training-free advantages but also significantly
outperforms other state-of-the-art methods with much less training cost on
PASCAL VOC and MS COCO.
