Dynamic Multimodal Prototype Learning in Vision-Language Models
Journal:
arXiv
Published Date:
Jul 4, 2025
Abstract
With the increasing attention to pre-trained vision-language models (VLMs),
\eg, CLIP, substantial efforts have been devoted to many downstream tasks,
especially in test-time adaptation (TTA). However, previous works focus on
learning prototypes only in the textual modality while overlooking the
ambiguous semantics in class names. These ambiguities lead to textual
prototypes that are insufficient to capture visual concepts, resulting in
limited performance. To address this issue, we introduce \textbf{ProtoMM}, a
training-free framework that constructs multimodal prototypes to adapt VLMs
during the test time. By viewing the prototype as a discrete distribution over
the textual descriptions and visual particles, ProtoMM has the ability to
combine the multimodal features for comprehensive prototype learning. More
importantly, the visual particles are dynamically updated as the testing stream
flows. This allows our multimodal prototypes to continually learn from the
data, enhancing their generalizability in unseen scenarios. In addition, we
quantify the importance of the prototypes and test images by formulating their
semantic distance as an optimal transport problem. Extensive experiments on 15
zero-shot benchmarks demonstrate the effectiveness of our method, achieving a
1.03\% average accuracy improvement over state-of-the-art methods on ImageNet
and its variant datasets.
