Post-hoc Probabilistic Vision-Language Models
Journal:
arXiv
Published Date:
Dec 8, 2024
Abstract
Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable
success in classification, retrieval, and generative tasks. For this, VLMs
deterministically map images and text descriptions to a joint latent space in
which their similarity is assessed using the cosine similarity. However, a
deterministic mapping of inputs fails to capture uncertainties over concepts
arising from domain shifts when used in downstream tasks. In this work, we
propose post-hoc uncertainty estimation in VLMs that does not require
additional training. Our method leverages a Bayesian posterior approximation
over the last layers in VLMs and analytically quantifies uncertainties over
cosine similarities. We demonstrate its effectiveness for uncertainty
quantification and support set selection in active learning. Compared to
baselines, we obtain improved and well-calibrated predictive uncertainties,
interpretable uncertainty estimates, and sample-efficient active learning. Our
results show promise for safety-critical applications of large-scale models.
