ViLU: Learning Vision-Language Uncertainties for Failure Prediction
Journal:
arXiv
Published Date:
Jul 10, 2025
Abstract
Reliable Uncertainty Quantification (UQ) and failure prediction remain open
challenges for Vision-Language Models (VLMs). We introduce ViLU, a new
Vision-Language Uncertainty quantification framework that contextualizes
uncertainty estimates by leveraging all task-relevant textual representations.
ViLU constructs an uncertainty-aware multi-modal representation by integrating
the visual embedding, the predicted textual embedding, and an image-conditioned
textual representation via cross-attention. Unlike traditional UQ methods based
on loss prediction, ViLU trains an uncertainty predictor as a binary classifier
to distinguish correct from incorrect predictions using a weighted binary
cross-entropy loss, making it loss-agnostic. In particular, our proposed
approach is well-suited for post-hoc settings, where only vision and text
embeddings are available without direct access to the model itself. Extensive
experiments on diverse datasets show the significant gains of our method
compared to state-of-the-art failure prediction methods. We apply our method to
standard classification datasets, such as ImageNet-1k, as well as large-scale
image-caption datasets like CC12M and LAION-400M. Ablation studies highlight
the critical role of our architecture and training in achieving effective
uncertainty quantification. Our code is publicly available and can be found
here: https://github.com/ykrmm/ViLU.
