MAVias: Mitigate any Visual Bias
Journal:
arXiv
Published Date:
Dec 9, 2024
Abstract
Mitigating biases in computer vision models is an essential step towards the
trustworthiness of artificial intelligence models. Existing bias mitigation
methods focus on a small set of predefined biases, limiting their applicability
in visual datasets where multiple, possibly unknown biases exist. To address
this limitation, we introduce MAVias, an open-set bias mitigation approach
leveraging foundation models to discover spurious associations between visual
attributes and target classes. MAVias first captures a wide variety of visual
features in natural language via a foundation image tagging model, and then
leverages a large language model to select those visual features defining the
target class, resulting in a set of language-coded potential visual biases. We
then translate this set of potential biases into vision-language embeddings and
introduce an in-processing bias mitigation approach to prevent the model from
encoding information related to them. Our experiments on diverse datasets,
including CelebA, Waterbirds, ImageNet, and UrbanCars, show that MAVias
effectively detects and mitigates a wide range of biases in visual recognition
tasks outperforming current state-of-the-art.
