Vision-Language Models Do Not Understand Negation
Journal:
arXiv
Published Date:
Jan 16, 2025
Abstract
Many practical vision-language applications require models that understand
negation, e.g., when using natural language to retrieve images which contain
certain objects but not others. Despite advancements in vision-language models
(VLMs) through large-scale training, their ability to comprehend negation
remains underexplored. This study addresses the question: how well do current
VLMs understand negation? We introduce NegBench, a new benchmark designed to
evaluate negation understanding across 18 task variations and $79$k examples
spanning image, video, and medical datasets. The benchmark consists of two core
tasks designed to evaluate negation understanding in diverse multimodal
settings: Retrieval with Negation and Multiple Choice Questions with Negated
Captions. Our evaluation reveals that modern VLMs struggle significantly with
negation, often performing at chance level. To address these shortcomings, we
explore a data-centric approach wherein we finetune CLIP models on large-scale
synthetic datasets containing millions of negated captions. We show that this
approach can result in a 10% increase in recall on negated queries and a 28%
boost in accuracy on multiple-choice questions with negated captions.
