SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models
Journal:
arXiv
Published Date:
Feb 24, 2025
Abstract
Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs)
faces significant challenges without training data, model tuning, or
architectural modifications. Existing approaches require prompt tuning or
architectural adaptations, limiting zero-shot applicability. Our work proposes
a novel solution treating VLMs as black boxes, leveraging scores without
training data or ground truth. Using large language model insights on object
co-occurrence, we introduce compound prompts grounded in realistic object
combinations. Analysis of these prompt scores reveals VLM biases and
``AND''/``OR'' signal ambiguities, notably that maximum compound scores are
surprisingly suboptimal compared to second-highest scores. We address these
through a debiasing and score-fusion algorithm that corrects image bias and
clarifies VLM response behaviors. Our method enhances other zero-shot
approaches, consistently improving their results. Experiments show superior
mean Average Precision (mAP) compared to methods requiring training data,
achieved through refined object ranking for robust zero-shot MLR.
