Enhancing Zero-Shot Image Recognition in Vision-Language Models through Human-like Concept Guidance
Journal:
arXiv
Published Date:
Mar 20, 2025
Abstract
In zero-shot image recognition tasks, humans demonstrate remarkable
flexibility in classifying unseen categories by composing known simpler
concepts. However, existing vision-language models (VLMs), despite achieving
significant progress through large-scale natural language supervision, often
underperform in real-world applications because of sub-optimal prompt
engineering and the inability to adapt effectively to target classes. To
address these issues, we propose a Concept-guided Human-like Bayesian Reasoning
(CHBR) framework. Grounded in Bayes' theorem, CHBR models the concept used in
human image recognition as latent variables and formulates this task by summing
across potential concepts, weighted by a prior distribution and a likelihood
function. To tackle the intractable computation over an infinite concept space,
we introduce an importance sampling algorithm that iteratively prompts large
language models (LLMs) to generate discriminative concepts, emphasizing
inter-class differences. We further propose three heuristic approaches
involving Average Likelihood, Confidence Likelihood, and Test Time Augmentation
(TTA) Likelihood, which dynamically refine the combination of concepts based on
the test image. Extensive evaluations across fifteen datasets demonstrate that
CHBR consistently outperforms existing state-of-the-art zero-shot
generalization methods.
