Mitigating Image Captioning Hallucinations in Vision-Language Models
Journal:
arXiv
Published Date:
May 6, 2025
Abstract
Hallucinations in vision-language models (VLMs) hinder reliability and
real-world applicability, usually stemming from distribution shifts between
pretraining data and test samples. Existing solutions, such as retraining or
fine-tuning on additional data, demand significant computational resources and
labor-intensive data collection, while ensemble-based methods incur additional
costs by introducing auxiliary VLMs. To address these challenges, we propose a
novel test-time adaptation framework using reinforcement learning to mitigate
hallucinations during inference without retraining or any auxiliary VLMs. By
updating only the learnable parameters in the layer normalization of the
language model (approximately 0.003% of the model parameters), our method
reduces distribution shifts between test samples and pretraining samples. A
CLIP-based hallucination evaluation model is proposed to provide dual rewards
to VLMs. Experimental results demonstrate a 15.4% and 17.3% reduction in
hallucination rates on LLaVA and InstructBLIP, respectively. Our approach
outperforms state-of-the-art baselines with a 68.3% improvement in
hallucination mitigation, demonstrating its effectiveness.
