INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Sampling
Journal:
arXiv
Published Date:
Jul 7, 2025
Abstract
Hallucinations in large vision-language models (LVLMs) pose significant
challenges for real-world applications, as LVLMs may generate responses that
appear plausible yet remain inconsistent with the associated visual content.
This issue rarely occurs in human cognition. We argue that this discrepancy
arises from humans' ability to effectively leverage multimodal interaction
information in data samples. Specifically, humans typically first gather
multimodal information, analyze the interactions across modalities for
understanding, and then express their understanding through language. Motivated
by this observation, we conduct extensive experiments on popular LVLMs and
obtained insights that surprisingly reveal human-like, though less pronounced,
cognitive behavior of LVLMs on multimodal samples. Building on these findings,
we further propose \textbf{INTER}: \textbf{Inter}action Guidance Sampling, a
novel training-free algorithm that mitigate hallucinations without requiring
additional data. Specifically, INTER explicitly guides LVLMs to effectively
reapply their understanding of multimodal interaction information when
generating responses, thereby reducing potential hallucinations. On six
benchmarks including VQA and image captioning tasks, INTER achieves an average
improvement of up to 3.4\% on five LVLMs compared to the state-of-the-art
decoding strategy. The code will be released when the paper is accepted.
