Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs
Journal:
arXiv
Published Date:
May 26, 2025
Abstract
Large Vision-Language Models (LVLMs) are susceptible to hallucinations, where
generated responses seem semantically plausible yet exhibit little or no
relevance to the input image. Previous studies reveal that this issue primarily
stems from LVLMs' over-reliance on language priors while disregarding the
visual information during decoding. To alleviate this issue, we introduce a
novel Conditional Pointwise Mutual Information (C-PMI) calibrated decoding
strategy, which adaptively strengthens the mutual dependency between generated
texts and input images to mitigate hallucinations. Unlike existing methods
solely focusing on text token sampling, we propose to jointly model the
contributions of visual and textual tokens to C-PMI, formulating hallucination
mitigation as a bi-level optimization problem aimed at maximizing mutual
information. To solve it, we design a token purification mechanism that
dynamically regulates the decoding process by sampling text tokens remaining
maximally relevant to the given image, while simultaneously refining image
tokens most pertinent to the generated response. Extensive experiments across
various benchmarks reveal that the proposed method significantly reduces
hallucinations in LVLMs while preserving decoding efficiency.
