Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
Journal:
arXiv
Published Date:
May 17, 2025
Abstract
Large Vision-Language Models (LVLMs) have exhibited impressive capabilities
across various visual tasks, yet they remain hindered by the persistent
challenge of hallucinations. To address this critical issue, we propose Mixture
of Decoding (MoD), a novel approach for hallucination mitigation that
dynamically adapts decoding strategies by evaluating the correctness of the
model's attention on image tokens. Specifically, MoD measures the consistency
between outputs generated from the original image tokens and those derived from
the model's attended image tokens, to distinguish the correctness
aforementioned. If the outputs are consistent, indicating correct attention,
MoD employs a complementary strategy to amplify critical information.
Conversely, if the outputs are inconsistent, suggesting erroneous attention,
MoD utilizes a contrastive strategy to suppress misleading information.
Extensive experiments demonstrate that MoD significantly outperforms existing
decoding methods across multiple mainstream benchmarks, effectively mitigating
hallucinations in LVLMs. The code is available at
https://github.com/xlchen0205/MoD.
