Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation
Journal:
arXiv
Published Date:
May 22, 2025
Abstract
Large vision-language models (LVLMs) have achieved remarkable performance on
multimodal tasks such as visual question answering (VQA) and image captioning.
However, they still suffer from hallucinations, generating text inconsistent
with visual input, posing significant risks in real-world applications.
Existing approaches to address this issue focus on incorporating external
knowledge bases, alignment training, or decoding strategies, all of which
require substantial computational cost and time. Recent works try to explore
more efficient alternatives by adjusting LVLMs' internal representations.
Although promising, these methods may cause hallucinations to be insufficiently
suppressed or lead to excessive interventions that negatively affect normal
semantics. In this work, we leverage sparse autoencoders (SAEs) to identify
semantic directions closely associated with either hallucinations or actuality,
realizing more precise and direct hallucination-related representations. Our
analysis demonstrates that interventions along the faithful direction we
identified can mitigate hallucinations, while those along the hallucinatory
direction can exacerbate them. Building on these insights, we propose Steering
LVLMs via SAE Latent Directions (SSL), a training-free method based on
SAE-derived latent directions to mitigate hallucinations in LVLMs. Extensive
experiments demonstrate that SSL significantly outperforms existing decoding
approaches in mitigating hallucinations, while maintaining transferability
across different model architectures with negligible additional time overhead.
