Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation
Journal:
arXiv
Published Date:
Dec 19, 2024
Abstract
Direct Preference Optimization (DPO) has been demonstrated to be highly
effective in mitigating hallucinations in Large Vision Language Models (LVLMs)
by aligning their outputs more closely with human preferences. Despite the
recent progress, existing methods suffer from two drawbacks: 1) Lack of
scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this
end, we propose a novel Token Preference Optimization model with
self-calibrated rewards (dubbed as TPO), which adaptively attends to
visual-correlated tokens without fine-grained annotations. Specifically, we
introduce a token-level \emph{visual-anchored} \emph{reward} as the difference
of the logistic distributions of generated tokens conditioned on the raw image
and the corrupted one. In addition, to highlight the informative
visual-anchored tokens, a visual-aware training objective is proposed to
enhance more accurate token-level optimization. Extensive experimental results
have manifested the state-of-the-art performance of the proposed TPO. For
example, by building on top of LLAVA-1.5-7B, our TPO boosts the performance
absolute improvement for hallucination benchmarks.
