OViP: Online Vision-Language Preference Learning
Journal:
arXiv
Published Date:
May 21, 2025
Abstract
Large vision-language models (LVLMs) remain vulnerable to hallucination,
often generating content misaligned with visual inputs. While recent approaches
advance multi-modal Direct Preference Optimization (DPO) to mitigate
hallucination, they typically rely on predefined or randomly edited negative
samples that fail to reflect actual model errors, limiting training efficacy.
In this work, we propose an Online Vision-language Preference Learning (OViP)
framework that dynamically constructs contrastive training data based on the
model's own hallucinated outputs. By identifying semantic differences between
sampled response pairs and synthesizing negative images using a diffusion
model, OViP generates more relevant supervision signals in real time. This
failure-driven training enables adaptive alignment of both textual and visual
preferences. Moreover, we refine existing evaluation protocols to better
capture the trade-off between hallucination suppression and expressiveness.
Experiments on hallucination and general benchmarks demonstrate that OViP
effectively reduces hallucinations while preserving core multi-modal
capabilities.
