I'll believe it when I see it: Images increase misinformation sharing in Vision-Language Models
Journal:
arXiv
Published Date:
May 19, 2025
Abstract
Large language models are increasingly integrated into news recommendation
systems, raising concerns about their role in spreading misinformation. In
humans, visual content is known to boost credibility and shareability of
information, yet its effect on vision-language models (VLMs) remains unclear.
We present the first study examining how images influence VLMs' propensity to
reshare news content, whether this effect varies across model families, and how
persona conditioning and content attributes modulate this behavior. To support
this analysis, we introduce two methodological contributions: a
jailbreaking-inspired prompting strategy that elicits resharing decisions from
VLMs while simulating users with antisocial traits and political alignments;
and a multimodal dataset of fact-checked political news from PolitiFact, paired
with corresponding images and ground-truth veracity labels. Experiments across
model families reveal that image presence increases resharing rates by 4.8% for
true news and 15.0% for false news. Persona conditioning further modulates this
effect: Dark Triad traits amplify resharing of false news, whereas
Republican-aligned profiles exhibit reduced veracity sensitivity. Of all the
tested models, only Claude-3-Haiku demonstrates robustness to visual
misinformation. These findings highlight emerging risks in multimodal model
behavior and motivate the development of tailored evaluation frameworks and
mitigation strategies for personalized AI systems. Code and dataset are
available at: https://github.com/3lis/misinfo_vlm