Transferable Adversarial Attacks on Black-Box Vision-Language Models
Journal:
arXiv
Published Date:
May 2, 2025
Abstract
Vision Large Language Models (VLLMs) are increasingly deployed to offer
advanced capabilities on inputs comprising both text and images. While prior
research has shown that adversarial attacks can transfer from open-source to
proprietary black-box models in text-only and vision-only contexts, the extent
and effectiveness of such vulnerabilities remain underexplored for VLLMs. We
present a comprehensive analysis demonstrating that targeted adversarial
examples are highly transferable to widely-used proprietary VLLMs such as
GPT-4o, Claude, and Gemini. We show that attackers can craft perturbations to
induce specific attacker-chosen interpretations of visual information, such as
misinterpreting hazardous content as safe, overlooking sensitive or restricted
material, or generating detailed incorrect responses aligned with the
attacker's intent. Furthermore, we discover that universal perturbations --
modifications applicable to a wide set of images -- can consistently induce
these misinterpretations across multiple proprietary VLLMs. Our experimental
results on object recognition, visual question answering, and image captioning
show that this vulnerability is common across current state-of-the-art models,
and underscore an urgent need for robust mitigations to ensure the safe and
secure deployment of VLLMs.
