Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
Journal:
arXiv
Published Date:
Feb 3, 2025
Abstract
Multi-modal Large Language Models (MLLMs) excel in vision-language tasks but
remain vulnerable to visual adversarial perturbations that can induce
hallucinations, manipulate responses, or bypass safety mechanisms. Existing
methods seek to mitigate these risks by applying constrained adversarial
fine-tuning to CLIP vision encoders on ImageNet-scale data, ensuring their
generalization ability is preserved. However, this limited adversarial training
restricts robustness and broader generalization. In this work, we explore an
alternative approach of leveraging existing vision classification models that
have been adversarially pre-trained on large-scale data. Our analysis reveals
two principal contributions: (1) the extensive scale and diversity of
adversarial pre-training enables these models to demonstrate superior
robustness against diverse adversarial threats, ranging from imperceptible
perturbations to advanced jailbreaking attempts, without requiring additional
adversarial training, and (2) end-to-end MLLM integration with these robust
models facilitates enhanced adaptation of language components to robust visual
features, outperforming existing plug-and-play methodologies on complex
reasoning tasks. Through systematic evaluation across visual
question-answering, image captioning, and jail-break attacks, we demonstrate
that MLLMs trained with these robust models achieve superior adversarial
robustness while maintaining favorable clean performance. Our framework
achieves 2x and 1.5x average robustness gains in captioning and VQA tasks,
respectively, and delivers over 10% improvement against jailbreak attacks. Code
and pretrained models will be available at
https://github.com/HashmatShadab/Robust-LLaVA.
