NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models
Journal:
arXiv
Published Date:
Jun 15, 2025
Abstract
Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable
capabilities in understanding relationships between visual and textual data
through joint embedding spaces. Despite their effectiveness, these models
remain vulnerable to adversarial attacks, particularly in the image modality,
posing significant security concerns. Building upon our previous work on
Adversarial Prompt Tuning (AdvPT), which introduced learnable text prompts to
enhance adversarial robustness in VLMs without extensive parameter training, we
present a significant extension by introducing the Neural Augmentor framework
for Multi-modal Adversarial Prompt Tuning (NAP-Tuning).Our key innovations
include: (1) extending AdvPT from text-only to multi-modal prompting across
both text and visual modalities, (2) expanding from single-layer to multi-layer
prompt architectures, and (3) proposing a novel architecture-level redesign
through our Neural Augmentor approach, which implements feature purification to
directly address the distortions introduced by adversarial attacks in feature
space. Our NAP-Tuning approach incorporates token refiners that learn to
reconstruct purified features through residual connections, allowing for
modality-specific and layer-specific feature correction.Comprehensive
experiments demonstrate that NAP-Tuning significantly outperforms existing
methods across various datasets and attack types. Notably, our approach shows
significant improvements over the strongest baselines under the challenging
AutoAttack benchmark, outperforming them by 33.5% on ViT-B16 and 33.0% on
ViT-B32 architectures while maintaining competitive clean accuracy.
