Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation
Journal:
arXiv
Published Date:
May 26, 2025
Abstract
The rapid advancement of diffusion models has enhanced their image inpainting
and editing capabilities but also introduced significant societal risks.
Adversaries can exploit user images from social media to generate misleading or
harmful content. While adversarial perturbations can disrupt inpainting, global
perturbation-based methods fail in mask-guided editing tasks due to spatial
constraints. To address these challenges, we propose Structure Disruption
Attack (SDA), a powerful protection framework for safeguarding sensitive image
regions against inpainting-based editing. Building upon the contour-focused
nature of self-attention mechanisms of diffusion models, SDA optimizes
perturbations by disrupting queries in self-attention during the initial
denoising step to destroy the contour generation process. This targeted
interference directly disrupts the structural generation capability of
diffusion models, effectively preventing them from producing coherent images.
We validate our motivation through visualization techniques and extensive
experiments on public datasets, demonstrating that SDA achieves
state-of-the-art (SOTA) protection performance while maintaining strong
robustness.
