Real-time Identity Defenses against Malicious Personalization of Diffusion Models
Journal:
arXiv
Published Date:
Dec 13, 2024
Abstract
Personalized generative diffusion models, capable of synthesizing highly
realistic images based on a few reference portraits, may pose substantial
social, ethical, and legal risks via identity replication. Existing defense
mechanisms rely on computationally intensive adversarial perturbations tailored
to individual images, rendering them impractical for real-world deployment.
This study introduces the Real-time Identity Defender (RID), a neural network
designed to generate adversarial perturbations through a single forward pass,
bypassing the need for image-specific optimization. RID achieves unprecedented
efficiency, with defense times as low as 0.12 seconds on a single NVIDIA A100
80G GPU (4,400 times faster than leading methods) and 1.1 seconds per image on
a standard Intel i9 CPU, making it suitable for edge devices such as
smartphones. Despite its efficiency, RID achieves promising protection
performance across visual and quantitative benchmarks, effectively mitigating
identity replication risks. Our analysis reveals that RID's perturbations mimic
the efficacy of traditional defenses while exhibiting properties distinct from
natural noise, such as Gaussian perturbations. To enhance robustness, we extend
RID into an ensemble framework that integrates multiple pre-trained
text-to-image diffusion models, ensuring resilience against black-box attacks
and post-processing techniques, including image compression and purification.
Our model is envisioned to play a crucial role in safeguarding portrait rights,
thereby preventing illegal and unethical uses.
