Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization
Journal:
arXiv
Published Date:
Dec 2, 2024
Abstract
As large-scale diffusion models continue to advance, they excel at producing
high-quality images but often generate unwanted content, such as sexually
explicit or violent content. Existing methods for concept removal generally
guide the image generation process but can unintentionally modify unrelated
regions, leading to inconsistencies with the original model. We propose a novel
approach for targeted concept replacing in diffusion models, enabling specific
concepts to be removed without affecting non-target areas. Our method
introduces a dedicated concept localizer for precisely identifying the target
concept during the denoising process, trained with few-shot learning to require
minimal labeled data. Within the identified region, we introduce a
training-free Dual Prompts Cross-Attention (DPCA) module to substitute the
target concept, ensuring minimal disruption to surrounding content. We evaluate
our method on concept localization precision and replacement efficiency.
Experimental results demonstrate that our method achieves superior precision in
localizing target concepts and performs coherent concept replacement with
minimal impact on non-target areas, outperforming existing approaches.
