Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization
Journal:
arXiv
Published Date:
Apr 12, 2025
Abstract
Text-to-image (T2I) diffusion models have achieved remarkable success in
generating high-quality images from textual prompts. However, their ability to
store vast amounts of knowledge raises concerns in scenarios where selective
forgetting is necessary, such as removing copyrighted content, reducing biases,
or eliminating harmful concepts. While existing unlearning methods can remove
certain concepts, they struggle with multi-concept forgetting due to
instability, residual knowledge persistence, and generation quality
degradation. To address these challenges, we propose \textbf{Dynamic Mask
coupled with Concept-Aware Loss}, a novel unlearning framework designed for
multi-concept forgetting in diffusion models. Our \textbf{Dynamic Mask}
mechanism adaptively updates gradient masks based on current optimization
states, allowing selective weight modifications that prevent interference with
unrelated knowledge. Additionally, our \textbf{Concept-Aware Loss} explicitly
guides the unlearning process by enforcing semantic consistency through
superclass alignment, while a regularization loss based on knowledge
distillation ensures that previously unlearned concepts remain forgotten during
sequential unlearning. We conduct extensive experiments to evaluate our
approach. Results demonstrate that our method outperforms existing unlearning
techniques in forgetting effectiveness, output fidelity, and semantic
coherence, particularly in multi-concept scenarios. Our work provides a
principled and flexible framework for stable and high-fidelity unlearning in
generative models. The code will be released publicly.
