Precise, Fast, and Low-cost Concept Erasure in Value Space: Orthogonal Complement Matters
Journal:
arXiv
Published Date:
Dec 9, 2024
Abstract
Recent success of text-to-image (T2I) generation and its increasing practical
applications, enabled by diffusion models, require urgent consideration of
erasing unwanted concepts, e.g., copyrighted, offensive, and unsafe ones, from
the pre-trained models in a precise, timely, and low-cost manner. The twofold
demand of concept erasure includes not only a precise removal of the target
concept (i.e., erasure efficacy) but also a minimal change on non-target
content (i.e., prior preservation), during generation. Existing methods face
challenges in maintaining an effective balance between erasure efficacy and
prior preservation, and they can be computationally costly. To improve, we
propose a precise, fast, and low-cost concept erasure method, called Adaptive
Value Decomposer (AdaVD), which is training-free. Our method is grounded in a
classical linear algebraic operation of computing the orthogonal complement,
implemented in the value space of each cross-attention layer within the UNet of
diffusion models. We design a shift factor to adaptively navigate the erasure
strength, enhancing effective prior preservation without sacrificing erasure
efficacy. Extensive comparative experiments with both training-based and
training-free state-of-the-art methods demonstrate that the proposed AdaVD
excels in both single and multiple concept erasure, showing 2 to 10 times
improvement in prior preservation than the second best, meanwhile achieving the
best or near best erasure efficacy. AdaVD supports a series of diffusion models
and downstream image generation tasks, with code available on:
https://github.com/WYuan1001/AdaVD.
