UniErase: Unlearning Token as a Universal Erasure Primitive for Language Models
Journal:
arXiv
Published Date:
May 21, 2025
Abstract
Large language models require iterative updates to address challenges such as
knowledge conflicts and outdated information (e.g., incorrect, private, or
illegal contents). Machine unlearning provides a systematic methodology for
targeted knowledge removal from trained models, enabling elimination of
sensitive information influences. However, mainstream fine-tuning-based
unlearning methods often fail to balance unlearning efficacy and model ability,
frequently resulting in catastrophic model collapse under extensive knowledge
removal. Meanwhile, in-context unlearning, which relies solely on contextual
prompting without modifying the model's intrinsic mechanisms, suffers from
limited generalizability and struggles to achieve true unlearning. In this
work, we introduce UniErase, a novel unlearning paradigm that employs learnable
parametric suffix (unlearning token) to steer language models toward targeted
forgetting behaviors. UniErase operates through two key phases: (I) an
optimization stage that binds desired unlearning outputs to the model's
autoregressive probability distribution via token optimization, followed by
(II) a lightweight model editing phase that activates the learned token to
probabilistically induce specified forgetting objective. Serving as a new
research direction for token learning to induce unlearning target, UniErase
achieves state-of-the-art (SOTA) performance across batch, sequential, and
precise unlearning under fictitious and real-world knowledge settings.
Remarkably, in terms of TOFU benchmark, UniErase, modifying only around 3.66%
of the LLM parameters, outperforms previous forgetting SOTA baseline by around
4.01 times for model ability with even better unlearning efficacy. Similarly,
UniErase, maintaining more ability, also surpasses previous retaining SOTA by
35.96% for unlearning efficacy, showing dual top-tier performances in current
unlearing domain.
