SAFER: Probing Safety in Reward Models with Sparse Autoencoder
Journal:
arXiv
Published Date:
Jul 1, 2025
Abstract
Reinforcement learning from human feedback (RLHF) is a key paradigm for
aligning large language models (LLMs) with human values, yet the reward models
at its core remain largely opaque. In this work, we present sparse Autoencoder
For Enhanced Reward model (\textbf{SAFER}), a novel framework for interpreting
and improving reward models through mechanistic analysis. Leveraging Sparse
Autoencoders (SAEs), we uncover human-interpretable features in reward model
activations, enabling insight into safety-relevant decision-making. We apply
SAFER to safety-oriented preference datasets and quantify the salience of
individual features by activation differences between chosen and rejected
responses. Using these feature-level signals, we design targeted data poisoning
and denoising strategies. Experiments show that SAFER can precisely degrade or
enhance safety alignment with minimal data modification, without sacrificing
general chat performance. Our approach contributes to interpreting, auditing
and refining reward models in high-stakes LLM alignment tasks. Our codes are
available at https://github.com/xzy-101/SAFER-code. \textit{This paper
discusses topics related to large language model safety and may include
discussions or examples that highlight potential risks or unsafe outcomes.}
