A Survey on Progress in LLM Alignment from the Perspective of Reward Design
Journal:
arXiv
Published Date:
May 5, 2025
Abstract
The alignment of large language models (LLMs) with human values and
intentions represents a core challenge in current AI research, where reward
mechanism design has become a critical factor in shaping model behavior. This
study conducts a comprehensive investigation of reward mechanisms in LLM
alignment through a systematic theoretical framework, categorizing their
development into three key phases: (1) feedback (diagnosis), (2) reward design
(prescription), and (3) optimization (treatment). Through a four-dimensional
analysis encompassing construction basis, format, expression, and granularity,
this research establishes a systematic classification framework that reveals
evolutionary trends in reward modeling. The field of LLM alignment faces
several persistent challenges, while recent advances in reward design are
driving significant paradigm shifts. Notable developments include the
transition from reinforcement learning-based frameworks to novel optimization
paradigms, as well as enhanced capabilities to address complex alignment
scenarios involving multimodal integration and concurrent task coordination.
Finally, this survey outlines promising future research directions for LLM
alignment through innovative reward design strategies.
