Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas
Journal:
arXiv
Published Date:
Jan 20, 2025
Abstract
LLMs are tuned to follow instructions (aligned) by learning which of two
outputs users prefer for a prompt. However, this preference data format does
not convey why users prefer responses that are chosen or rejected, so LLMs
trained on these datasets cannot tailor responses to varied user needs. To
surface these parameters of personalization, we apply abductive reasoning to
preference data, inferring needs and interests of users, i.e. personas, that
may prefer each output. We test this idea in two steps: Persona Inference
(PI)-abductively inferring personas of users who prefer chosen or rejected
outputs-and Persona Tailoring (PT)-training models to tailor responses to
personas from PI. We find: 1) LLMs infer personas accurately explaining why
different users may prefer both chosen or rejected outputs; 2) Training on
preference data augmented with PI personas via PT boosts personalization,
enabling models to support user-written personas; and 3) Rejected response
personas form harder personalization evaluations, showing PT better aids users
with uncommon preferences versus typical alignment methods. We argue for an
abductive view of preferences for personalization, asking not only which
response is better but when, why, and for whom.
