Inside you are many wolves: Using cognitive models to interpret value trade-offs in LLMs
Journal:
arXiv
Published Date:
Jun 25, 2025
Abstract
Navigating everyday social situations often requires juggling conflicting
goals, such as conveying a harsh truth, maintaining trust, all while still
being mindful of another person's feelings. These value trade-offs are an
integral part of human decision-making and language use, however, current tools
for interpreting such dynamic and multi-faceted notions of values in LLMs are
limited. In cognitive science, so-called "cognitive models" provide formal
accounts of these trade-offs in humans, by modeling the weighting of a
speaker's competing utility functions in choosing an action or utterance. In
this work, we use a leading cognitive model of polite speech to interpret the
extent to which LLMs represent human-like trade-offs. We apply this lens to
systematically evaluate value trade-offs in two encompassing model settings:
degrees of reasoning "effort" in frontier black-box models, and RL
post-training dynamics of open-source models. Our results highlight patterns of
higher informational utility than social utility in reasoning models, and in
open-source models shown to be stronger in mathematical reasoning. Our findings
from LLMs' training dynamics suggest large shifts in utility values early on in
training with persistent effects of the choice of base model and pretraining
data, compared to feedback dataset or alignment method. We show that our method
is responsive to diverse aspects of the rapidly evolving LLM landscape, with
insights for forming hypotheses about other high-level behaviors, shaping
training regimes for reasoning models, and better controlling trade-offs
between values during model training.
