Leveraging Implicit Sentiments: Enhancing Reliability and Validity in Psychological Trait Evaluation of LLMs
Journal:
arXiv
Published Date:
Mar 26, 2025
Abstract
Recent advancements in Large Language Models (LLMs) have led to their
increasing integration into human life. With the transition from mere tools to
human-like assistants, understanding their psychological aspects-such as
emotional tendencies and personalities-becomes essential for ensuring their
trustworthiness. However, current psychological evaluations of LLMs, often
based on human psychological assessments like the BFI, face significant
limitations. The results from these approaches often lack reliability and have
limited validity when predicting LLM behavior in real-world scenarios. In this
work, we introduce a novel evaluation instrument specifically designed for
LLMs, called Core Sentiment Inventory (CSI). CSI is a bilingual tool, covering
both English and Chinese, that implicitly evaluates models' sentiment
tendencies, providing an insightful psychological portrait of LLM across three
dimensions: optimism, pessimism, and neutrality. Through extensive experiments,
we demonstrate that: 1) CSI effectively captures nuanced emotional patterns,
revealing significant variation in LLMs across languages and contexts; 2)
Compared to current approaches, CSI significantly improves reliability,
yielding more consistent results; and 3) The correlation between CSI scores and
the sentiment of LLM's real-world outputs exceeds 0.85, demonstrating its
strong validity in predicting LLM behavior. We make CSI public available via:
https://github.com/dependentsign/CSI.
