Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback
Journal:
arXiv
Published Date:
May 6, 2025
Abstract
Large language models (LLMs) have shown promise in providing scalable mental
health support, while evaluating their counseling capability remains crucial to
ensure both efficacy and safety. Existing evaluations are limited by the static
assessment that focuses on knowledge tests, the single perspective that centers
on user experience, and the open-loop framework that lacks actionable feedback.
To address these issues, we propose {\Psi}-Arena, an interactive framework for
comprehensive assessment and optimization of LLM-based counselors, featuring
three key characteristics: (1) Realistic arena interactions that simulate
real-world counseling through multi-stage dialogues with psychologically
profiled NPC clients, (2) Tripartite evaluation that integrates assessments
from the client, counselor, and supervisor perspectives, and (3) Closed-loop
optimization that iteratively improves LLM counselors using diagnostic
feedback. Experiments across eight state-of-the-art LLMs show significant
performance variations in different real-world scenarios and evaluation
perspectives. Moreover, reflection-based optimization results in up to a 141%
improvement in counseling performance. We hope PsychoArena provides a
foundational resource for advancing reliable and human-aligned LLM applications
in mental healthcare.
