Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies
Journal:
arXiv
Published Date:
May 15, 2025
Abstract
Designing and evaluating personalized and proactive assistant agents remains
challenging due to the time, cost, and ethical concerns associated with
human-in-the-loop experimentation. Existing Human-Computer Interaction (HCI)
methods often require extensive physical setup and human participation, which
introduces privacy concerns and limits scalability. Simulated environments
offer a partial solution but are typically constrained by rule-based scenarios
and still depend heavily on human input to guide interactions and interpret
results. Recent advances in large language models (LLMs) have introduced the
possibility of generative agents that can simulate realistic human behavior,
reasoning, and social dynamics. However, their effectiveness in modeling
human-assistant interactions remains largely unexplored. To address this gap,
we present a generative agent-based simulation platform designed to simulate
human-assistant interactions. We identify ten prior studies on assistant agents
that span different aspects of interaction design and replicate these studies
using our simulation platform. Our results show that fully simulated
experiments using generative agents can approximate key aspects of
human-assistant interactions. Based on these simulations, we are able to
replicate the core conclusions of the original studies. Our work provides a
scalable and cost-effective approach for studying assistant agent design
without requiring live human subjects. We will open source both the platform
and collected results from the experiments on our website:
https://dash-gidea.github.io/.
