A Mixed-Methods Evaluation of LLM-Based Chatbots for Menopause
Journal:
arXiv
Published Date:
Feb 5, 2025
Abstract
The integration of Large Language Models (LLMs) into healthcare settings has
gained significant attention, particularly for question-answering tasks. Given
the high-stakes nature of healthcare, it is essential to ensure that
LLM-generated content is accurate and reliable to prevent adverse outcomes.
However, the development of robust evaluation metrics and methodologies remains
a matter of much debate. We examine the performance of publicly available
LLM-based chatbots for menopause-related queries, using a mixed-methods
approach to evaluate safety, consensus, objectivity, reproducibility, and
explainability. Our findings highlight the promise and limitations of
traditional evaluation metrics for sensitive health topics. We propose the need
for customized and ethically grounded evaluation frameworks to assess LLMs to
advance safe and effective use in healthcare.
