Towards Optimizing LLM Use in Healthcare: Identifying Patient Questions in MyChart Messages.

Journal: AMIA ... Annual Symposium proceedings. AMIA Symposium
Published Date: May 22, 2025

Abstract

The volume of patient-provider messages is on the rise, and Large Language Models (LLMs) can potentially streamline the clinical messaging process, but their success hinges on triaging messages they can optimally address. In this study, we analyzed Electronic Health Records with over 4 million messages exchanged between patients and providers to characterize the utility of using LLMs for messages containing knowledge questions. We implemented a rule-based Syntactic Question Detector as a triage tool, and we evaluated it on 500 messages. The interrater reliability metrics and comparison with LLMs show the difficulty of detecting questions due to the informal text and implicit requests. Our results show that 25% of MyChart messages with questions do not have a response from the clinical team. This paper provides insights into the challenges of real-world data, highlights the importance and non-triviality of detecting questions, and suggests a pipeline for LLM use in healthcare.

Authors

  • Akhila Chekuri
    University of California San Diego, La Jolla, CA Computer Science and Engineering.
  • Armaan S Johal
    University of California San Diego, La Jolla, CA Computer Science and Engineering.
  • Matthew R Allen
    University of California San Diego, La Jolla, CA Division of Biomedical Informatics.
  • John W Ayers
    Qualcomm Institute, University of California San Diego, La Jolla.
  • Michael Hogarth
    Division of Biomedical Informatics, Department of Medicine, University of California San Diego, La Jolla, CA, United States.
  • Emilia Farcas
    University of California San Diego, La Jolla, CA Qualcomm Institute.

Keywords

External Resources

Popular Topics
Recent Journals