Understanding contraceptive switching rationales from real world clinical notes using large language models.

Journal: NPJ digital medicine
Published Date: Apr 23, 2025

Abstract

Understanding reasons for treatment switching is of significant medical interest, but these factors are often only found in unstructured clinical notes and can be difficult to extract. We evaluated the zero-shot abilities of GPT-4 and eight other open-source large language models (LLMs) to extract contraceptive switching information from 1964 clinical notes derived from the UCSF Information Commons dataset. GPT-4 extracted the contraceptives started and stopped at each switch with microF1 scores of 0.85 and 0.88, respectively, compared to 0.81 and 0.88 for the best open-source model. When evaluated by clinical experts, GPT-4 extracted reasons for switching with an accuracy of 91.4% (2.2% hallucination rate). Transformer-based topic modeling identified patient preference, adverse events, and insurance coverage as key reasons. These findings demonstrate the value of LLMs in identifying complex treatment factors and provide insights into reasons for contraceptive switching in real-world settings.

Authors

  • Brenda Y Miao
    Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA, USA. miao.brenda1@gmail.com.
  • Christopher Y K Williams
    Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA, USA.
  • Ebenezer Chinedu-Eneh
    Department of Medicine, University of California San Francisco, San Francisco, CA, USA.
  • Travis Zack
    Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA.
  • Emily Alsentzer
    Biomedical Informatics Training Program, Stanford University, Stanford, CA.
  • Atul J Butte
    Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA.
  • Irene Y Chen
    Computational Precision Health, University of California San Francisco, San Francisco, CA, University of California Berkeley, Berkeley, CA.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals