Algorithmic Identification of Treatment-Emergent Adverse Events From Clinical Notes Using Large Language Models: A Pilot Study in Inflammatory Bowel Disease.

Journal: Clinical pharmacology and therapeutics
PMID: 38459719

Abstract

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California - San Francisco (UCSF)-BERT, to identify serious AEs (SAEs) occurring after treatment with a non-steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE-associated hospitalizations occurring after treatment with a non-steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF-BERT achieved the highest numerical performance on identifying drug-SAE pairs from this corpus (accuracy 88-92%, macro F1 61-68%), with 5-10% greater accuracy than previously published models. UCSF-BERT was significantly superior at identifying hospitalization events emergent to medication use (P < 0.01). LLMs like UCSF-BERT achieve numerically superior accuracy on the challenging task of SAE detection from clinical notes compared with prior methods. Future work is needed to adapt this methodology to improve model performance and evaluation using multicenter data and newer architectures like Generative pre-trained transformer (GPT). Our findings support the potential value of using large language models to enhance pharmacovigilance.

Authors

  • Anna L Silverman
    Division of Gastroenterology and Hepatology, Department of Medicine, Mayo Clinic, Phoenix, Arizona, USA.
  • Madhumita Sushil
    Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, California, USA.
  • Balu Bhasuran
    DRDO-BU Center for Life Sciences, Bharathiar University Campus, Coimbatore, Tamilnadu, India.
  • Dana Ludwig
    Bakar Computational Health Sciences Institute, San Francisco, California, USA.
  • James Buchanan
    Bakar Computational Health Sciences Institute, San Francisco, California, USA.
  • Rebecca Racz
    University of Michigan Medical School, Ann Arbor, MI 48109 USA.
  • Mahalakshmi Parakala
    Department of Public Health, University of California, Berkeley, Berkeley, California, USA.
  • Samer El-Kamary
    United States Food and Drug Administration, Silver Spring, Maryland, USA.
  • Ohenewaa Ahima
    United States Food and Drug Administration, Silver Spring, Maryland, USA.
  • Artur Belov
    United States Food and Drug Administration, Silver Spring, Maryland, USA.
  • Lauren Choi
    United States Food and Drug Administration, Silver Spring, Maryland, USA.
  • Monisha Billings
    United States Food and Drug Administration, Silver Spring, Maryland, USA.
  • Yan Li
    Interdisciplinary Research Center for Biology and Chemistry, Liaoning Normal University, Dalian, China.
  • Nadia Habal
    United States Food and Drug Administration, Silver Spring, Maryland, USA.
  • Qi Liu
    National Institute of Traditional Chinese Medicine Constitution and Preventive Medicine, Beijing University of Chinese Medicine, Beijing, China.
  • Jawahar Tiwari
    United States Food and Drug Administration, Silver Spring, Maryland, USA.
  • Atul J Butte
    Bakar Computational Health Sciences Institute, University of California San Francisco, San Francisco, CA.
  • Vivek A Rudrapatna
    Bakar Computational Health Sciences Institute, San Francisco, California, USA.

Keywords

External Resources

Popular Topics
Recent Journals