Feasibility of Real-Time Integration of a Large Language Model in the EHR to Identify Goals-of-Care Documentation.

CONTEXT: This article describes integration of a large language model (LLM) into the electronic health record (EHR) to identify unstructured goals-of-care (GOC) documentation or discrete data entry. OBJECTIVES: To determine whether a prompt-engineered LLM reliably detects GOC documentation in clinical notes comparable to human experts and can display this evaluation to clinicians in real time. METHODS: An interdisciplinary team of palliative care experts created an adjudicated reference standard, which was used to evaluate LLM models in detecting GOC documentation. Given this, the best-fit model (GPT-4o) was then integrated with the EHR for a six-month pilot at four hospitals, then released enterprise-wide. RESULTS: GPT-4o achieved the best performance (specificity 0.95, recall 0.74, precision 0.78). Manual audit on AI-flagged notes (n=325) confirmed substantial expert agreement (86%). In subgroup analyses, there was no statistically significant differences across race, sex, or age (specificity range 0.91-1; all p-values > 0.05). At enterprise scale, the model identified 37,000 patients with GOC documentation within approximately two months. In comparison to those without, patients with AI-flagged GOC documentation had higher acuity rates (ICU stay ≥5 days: 30.2% vs 8.2%; in-hospital mortality: 14.9% vs 0.6%; hospice discharge: 10.5% vs 0.4%). Code status changes were also significantly more frequent (29.3% vs 9.9%; p<0.05). CONCLUSION: Our initiative met the original objectives by providing a financially sustainable EHR optimization that converts unstructured, high-value GOC narratives into real-time discoverable and trackable documentation. Detecting and displaying these conversations is crucial, increasing clinical situational awareness while reducing documentation burden, leading to enhanced goal-concordant care.