Assessing the Quality of AI-Generated Clinical Notes: A Validated Evaluation of a Large Language Model Scribe
Journal:
arXiv
Published Date:
May 15, 2025
Abstract
In medical practices across the United States, physicians have begun
implementing generative artificial intelligence (AI) tools to perform the
function of scribes in order to reduce the burden of documenting clinical
encounters. Despite their widespread use, no established methods exist to gauge
the quality of AI scribes. To address this gap, we developed a blinded study
comparing the relative performance of large language model (LLM) generated
clinical notes with those from field experts based on audio-recorded clinical
encounters. Quantitative metrics from the Physician Documentation Quality
Instrument (PDQI9) provided a framework to measure note quality, which we
adapted to assess relative performance of AI generated notes. Clinical experts
spanning 5 medical specialties used the PDQI9 tool to evaluate
specialist-drafted Gold notes and LLM authored Ambient notes. Two evaluators
from each specialty scored notes drafted from a total of 97 patient visits. We
found uniformly high inter rater agreement (RWG greater than 0.7) between
evaluators in general medicine, orthopedics, and obstetrics and gynecology, and
moderate (RWG 0.5 to 0.7) to high inter rater agreement in pediatrics and
cardiology. We found a modest yet significant difference in the overall note
quality, wherein Gold notes achieved a score of 4.25 out of 5 and Ambient notes
scored 4.20 out of 5 (p = 0.04). Our findings support the use of the PDQI9
instrument as a practical method to gauge the quality of LLM authored notes, as
compared to human-authored notes.
