Development and validation of the provider documentation summarization quality instrument for large language models.

Journal: Journal of the American Medical Informatics Association : JAMIA
Published Date: Jun 1, 2025

Abstract

OBJECTIVES: As large language models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation and as models and documentation practices evolve. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed to evaluate LLM-generated clinical summaries. This study aimed to validate the PDSQI-9 across key aspects of construct validity.

Authors

  • Emma Croxford
    Department of Biostatistics and Medical Informatics, University of Wisconsin, Madison, WI 53792, United States.
  • Yanjun Gao
    Department of Biomedical Informatics, University of Colorado-Anschutz Medical, Aurora, CO 80045, United States.
  • Nicholas Pellegrino
    Epic Systems, Verona, WI 53593, United States.
  • Karen Wong
    Epic Systems, Verona, WI 53593, United States.
  • Graham Wills
    UW Health, Madison, WI 53726, United States.
  • Elliot First
    Epic Systems, Verona, WI 53593, United States.
  • Miranda Schnier
    Epic Systems, Verona, WI 53593, United States.
  • Kyle Burton
    UW Health, Madison, WI 53726, United States.
  • Cris Ebby
    Department of Pediatrics, University of Wisconsin, Madison, WI 53792, United States.
  • Jillian Gorski
    UW Health, Madison, WI 53726, United States.
  • Matthew Kalscheur
    UW Health, Madison, WI 53726, United States.
  • Samy Khalil
    UW Health, Madison, WI 53726, United States.
  • Marie Pisani
    Department of Medicine, University of Wisconsin, Madison, WI 53792, United States.
  • Tyler Rubeor
    UW Health, Madison, WI 53726, United States.
  • Peter Stetson
    Memorial Sloan Kettering Cancer Center, New York, NY 10065, United States.
  • Frank Liao
    UW Health, Madison, WI 53726, United States.
  • Cherodeep Goswami
    UW Health, Madison, WI 53726, United States.
  • Brian Patterson
    UW Health, Madison, WI 53726, United States.
  • Majid Afshar
    Loyola University Chicago, Chicago, IL.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals