Narrowing the gap: expected versus deployment performance.

Journal: Journal of the American Medical Informatics Association : JAMIA
Published Date: Aug 18, 2023

Abstract

OBJECTIVES: Successful model development requires both an accurate a priori understanding of future performance and high performance on deployment. Optimistic estimations of model performance that are unrealized in real-world clinical settings can contribute to nonuse of predictive models. This study used 2 tasks, predicting ICU mortality and Bi-Level Positive Airway Pressure failure, to quantify: (1) how well internal test performances derived from different methods of partitioning data into development and test sets estimate future deployment performance of Recurrent Neural Network models and (2) the effects of including older data in the training set on models' performance.

Authors

  • Alice X Zhou
    Department of Anesthesiology and Critical Care Medicine, Children's Hospital Los Angeles, Los Angeles, California, USA.
  • Melissa D Aczon
    Laura P. and Leland K. Whittier Virtual Pediatric Intensive Care Unit, Children's Hospital Los Angeles, Los Angeles, CA, USA.
  • Eugene Laksana
    The Laura P. and Leland K. Whittier Virtual Pediatric Intensive Care Unit, Children's Hospital Los Angeles, 4650 Sunset Blvd, Los Angeles, CA 90027, United States. Electronic address: elaksana@chla.usc.edu.
  • David R Ledbetter
  • Randall C Wetzel

Keywords

External Resources

Popular Topics
Recent Journals