DeepCOVID-XR: An Artificial Intelligence Algorithm to Detect COVID-19 on Chest Radiographs Trained and Tested on a Large U.S. Clinical Data Set.

Journal: Radiology
Published Date: Nov 24, 2020

Abstract

Background There are characteristic findings of coronavirus disease 2019 (COVID-19) on chest images. An artificial intelligence (AI) algorithm to detect COVID-19 on chest radiographs might be useful for triage or infection control within a hospital setting, but prior reports have been limited by small data sets, poor data quality, or both. Purpose To present DeepCOVID-XR, a deep learning AI algorithm to detect COVID-19 on chest radiographs, that was trained and tested on a large clinical data set. Materials and Methods DeepCOVID-XR is an ensemble of convolutional neural networks developed to detect COVID-19 on frontal chest radiographs, with reverse-transcription polymerase chain reaction test results as the reference standard. The algorithm was trained and validated on 14 788 images (4253 positive for COVID-19) from sites across the Northwestern Memorial Health Care System from February 2020 to April 2020 and was then tested on 2214 images (1192 positive for COVID-19) from a single hold-out institution. Performance of the algorithm was compared with interpretations from five experienced thoracic radiologists on 300 random test images using the McNemar test for sensitivity and specificity and the DeLong test for the area under the receiver operating characteristic curve (AUC). Results A total of 5853 patients (mean age, 58 years ± 19 [standard deviation]; 3101 women) were evaluated across data sets. For the entire test set, accuracy of DeepCOVID-XR was 83%, with an AUC of 0.90. For 300 random test images (134 positive for COVID-19), accuracy of DeepCOVID-XR was 82%, compared with that of individual radiologists (range, 76%-81%) and the consensus of all five radiologists (81%). DeepCOVID-XR had a significantly higher sensitivity (71%) than one radiologist (60%, < .001) and significantly higher specificity (92%) than two radiologists (75%, < .001; 84%, = .009). AUC of DeepCOVID-XR was 0.88 compared with the consensus AUC of 0.85 ( = .13 for comparison). With consensus interpretation as the reference standard, the AUC of DeepCOVID-XR was 0.95 (95% CI: 0.92, 0.98). Conclusion DeepCOVID-XR, an artificial intelligence algorithm, detected coronavirus disease 2019 on chest radiographs with a performance similar to that of experienced thoracic radiologists in consensus. © RSNA, 2020 See also the editorial by van Ginneken in this issue.

Authors

  • Ramsey M Wehbe
    Division of Cardiology, Department of Medicine, Northwestern University Feinberg School of Medicine, 676 North St. Clair Street, Suite 600, Chicago, IL 60611, USA. Electronic address: https://twitter.com/ramseywehbemd.
  • Jiayue Sheng
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Shinjan Dutta
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Siyuan Chai
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Amil Dravid
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Semih Barutcu
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Yunan Wu
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Donald R Cantrell
    Department of Radiology, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA.
  • Nicholas Xiao
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Bradley D Allen
    Department of Biomedical Engineering and Physics (J.T.P.) and Department of Radiology & Nuclear Medicine (P.v.O.), Academic Medical Center, Amsterdam University Medical Centers, Location AMC, Meibergdreef 9, 1105 AZ Amsterdam, the Netherlands; Department of Radiology (B.D.A., J.C.C., M.M.), Department of Medicine-Cardiology (R.O.B., L.C.), and Department of Biomedical Engineering (M.M.), Northwestern University, Chicago, Ill; and Department of Radiology & Bioengineering, Children's Hospital Colorado, University of Colorado Anschutz Medical Campus, Denver, Colo (A.J.B.).
  • Gregory A MacNealy
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Hatice Savas
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Rishi Agrawal
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Nishant Parekh
    From the Division of Cardiology, Department of Medicine and Bluhm Cardiovascular Institute (R.M.W.), Division of Neurointerventional Radiology (D.R.C.), Division of Interventional Radiology (N.X.), and Division of Thoracic Imaging (B.D.A., G.A.M., H.S., R.A., N.P.), Department of Radiology, Northwestern Memorial Hospital, 676 N St Clair St, Chicago, IL 60611; and Department of Electrical and Computer Engineering, McCormick School of Engineering, Northwestern University, Evanston, Ill (J.S., S.D., S.C., A.D., S.B., Y.W., A.K.K.).
  • Aggelos K Katsaggelos
    Department of Electrical and Computer Engineering, McCormick School of Engineering and Applied Science, Northwestern University, Evanston, Illinois.

Keywords

External Resources

Popular Topics
Recent Journals