COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System.

Journal: Radiology
Published Date: May 8, 2020

Abstract

Background Chest radiography may play an important role in triage for coronavirus disease 2019 (COVID-19), particularly in low-resource settings. Purpose To evaluate the performance of an artificial intelligence (AI) system for detection of COVID-19 pneumonia on chest radiographs. Materials and Methods An AI system (CAD4COVID-XRay) was trained on 24 678 chest radiographs, including 1540 used only for validation while training. The test set consisted of a set of continuously acquired chest radiographs ( = 454) obtained in patients suspected of having COVID-19 pneumonia between March 4 and April 6, 2020, at one center (223 patients with positive reverse transcription polymerase chain reaction [RT-PCR] results, 231 with negative RT-PCR results). Radiographs were independently analyzed by six readers and by the AI system. Diagnostic performance was analyzed with the receiver operating characteristic curve. Results For the test set, the mean age of patients was 67 years ± 14.4 (standard deviation) (56% male). With RT-PCR test results as the reference standard, the AI system correctly classified chest radiographs as COVID-19 pneumonia with an area under the receiver operating characteristic curve of 0.81. The system significantly outperformed each reader ( < .001 using the McNemar test) at their highest possible sensitivities. At their lowest sensitivities, only one reader significantly outperformed the AI system ( = .04). Conclusion The performance of an artificial intelligence system in the detection of coronavirus disease 2019 on chest radiographs was comparable with that of six independent readers. © RSNA, 2020.

Authors

  • Keelin Murphy
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Henk Smits
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Arnoud J G Knoops
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Michael B J M Korst
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Tijs Samson
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Ernst T Scholten
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Steven Schalekamp
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Cornelia M Schaefer-Prokop
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Rick H H M Philipsen
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Annet Meijers
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Jaime Melendez
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).
  • Bram van Ginneken
    Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands; Fraunhofer Mevis, Bremen, Germany.
  • Matthieu Rutten
    From the Diagnostic Image Analysis Group, Radboud University Medical Center, Geert Groteplein 10, Nijmegen 6500 HB, the Netherlands (K.M., E.T.S., S.S., C.M.S., B.v.G.); Department of Radiology, Bernhoven Hospital, Uden, the Netherlands (H.S.); Department of Radiology, Jeroen Bosch Hospital, 's-Hertogenbosch, the Netherlands (A.J.G.K., M.B.J.M.K., T.S., M.R.); Department of Radiology, Meander Medisch Centrum, Amersfoort, the Netherlands (C.M.S.); and Thirona, Nijmegen, the Netherlands (R.H.H.M.P., A.M., J.M.).

Keywords

External Resources

Popular Topics
Recent Journals