Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition.

Journal: AJNR. American journal of neuroradiology
Published Date:

Abstract

BACKGROUND AND PURPOSE: Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency.

Authors

  • Bin Jiang
    Department of Urology, Chinese People's Liberation Army General Hospital, Beijing, 100039 China.
  • Burak B Ozkara
    Department of Neuroradiology, MD Anderson Cancer Center, Houston, Texas, USA.
  • Guangming Zhu
    1 Department of Radiology, Neuroradiology Division, Stanford University School of Medicine, 300 Pasteur Dr, Stanford, CA 94305.
  • Derek Boothroyd
    Department of Medicine (D.B., V.D.), Stanford University School of Medicine, Stanford, California.
  • Jason W Allen
    Department of Biomedical Informatics, Emory School of Medicine, Atlanta, GA, USA.
  • Daniel P Barboriak
    Department of Radiology (D.P.B.), Duke University Medical Center, Durham, North Carolina.
  • Peter Chang
    Department of Urology, Beth Israel Deaconess Medical Center, Boston, MA, USA.
  • Cynthia Chan
    From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Zaharchuk), Neuroradiology Division, Stanford University, Stanford, California.
  • Ruchir Chaudhari
    From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Zaharchuk), Neuroradiology Division, Stanford University, Stanford, California.
  • Hui Chen
    Xiangyang Central HospitalAffiliated Hospital of Hubei University of Arts and Science Xiangyang 441000 China.
  • Anjeza Chukus
    From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Zaharchuk), Neuroradiology Division, Stanford University, Stanford, California.
  • Victoria Ding
    Department of Medicine (D.B., V.D.), Stanford University School of Medicine, Stanford, California.
  • David Douglas
    From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Zaharchuk), Neuroradiology Division, Stanford University, Stanford, California.
  • Christopher G Filippi
    Radiology, North Shore LIJ Health System, 300 Community Drive, Manhasset, NY, USA.
  • Adam E Flanders
  • Ryan Godwin
    Wake Forest School of Medicine, Winston-Salem, NC, United States.
  • Syed Hashmi
    From the Department of Radiology (B.J., C.C., R.C., A.C., D.D., S.H., G.Zaharchuk), Neuroradiology Division, Stanford University, Stanford, California.
  • Christopher Hess
    Department of Radiology and Biomedical Imaging (C.H.), University of California, San Francisco, San Francisco, California.
  • Kevin Hsu
    Department of Radiology (K.H., Y.W.L), New York University Grossman School of Medicine, New York, New York.
  • Yvonne W Lui
    Center for Advanced Imaging Innovation and Research (CAI2R), School of Medicine, New York University, 660 First Avenue, New York, NY 10016, USA; Bernard and Irene Schwartz Center for Biomedical Imaging, School of Medicine, New York University, 660 First Avenue, New York, NY 10016, USA.
  • Joseph A Maldjian
    Department of Radiology, UT Southwestern Medical Center, Dallas, USA.
  • Patrik Michel
    3 Department of Radiology, Centre Hospitalier Universitaire Vaudois, Lausanne, Switzerland.
  • Sahil S Nalawade
    Department of Radiology, University of Texas Southwestern Medical Center, Dallas, Texas.
  • Vishal Patel
  • Prashant Raghavan
    Department of Diagnostic Radiology and Nuclear Medicine (P.R.), University of Maryland School of Medicine, Baltimore, Maryland.
  • Haris I Sair
    The Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins University School of Medicine, 601 N. Caroline St., Room 4223, Baltimore, MD, 21287, USA.
  • Jody Tanabe
    Department of Radiology (J.T.), University of Colorado, Aurora, Colorado.
  • Kirk Welker
    Department of Radiology (K.W.), Mayo Clinic, Rochester, Minnesota.
  • Christopher T Whitlow
    Radiology Informatics and Image Processing Laboratory (RIIPL), Wake Forest School of Medicine, Medical Center Boulevard, Winston-Salem, NC 27157, USA; Division of Neuroradiology, Department of Radiology, Wake Forest School of Medicine, Medical Center Boulevard, Winston-Salem, NC 27157, USA; Department of Biomedical Engineering, Wake Forest School of Medicine, Medical Center Boulevard, Winston-Salem, NC 27157, USA; Clinical and Translational Sciences Institute (CTSI), Wake Forest School of Medicine, Medical Center Boulevard, Winston-Salem, NC 27157, USA. Electronic address: cwhitlow@wakehealth.edu.
  • Greg Zaharchuk
    Stanford University, Stanford CA 94305, USA.
  • Max Wintermark
    Department of Radiology, Stanford University, Stanford, California, USA.