AI in Action: A Roadmap from the Radiology AI Council for Effective Model Evaluation and Deployment.

Journal: Journal of the American College of Radiology : JACR
Published Date: May 23, 2025

Abstract

As the integration of artificial intelligence (AI) into radiology workflows continues to evolve, establishing standardized processes for the evaluation and deployment of AI models is crucial to ensure success. This paper outlines the creation of a Radiology AI Council at a large academic center and subsequent development of framework in the form of a rubric to formalize the evaluation of radiology AI models and onboard them into clinical workflows. The rubric aims to address the challenges faced during the deployment of AI models, such as real-world model performance, workflow implementation, resource allocation, return on investment (ROI), and impact to the broader health system. Using this comprehensive rubric, the council aims to ensure that the process for selecting AI models is both standardized and transparent. This paper outlines the steps taken to establish this rubric, its components, and initial results from evaluation of 13 models over an 8-month period. We emphasize the importance of holistic model evaluation beyond performance metrics, and transparency and objectivity in AI model evaluation with the goal of improving the efficacy and safety of AI models in radiology.

Authors

  • Hari Trivedi
    Department of Radiology, Medical College of Georgia at Augusta University, 1120 15th St, Augusta, GA 30912 (Y.T.); and Department of Radiology, Emory University, Atlanta, Ga (B.V., E.K., A.P., J.G., N.S., H.T.).
  • Bardia Khosravi
    Department of Radiology, Radiology Informatics Lab, Mayo Clinic, Rochester, MN 55905, United States.
  • Judy Gichoya
    Department of Radiology, Medical College of Georgia at Augusta University, 1120 15th St, Augusta, GA 30912 (Y.T.); and Department of Radiology, Emory University, Atlanta, Ga (B.V., E.K., A.P., J.G., N.S., H.T.).
  • Laura Benson
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Damian Dyckman
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • James Galt
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Brian Howard
    Sciome LLC, Research Triangle Park, NC, USA.
  • Elias Kikano
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Jean Kunjummen
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Neil Lall
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Xiao Li
    Department of Inner Mongolia Clinical Medicine College, Inner Mongolia Medical University, Hohhot, Inner Mongolia, China.
  • Sumir Patel
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Nabile Safdar
    Department of Radiology, Medical College of Georgia at Augusta University, 1120 15th St, Augusta, GA 30912 (Y.T.); and Department of Radiology, Emory University, Atlanta, Ga (B.V., E.K., A.P., J.G., N.S., H.T.).
  • Ninad Salastekar
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Colin Segovis
    Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
  • Marly van Assen
    Division of Cardiovascular Imaging, Department of Radiology and Radiological Science, Medical University of South Carolina, Ashley River Tower, 25 Courtenay Dr, Charleston, SC 29425-2260 (S.S.M., D.M., M.v.A., C.N.D.C., R.R.B., C.T., A.V.S., A.M.F., B.E.J., L.P.G., U.J.S.); Department of Diagnostic and Interventional Radiology, University Hospital Frankfurt, Frankfurt, Germany (S.S.M., T.J.V.); Stanford University School of Medicine, Department of Radiology, Stanford, Calif (D.M.); Division of Cardiothoracic Imaging, Nuclear Medicine and Molecular Imaging, Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.N.D.C.); Division of Cardiology, Department of Medicine, Medical University of South Carolina, Charleston, SC (R.R.B.); Department of Cardiology and Intensive Care Medicine, Heart Center Munich-Bogenhausen, Munich, Germany (C.T.); Department of Cardiology, Munich University Clinic, Ludwig-Maximilians-University, Munich, Germany (C.T.); Siemens Medical Solutions USA, Malvern, Pa (P.S.); and Department of Emergency Medicine, Medical University of South Carolina, Charleston, SC (A.J.M.).
  • Peter Harri
    Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, Georgia.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals