Assessment of Radiology Artificial Intelligence Software: A Validation and Evaluation Framework.

Journal: Canadian Association of Radiologists journal = Journal l'Association canadienne des radiologistes
Published Date: Nov 6, 2022

Abstract

Artificial intelligence (AI) software in radiology is becoming increasingly prevalent and performance is improving rapidly with new applications for given use cases being developed continuously, oftentimes with development and validation occurring in parallel. Several guidelines have provided reporting standards for publications of AI-based research in medicine and radiology. Yet, there is an unmet need for recommendations on the assessment of AI software before adoption and after commercialization. As the radiology AI ecosystem continues to grow and mature, a formalization of system assessment and evaluation is paramount to ensure patient safety, relevance and support to clinical workflows, and optimal allocation of limited AI development and validation resources before broader implementation into clinical practice. To fulfil these needs, we provide a glossary for AI software types, use cases and roles within the clinical workflow; list healthcare needs, key performance indicators and required information about software prior to assessment; and lay out examples of software performance metrics per software category. This conceptual framework is intended to streamline communication with the AI software industry and provide healthcare decision makers and radiologists with tools to assess the potential use of these software. The proposed software evaluation framework lays the foundation for a radiologist-led prospective validation network of radiology AI software. Learning Points: The rapid expansion of AI applications in radiology requires standardization of AI software specification, classification, and evaluation. The Canadian Association of Radiologists' AI Tech & Apps Working Group Proposes an AI Specification document format and supports the implementation of a clinical expert evaluation process for Radiology AI software.

Authors

  • William Tanguay
    Département de Radiologie, Radio-Oncologie et Médecine Nucléaire, Faculté de médecine, Université de Montréal, Montréal, Canada.
  • Philippe Acar
    60352Centre hospitalier de l'Université de Montréal, Montréal, QC, Canada.
  • Benjamin Fine
    Institute for Medical Engineering and Science, Massachusetts Institute of Technology, Cambridge, MA, USA (LAC); Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA (LAC); Department of Diagnostic Imaging and Operational Analytics Lab, Trillium Health Partners, Mississauga, ON, Canada (BF); Department of Medical Imaging, University of Toronto, ON, Canada (BF); Departments of Anesthesiology and Neurosurgery and the Center for Advanced Data Analytics, University of Virginia, Charlottesville, USA (DJS).
  • Mohamed Abdolell
    Department of Radiology, Dalhousie University, Halifax, NS, Canada.
  • Bo Gong
    MD Undergraduate Program, University of British Columbia, Vancouver, British Columbia, Canada; Department of Radiology, Vancouver General Hospital, University of British Columbia, Vancouver, 899 12th Avenue West, British Columbia V5Z 1M9, Canada. Electronic address: bogong.ustc@gmail.com.
  • Alexandre Cadrin-Chênevert
    Department of Medical Imaging, CISSS Lanaudière, Université Laval, Joliette, Québec, Canada.
  • Carl Chartrand-Lefebvre
    Centre hospitalier de l'Université de Montréal, Montréal, Canada.
  • Jean Chalaoui
    60352Centre hospitalier de l'Université de Montréal, Montréal, QC, Canada.
  • Andrei Gorgos
    60352Centre hospitalier de l'Université de Montréal, Montréal, QC, Canada.
  • Anne Shu-Lei Chin
    60352Centre hospitalier de l'Université de Montréal, Montréal, QC, Canada.
  • Julie Prénovault
    60352Centre hospitalier de l'Université de Montréal, Montréal, QC, Canada.
  • Francois Guilbert
    Department of Radiology, Centre Hospitalier de l'Université de Montréal (CHUM), 1051, rue Sanguinet, Montréal, Quebec H2X 0C1, Canada; Centre de Recherche du CHUM (CRCHUM), 900 St Denis St, Montréal, Quebec H2X 0A9, Canada.
  • Laurent Letourneau-Guillon
    Department of Radiology, Centre Hospitalier de l'Université de Montréal (CHUM), 1051, rue Sanguinet, Montréal, Quebec H2X 0C1, Canada; Centre de Recherche du CHUM (CRCHUM), 900 St Denis St, Montréal, Quebec H2X 0A9, Canada. Electronic address: laurent.letourneau-guillon.1@umontreal.ca.
  • Jaron Chong
    Department of Radiology, McGill University Health Center, Montréal, Québec, Canada.
  • An Tang
    Centre de recherche du Centre hospitalier de l'Université de Montréal (CRCHUM), Montreal, Canada.

Keywords

External Resources

Popular Topics
Recent Journals