Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models.

Journal: Modern pathology : an official journal of the United States and Canadian Academy of Pathology, Inc
Published Date: Apr 1, 2024

Abstract

This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).

Authors

  • Katherine Elfer
    United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland; National Institutes of Health, National Cancer Institute, Division of Cancer Prevention, Cancer Prevention Fellowship Program, Bethesda, Maryland. Electronic address: Katherine.Elfer@fda.hhs.gov.
  • Emma Gardecki
    United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.
  • Victor Garcia
    United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.
  • Amy Ly
    Department of Pathology, Massachusetts General Hospital, Boston, Massachusetts.
  • Evangelos Hytopoulos
    System Development, iRhythm Technologies Inc, San Francisco, California.
  • Si Wen
    United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.
  • Matthew G Hanna
    Department of Pathology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania; Computational Pathology and AI Center of Excellence (CPACE), University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania.
  • Dieter J E Peeters
    Department of Pathology, University Hospital Antwerp/University of Antwerp, Antwerp, Belgium; Department of Pathology, Sint-Maarten Hospital, Mechelen, Belgium.
  • Joel Saltz
    Department of Biomedical Informatics, Stony Brook University, Stony Brook, New York.
  • Anna Ehinger
    Department of Clinical Genetics, Pathology and Molecular Diagnostics, Laboratory Medicine, Lund University, Lund, Sweden.
  • Sarah N Dudgeon
    Department of Laboratory Medicine, Yale School of Medicine, New Haven, Connecticut.
  • Xiaoxian Li
    Department of Ultrasound, Sun Yat-sen University Cancer centre, State Key Laboratory of Oncology in South China, Collaborative Innovation centre for Cancer Medicine, Guangzhou, China.
  • Kim R M Blenman
    Department of Internal Medicine, Section of Medical Oncology, Yale School of Medicine and Yale Cancer Center, Yale University, New Haven, Connecticut; Department of Computer Science, School of Engineering and Applied Science, Yale University, New Haven, Connecticut.
  • Weijie Chen
  • Ursula Green
    Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
  • Ryan Birmingham
    United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland; Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
  • Tony Pan
    Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
  • Jochen K Lennerz
    Department of Pathology, Center for Integrated Diagnostics, Harvard Medical School, Massachusetts General Hospital, Boston, MA, USA.
  • Roberto Salgado
    Breast Cancer Translational Research Laboratory, Université Libre de Bruxelles, Brussels, Belgium; Department of Pathology, GZA Hospitals Antwerp, Belgium.
  • Brandon D Gallas
    Division of Imaging Diagnostics, and Software Reliability, US Food and Drug Administration, Silver Spring, MD, USA.

Keywords

External Resources

Popular Topics
Recent Journals