Understanding metric-related pitfalls in image analysis validation.

Journal: Nature methods
Published Date: Feb 12, 2024

Abstract

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Authors

  • Annika Reinke
    German Cancer Research Center DKFZ, Division of Computer Assisted Medical Interventions, Heidelberg, Germany. Electronic address: a.reinke@dkfz.de.
  • Minu D Tizabi
    Division of Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany; HIP Helmholtz Imaging Platform, German Cancer Research Center, Heidelberg, Germany.
  • Michael Baumgartner
    Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany.
  • Matthias Eisenmann
    German Cancer Research Center (DKFZ), Computer Assisted Medical Interventions, Heidelberg, Germany.
  • Doreen Heckmann-Nötzel
    Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Heidelberg, Germany.
  • A Emre Kavur
    Graduate School of Natural and Applied Sciences, Dokuz Eylül University, İzmir, Turkey.
  • Tim Rädsch
    German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
  • Carole H Sudre
    School of Biomedical Engineering and Imaging Sciences, King's College London, London, UK; Medical Research Council Unit for Lifelong Health and Ageing, Department of Population Science and Experimental Medicine, University College London, London, UK; Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK.
  • Laura Acion
    Instituto de Cálculo, Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires, CONICET, Buenos Aires, Argentina.
  • Michela Antonelli
    Centre for Medical Image Computing, University College London, London, UK.
  • Tal Arbel
    Centre for Intelligent Machines, McGill University, Montréal, Canada.
  • Spyridon Bakas
    Perelman School of Medicine, Philadelphia, PA, USA.
  • Arriel Benis
    Faculty of Industrial Engineering and Technology Management, Holon Institute of Technology, Holon, Israel.
  • Florian Buettner
    European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Wellcome Trust Genome Campus, Hinxton, Cambridge CB10 1SD, UK; Institute of Computational Biology, Helmholtz Zentrum München, Ingolstädter Landstr. 1, 85764 Neuherberg, Germany. Electronic address: buettner@ebi.ac.uk.
  • M Jorge Cardoso
    Department of Biomedical EngineeringSchool of Biomedical Engineering and Imaging SciencesKing's College London WC2R 2LS London U.K.
  • Veronika Cheplygina
    Medical Image Analysis, Department Biomedical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands. Electronic address: v.cheplygina@tue.nl.
  • Jianxu Chen
    Department of Biospectroscopy, Leibniz-Institut für Analytische Wissenschaften - ISAS - e.V., Dortmund, Germany.
  • Evangelia Christodoulou
    Department of Development & Regeneration, KU Leuven, Herestraat 49 box 805, Leuven, 3000 Belgium.
  • Beth A Cimini
    Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA 02142, USA.
  • Keyvan Farahani
    Image-Guided Interventions and Imaging Informatics, National Cancer Institute, National Institutes of Health, Rockville, Maryland, USA.
  • Luciana Ferrer
    Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Autónoma de Buenos Aires, Buenos Aires, Argentina.
  • Adrian Galdran
  • Bram van Ginneken
    Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands; Fraunhofer Mevis, Bremen, Germany.
  • Ben Glocker
    Kheiron Medical Technologies, London, UK.
  • Patrick Godau
    Division of Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany; National Center for Tumor Diseases (NCT), Heidelberg, Germany.
  • Daniel A Hashimoto
    Department of Surgery, Massachusetts General Hospital, Boston, MA.
  • Michael M Hoffman
    Princess Margaret Cancer Centre, Toronto, Ontario, Canada.
  • Merel Huisman
    Department of Radiology, University Medical Center Utrecht, Utrecht, The Netherlands. merel.huisman1@gmail.com.
  • Fabian Isensee
  • Pierre Jannin
  • Charles E Kahn
    Perelman School of Medicine, University of Pennsylvania, Philadelphia 19104, USA.
  • Dagmar Kainmueller
    Max-Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Biomedical Image Analysis and HI Helmholtz Imaging, Berlin, Germany.
  • Bernhard Kainz
    Biomedical Image Analysis (BioMedIA) Group, Department of Computing, Imperial College London, UK.
  • Alexandros Karargyris
    IHU Strasbourg, Strasbourg, France.
  • Jens Kleesiek
    AG Computational Radiology, Abteilung Radiologie, Deutsches Krebsforschungszentrum (DKFZ), Im Neuenheimer Feld 280, 69120, Heidelberg, Deutschland. j.kleesiek@dkfz-heidelberg.de.
  • Florian Kofler
    Department of Computer Science, Institute for AI in Medicine, Technical University of Munich, Munich, Germany; Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany; TranslaTUM, Central Institute for Translational Cancer Research of the Technical University of Munich, Munich, Germany; Helmholtz AI, Helmholtz Munich, Neuherberg, Germany.
  • Thijs Kooi
    Diagnostic Image Analysis Group, Department of Radiology, Radboud University Medical Center, Nijmegen, The Netherlands. Electronic address: thijs.kooi@radboudumc.nl.
  • Annette Kopp-Schneider
    Division of Biostatistics, German Cancer Research Center, Im Neuenheimer Feld 581, 69210, Heidelberg, Germany.
  • Michal Kozubek
    Faculty of Informatics, Masaryk University, Brno, Czech Republic.
  • Anna Kreshuk
    EMBL, Heidelberg, Germany.
  • Tahsin Kurc
    Department of Biomedical Informatics, Stony Brook Medicine, Stony Brook, NY 11794, USA.
  • Bennett A Landman
    Vanderbilt University, Nashville TN 37235, USA.
  • Geert Litjens
    Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands.
  • Amin Madani
    Department of Surgery, Columbia University Irving Medical Center, 161 Fort Washington Avenue, New York, NY 10032, USA.
  • Klaus Maier-Hein
    Medical Image Analysis, Division Medical Image Computing, DKFZ Heidelberg, Germany.
  • Anne L Martel
    Department of Medical Biophysics, University of Toronto, Canada; Department of Imaging Research, Sunnybrook Research Institute, Toronto, Canada.
  • Erik Meijering
    Biomedical Imaging Group Rotterdam, Departments of Medical Informatics and Radiology, Erasmus University Medical Center, Rotterdam, the Netherlands.
  • Bjoern Menze
  • Karel G M Moons
    Julius Center for Health Sciences and Primary Care, and Cochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands.
  • Henning Muller
  • Brennan Nichyporuk
    Centre for Intelligent Machines, Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada.
  • Felix Nickel
    Department of General, Visceral, and Transplantation Surgery, University of Heidelberg, Im Neuenheimer Feld 110, 69120, Heidelberg, Germany. felix.nickel@med.uni-heidelberg.de.
  • Jens Petersen
    Department of Neuroradiology, Heidelberg University Hospital, Heidelberg, Germany; Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany.
  • Susanne M Rafelski
    Allen Institute for Cell Science, Seattle, WA, USA.
  • Nasir Rajpoot
    Department of Computer Science, University of Warwick, Coventry, UK.
  • Mauricio Reyes
    Center for Artificial Intelligence in Medicine, University of Bern, Bern, Switzerland.
  • Michael A Riegler
    SimulaMet, Oslo, Norway.
  • Nicola Rieke
    NVIDIA, Santa Clara, CA, USA.
  • Julio Saez-Rodriguez
    Institute for Computational Biomedicine, Heidelberg University, Faculty of Medicine, Heidelberg University Hospital, Bioquant, Heidelberg, Germany.
  • Clara I Sanchez
  • Shravya Shetty
    Google AI, Mountain View, CA, USA.
  • Ronald M Summers
    National Institutes of Health, Clinical Center, Radiology and Imaging Sciences, 10 Center Drive, Bethesda, MD 20892, USA.
  • Abdel A Taha
    Research Studio Data Science, Research Studios Austria, Salzburg, Austria.
  • Aleksei Tiulpin
    Research Unit of Medical Imaging, Physics and Technology, University of Oulu, Oulu, Finland. aleksei.tiulpin@oulu.fi.
  • Sotirios A Tsaftaris
  • Ben Van Calster
  • Gael Varoquaux
    Parietal, INRIA, NeuroSpin, bat 145 CEA Saclay, 91191, Gif sur Yvette, France.
  • Ziv R Yaniv
    National Institute of Allergy and Infectious Diseases, Bethesda, MD, USA.
  • Paul F Jäger
    Interactive Machine Learning Group (IML), German Cancer Research Center (DKFZ), Heidelberg, Germany.
  • Lena Maier-Hein
    German Cancer Research Center (DKFZ), Computer Assisted Medical Interventions, Heidelberg, Germany.

Keywords

External Resources

Popular Topics
Recent Journals