Recommendations for the development and use of imaging test sets to investigate the test performance of artificial intelligence in health screening.

Journal: The Lancet. Digital health
Published Date:

Abstract

Rigorous evaluation of artificial intelligence (AI) systems for image classification is essential before deployment into health-care settings, such as screening programmes, so that adoption is effective and safe. A key step in the evaluation process is the external validation of diagnostic performance using a test set of images. We conducted a rapid literature review on methods to develop test sets, published from 2012 to 2020, in English. Using thematic analysis, we mapped themes and coded the principles using the Population, Intervention, and Comparator or Reference standard, Outcome, and Study design framework. A group of screening and AI experts assessed the evidence-based principles for completeness and provided further considerations. From the final 15 principles recommended here, five affect population, one intervention, two comparator, one reference standard, and one both reference standard and comparator. Finally, four are appliable to outcome and one to study design. Principles from the literature were useful to address biases from AI; however, they did not account for screening specific biases, which we now incorporate. The principles set out here should be used to support the development and use of test sets for studies that assess the accuracy of AI within screening programmes, to ensure they are fit for purpose and minimise bias.

Authors

  • Anastasia Chalkidou
    King's Technology Evaluation Centre, King's College London, London, UK. Electronic address: anastasia.chalkidou@nice.org.uk.
  • Farhad Shokraneh
    University of Nottingham, Nottingham, UK.
  • Goda Kijauskaite
    UK National Screening Committee, Office for Health Improvement and Disparities, Department of Health and Social Care, London, UK.
  • Sian Taylor-Phillips
    Division of Health Sciences, University of Warwick, Coventry, UK S.Taylor-Phillips@warwick.ac.uk.
  • Steve Halligan
    Centre for Medical Imaging, Division of Medicine, University College London, London, UK.
  • Louise Wilkinson
    Oxford Breast Imaging Centre, Oxford University, Oxford, UK.
  • Ben Glocker
    Kheiron Medical Technologies, London, UK.
  • Peter Garrett
    Department of Chemical Engineering and Analytical Science, University of Manchester, Manchester, UK.
  • Alastair K Denniston
    Centre for Patient Reported Outcomes Research Institute of Applied Health Research University of Birmingham Birmingham Reino Unido Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, Reino Unido.
  • Anne Mackie
    UK National Screening Committee, Office for Health Improvement and Disparities, Department of Health and Social Care, London, UK.
  • Farah Seedat
    UK National Screening Committee, Office for Health Improvement and Disparities, Department of Health and Social Care, London, UK.