Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines.

Journal: Nature communications
PMID: 38388497

Abstract

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77-94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.

Authors

  • Alexander P L Martindale
    Brighton and Sussex Medical School, Brighton, UK.
  • Carrie D Llewellyn
    Department of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, UK.
  • Richard O de Visser
    Department of Primary Care and Public Health, Brighton and Sussex Medical School, Brighton, UK.
  • Benjamin Ng
    Birmingham and Midland Eye Centre, Sandwell and West Birmingham NHS Trust, Birmingham, UK.
  • Victoria Ngai
    University College London Medical School, London, UK.
  • Aditya U Kale
    Department of Ophthalmology, University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK.
  • Lavinia Ferrante di Ruffano
    York Health Economics Consortium, University of York, Birmingham, UK.
  • Robert M Golub
    Journal of the American Medical Association, Chicago, IL, USA.
  • Gary S Collins
    Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Botnar Research Centre, University of Oxford, Windmill Road, Oxford, OX3 7LD UK; Oxford University Hospitals NHS Foundation Trust, Oxford, UK.
  • David Moher
    Centre for Journalology Clinical Epidemiology Program Ottawa Hospital Research Institute Ottawa Canadá Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canadá.
  • Melissa D McCradden
    Division of Neurosurgery (McCradden, Baba, Saha, Boparai, Fadaiefard, Cusimano), St. Michael's Hospital, Unity Health Toronto; Dalla Lana School of Public Health (Cusimano), University of Toronto, Toronto, Ont. injuryprevention@smh.ca.
  • Lauren Oakden-Rayner
    School of Public Health, University of Adelaide, Adelaide, SA, Australia; Australian Institute for Machine Learning, University of Adelaide, Adelaide, SA, Australia. Electronic address: lauren.oakden-rayner@adelaide.edu.au.
  • Samantha Cruz Rivera
    Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK.
  • Melanie Calvert
    Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK.
  • Christopher J Kelly
    Google Health, London, UK. cjkelly@google.com.
  • Cecilia S Lee
    Department of Ophthalmology, University of Washington, Seattle, Washington, USA.
  • Christopher Yau
    Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK.
  • An-Wen Chan
    Department of Medicine, Women's College Research Institute Women's College Hospital University of Toronto Ontario Canadá Department of Medicine, Women's College Research Institute, Women's College Hospital, University of Toronto, Ontario, Canadá.
  • Pearse A Keane
    National Institute for Health Research Biomedical Research Centre for Ophthalmology, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK.
  • Andrew L Beam
    Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
  • Alastair K Denniston
    Centre for Patient Reported Outcomes Research Institute of Applied Health Research University of Birmingham Birmingham Reino Unido Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, Reino Unido.
  • Xiaoxuan Liu
    Birmingham Health Partners Centre for Regulatory Science and Innovation University of Birmingham Birmingham Reino Unido Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, Reino Unido.

Keywords

External Resources

Popular Topics
Recent Journals