Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges.

Journal: Radiology. Artificial intelligence
PMID:

Abstract

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Authors

  • Felipe C Kitamura
  • Luciano M Prevedello
  • Errol Colak
    Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, ON, Canada.
  • Safwan S Halabi
  • Matthew P Lungren
  • Robyn L Ball
    From the Department of Radiology, Stanford University School of Medicine, Stanford University Medical Center, 725 Welch Rd, Room 1675, Stanford, Calif 94305-5913 (M.C.C., N.M., D.B.L., C.P.L., M.P.L.); Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, Calif (R.L.B., L.Y.); Department of Bioinformatics, University of Utah Medical Center, Salt Lake City, Utah (B.E.C.); and Department of Radiology, Duke University Medical Center, Durham, NC (T.J.A.).
  • Jayashree Kalpathy-Cramer
    Department of Radiology, MGH/Harvard Medical School, Charlestown, Massachusetts.
  • Charles E Kahn
    Perelman School of Medicine, University of Pennsylvania, Philadelphia 19104, USA.
  • Tyler Richards
    Department of Radiology and Imaging Sciences, University of Utah School of Medicine, Salt Lake City, Utah.
  • Jason F Talbott
    From the Department of Radiology and Biomedical Imaging and Brain and Spinal Injury Center, University of California, San Francisco, San Francisco, Calif; and Department of Radiology and Biomedical Imaging, Zuckerberg San Francisco General Hospital and Trauma Center, 1001 Potrero Ave, Bldg 5, Room 1X57C, San Francisco, CA 94110.
  • George Shih
  • Hui Ming Lin
    Department of Medical Imaging, St. Michael's Hospital, Unity Health Toronto, Toronto, Canada.
  • Katherine P Andriole
    Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, Mass (W.F.W., M.T.C., K.M., S.A.G., E.G., M.H.R., G.C.G., K.P.A.); and MGH & BWH Center for Clinical Data Science, Boston, Mass (W.F.W., M.T.C., K.M., K.P.A.).
  • Maryam Vazirabad
    From the Department of Applied Innovation and AI, Dasa, São Paulo, Brazil (F.C.K.); Department of Diagnostic Imaging, Universidade Federal de São Paulo (Unifesp), Av Prof Ascendino Reis, 1245, 131, São Paulo, SP, Brazil 04027-000 (F.C.K.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Department of Medical Imaging, University of Toronto, Toronto, Canada (E.C.); Ann and Robert H. Lurie Children's Hospital of Chicago, Chicago, Ill (S.S.H.); Microsoft HLS, Redmond, Wash (M.P.L.); Department of Biomedical Data Science, Stanford University, Stanford, Calif (M.P.L.); The Jackson Laboratory, Bar Harbor, Maine (R.L.B.); Department of Ophthalmology, University of Colorado Denver School of Medicine, Aurora, Colo (J.K.C.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (C.E.K.); Department of Radiology, University of Utah, Salt Lake City, Utah (T.R.); Department of Radiology and Biomedical Imaging (M.P.L., J.F.T., J.M.) and Center for Intelligent Imaging (J.M.), University of California San Francisco, San Francisco, Calif; Department of Radiology, Weill Cornell Medical College, New York, NY (G.S.); Department of Medical Imaging, Unity Health Toronto, Toronto, Canada (H.M.L.); Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, MGB Data Science Office, Boston, Mass (K.P.A.); Informatics Department, Radiological Society of North America, Oak Brook, Ill (M.V.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • Bradley J Erickson
    Department of Radiology, Radiology Informatics Lab, Mayo Clinic, Rochester, MN 55905, United States.
  • Adam E Flanders
  • John Mongan
    From the Departments of Urology (T.C., M.U., H.C.C., M.S.) and Radiology and Biomedical Imaging (J.M., M.P.K., A.T., P.J., R.G., S.W.), University of California, San Francisco. 505 Parnassus Ave, M-391, San Francisco, CA 94143; and Division of Urology, Faculty of Medicine, King Chulalongkorn Memorial Hospital, Chulalongkorn University, The Thai Red Cross Society, Bangkok, Thailand (M.U.).