The RSNA Pediatric Bone Age Machine Learning Challenge.

Journal: Radiology
Published Date: Nov 27, 2018

Abstract

Purpose The Radiological Society of North America (RSNA) Pediatric Bone Age Machine Learning Challenge was created to show an application of machine learning (ML) and artificial intelligence (AI) in medical imaging, promote collaboration to catalyze AI model creation, and identify innovators in medical imaging. Materials and Methods The goal of this challenge was to solicit individuals and teams to create an algorithm or model using ML techniques that would accurately determine skeletal age in a curated data set of pediatric hand radiographs. The primary evaluation measure was the mean absolute distance (MAD) in months, which was calculated as the mean of the absolute values of the difference between the model estimates and those of the reference standard, bone age. Results A data set consisting of 14 236 hand radiographs (12 611 training set, 1425 validation set, 200 test set) was made available to registered challenge participants. A total of 260 individuals or teams registered on the Challenge website. A total of 105 submissions were uploaded from 48 unique users during the training, validation, and test phases. Almost all methods used deep neural network techniques based on one or more convolutional neural networks (CNNs). The best five results based on MAD were 4.2, 4.4, 4.4, 4.5, and 4.5 months, respectively. Conclusion The RSNA Pediatric Bone Age Machine Learning Challenge showed how a coordinated approach to solving a medical imaging problem can be successfully conducted. Future ML challenges will catalyze collaboration and development of ML tools and methods that can potentially improve diagnostic accuracy and patient care. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Siegel in this issue.

Authors

  • Safwan S Halabi
  • Luciano M Prevedello
  • Jayashree Kalpathy-Cramer
    Department of Radiology, MGH/Harvard Medical School, Charlestown, Massachusetts.
  • Artem B Mamonov
    From the Department of Radiology, Stanford University, 300 Pasteur Dr, MC 5105, Stanford, CA 94305 (S.S.H.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital/Harvard Medical School, Boston, Mass (J.K.C.); Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Mass (A.B.M., K.A.); Department of Radiology, University of Toronto, Toronto, Ontario, Canada (A.B.); Department of Radiology, St. Michael's Hospital, Toronto, Ontario, Canada (M.C.); Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, RI (I.P.); Universidade Federal de Goiás, Goiânia, Brazil (L.A.P., R.T.S.); Universidade Federal de São Paulo, São Paulo, Brazil (N.A., F.C.K.); Visiana, Hørsholm, Denmark (H.H.T.); MD.ai, New York, NY (L.C.); Department of Radiology, Weill Cornell Medicine, New York, NY (G.S.) Department of Radiology, University of California-San Francisco, San Francisco, Calif (M.D.K.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • Alexander Bilbily
  • Mark Cicero
    From the *Department of Medical Imaging, St Michael's Hospital, and †Department of Pharmaceutical Sciences, University of Toronto, Toronto, Ontario, Canada.
  • Ian Pan
    Warren Alpert Medical School, Brown University, Providence, RI.
  • Lucas Araújo Pereira
    From the Department of Radiology, Stanford University, 300 Pasteur Dr, MC 5105, Stanford, CA 94305 (S.S.H.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital/Harvard Medical School, Boston, Mass (J.K.C.); Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Mass (A.B.M., K.A.); Department of Radiology, University of Toronto, Toronto, Ontario, Canada (A.B.); Department of Radiology, St. Michael's Hospital, Toronto, Ontario, Canada (M.C.); Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, RI (I.P.); Universidade Federal de Goiás, Goiânia, Brazil (L.A.P., R.T.S.); Universidade Federal de São Paulo, São Paulo, Brazil (N.A., F.C.K.); Visiana, Hørsholm, Denmark (H.H.T.); MD.ai, New York, NY (L.C.); Department of Radiology, Weill Cornell Medicine, New York, NY (G.S.) Department of Radiology, University of California-San Francisco, San Francisco, Calif (M.D.K.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • Rafael Teixeira Sousa
    From the Department of Radiology, Stanford University, 300 Pasteur Dr, MC 5105, Stanford, CA 94305 (S.S.H.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital/Harvard Medical School, Boston, Mass (J.K.C.); Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Mass (A.B.M., K.A.); Department of Radiology, University of Toronto, Toronto, Ontario, Canada (A.B.); Department of Radiology, St. Michael's Hospital, Toronto, Ontario, Canada (M.C.); Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, RI (I.P.); Universidade Federal de Goiás, Goiânia, Brazil (L.A.P., R.T.S.); Universidade Federal de São Paulo, São Paulo, Brazil (N.A., F.C.K.); Visiana, Hørsholm, Denmark (H.H.T.); MD.ai, New York, NY (L.C.); Department of Radiology, Weill Cornell Medicine, New York, NY (G.S.) Department of Radiology, University of California-San Francisco, San Francisco, Calif (M.D.K.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • Nitamar Abdala
    From the Department of Radiology, Stanford University, 300 Pasteur Dr, MC 5105, Stanford, CA 94305 (S.S.H.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital/Harvard Medical School, Boston, Mass (J.K.C.); Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Mass (A.B.M., K.A.); Department of Radiology, University of Toronto, Toronto, Ontario, Canada (A.B.); Department of Radiology, St. Michael's Hospital, Toronto, Ontario, Canada (M.C.); Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, RI (I.P.); Universidade Federal de Goiás, Goiânia, Brazil (L.A.P., R.T.S.); Universidade Federal de São Paulo, São Paulo, Brazil (N.A., F.C.K.); Visiana, Hørsholm, Denmark (H.H.T.); MD.ai, New York, NY (L.C.); Department of Radiology, Weill Cornell Medicine, New York, NY (G.S.) Department of Radiology, University of California-San Francisco, San Francisco, Calif (M.D.K.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • Felipe Campos Kitamura
    From the Department of Radiology, Stanford University, 300 Pasteur Dr, MC 5105, Stanford, CA 94305 (S.S.H.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital/Harvard Medical School, Boston, Mass (J.K.C.); Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Mass (A.B.M., K.A.); Department of Radiology, University of Toronto, Toronto, Ontario, Canada (A.B.); Department of Radiology, St. Michael's Hospital, Toronto, Ontario, Canada (M.C.); Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, RI (I.P.); Universidade Federal de Goiás, Goiânia, Brazil (L.A.P., R.T.S.); Universidade Federal de São Paulo, São Paulo, Brazil (N.A., F.C.K.); Visiana, Hørsholm, Denmark (H.H.T.); MD.ai, New York, NY (L.C.); Department of Radiology, Weill Cornell Medicine, New York, NY (G.S.) Department of Radiology, University of California-San Francisco, San Francisco, Calif (M.D.K.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • Hans H Thodberg
    From the Department of Radiology, Stanford University, 300 Pasteur Dr, MC 5105, Stanford, CA 94305 (S.S.H.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital/Harvard Medical School, Boston, Mass (J.K.C.); Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Mass (A.B.M., K.A.); Department of Radiology, University of Toronto, Toronto, Ontario, Canada (A.B.); Department of Radiology, St. Michael's Hospital, Toronto, Ontario, Canada (M.C.); Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, RI (I.P.); Universidade Federal de Goiás, Goiânia, Brazil (L.A.P., R.T.S.); Universidade Federal de São Paulo, São Paulo, Brazil (N.A., F.C.K.); Visiana, Hørsholm, Denmark (H.H.T.); MD.ai, New York, NY (L.C.); Department of Radiology, Weill Cornell Medicine, New York, NY (G.S.) Department of Radiology, University of California-San Francisco, San Francisco, Calif (M.D.K.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • Leon Chen
    From the Department of Radiology, Stanford University, 300 Pasteur Dr, MC 5105, Stanford, CA 94305 (S.S.H.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital/Harvard Medical School, Boston, Mass (J.K.C.); Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Mass (A.B.M., K.A.); Department of Radiology, University of Toronto, Toronto, Ontario, Canada (A.B.); Department of Radiology, St. Michael's Hospital, Toronto, Ontario, Canada (M.C.); Department of Diagnostic Imaging, Warren Alpert Medical School of Brown University, Rhode Island Hospital, Providence, RI (I.P.); Universidade Federal de Goiás, Goiânia, Brazil (L.A.P., R.T.S.); Universidade Federal de São Paulo, São Paulo, Brazil (N.A., F.C.K.); Visiana, Hørsholm, Denmark (H.H.T.); MD.ai, New York, NY (L.C.); Department of Radiology, Weill Cornell Medicine, New York, NY (G.S.) Department of Radiology, University of California-San Francisco, San Francisco, Calif (M.D.K.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); and Department of Radiology, Thomas Jefferson University, Philadelphia, Pa (A.E.F.).
  • George Shih
  • Katherine Andriole
    Brigham and Women's Hospital, Boston MA, USA.
  • Marc D Kohli
    Radiology and Biomedical Imaging, 505 Parnassus Ave, Moffit-391, San Francisco, CA, 94117, USA. marc.kohli@ucsf.edu.
  • Bradley J Erickson
    Department of Radiology, Radiology Informatics Lab, Mayo Clinic, Rochester, MN 55905, United States.
  • Adam E Flanders

Keywords

External Resources

Popular Topics
Recent Journals