Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial.

Journal: Radiology
Published Date: Dec 1, 2021

Abstract

Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with ( = 792) and without ( = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, = .02) and 24 months (0.5% vs 1.8%, = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 See also the editorial by Rubin in this issue.

Authors

  • David K Eng
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Nishith B Khandwala
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Jin Long
    Center for Artificial Intelligence in Medicine and Imaging, Stanford University, 1701 Page Mill Road, Palo Alto, CA, 94304, USA.
  • Nancy R Fefferman
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Shailee V Lala
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Naomi A Strubel
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Sarah S Milla
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Ross W Filice
    MedStar Health, MedStar Georgetown University Hospital, 3800 Reservoir Rd, NW CG201, Washington DC, 20007 (R.W.F.); and MedStar Health, National Center for Human Factors in Healthcare, Washington, DC (R.M.R.).
  • Susan E Sharp
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Alexander J Towbin
    Department of Radiology and Medical Imaging, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio.
  • Michael L Francavilla
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Summer L Kaplan
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Kirsten Ecklund
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Sanjay P Prabhu
    Dr Sanjay P Prabhu Department of Radiology, Boston Children's Hospital and Harvard Medical School, Boston, MA, 02115, USA. Electronic address: sanjay.prabhu@childrens.harvard.edu.
  • Brian J Dillon
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Brian M Everist
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Christopher G Anton
    From the Department of Radiology, Cincinnati Children's Hospital Medical Center, 3333, Burnet Ave, Cincinnati, OH 45329; and Department of Radiology, University of Cincinnati Medical School, Cincinnati, Ohio.
  • Mark E Bittman
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Rebecca Dennis
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • David B Larson
    Department of Radiology, Warren Alpert Medical School, Brown University, 593 Eddy St, Providence, RI 02903 (I.P.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (I.P.); Visiana, Hørsholm, Denmark (H.H.T.); Department of Radiology, Stanford University, Palo Alto, Calif (S.S.H., D.B.L.); and Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.).
  • Jayne M Seekins
    Department of Radiology, Stanford University School of Medicine, 725 Welch Road, Stanford, CA 94305, USA.
  • Cicero T Silva
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Arash R Zandieh
    From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.).
  • Curtis P Langlotz
    Stanford University, University Medical Line, Stanford, CA, 94305, US.
  • Matthew P Lungren
  • Safwan S Halabi

Keywords

External Resources

Popular Topics
Recent Journals