Multitask Deep Learning for Segmentation and Classification of Primary Bone Tumors on Radiographs.

Journal: Radiology
Published Date: Sep 7, 2021

Abstract

Background An artificial intelligence model that assesses primary bone tumors on radiographs may assist in the diagnostic workflow. Purpose To develop a multitask deep learning (DL) model for simultaneous bounding box placement, segmentation, and classification of primary bone tumors on radiographs. Materials and Methods This retrospective study analyzed bone tumors on radiographs acquired prior to treatment and obtained from patient data from January 2000 to June 2020. Benign or malignant bone tumors were diagnosed in all patients by using the histopathologic findings as the reference standard. By using split-sample validation, 70% of the patients were assigned to the training set, 15% were assigned to the validation set, and 15% were assigned to the test set. The final performance was evaluated on an external test set by using geographic validation, with accuracy, sensitivity, specificity, and 95% CIs being used for classification, the intersection over union (IoU) being used for bounding box placements, and the Dice score being used for segmentations. Results Radiographs from 934 patients (mean age, 33 years ± 19 [standard deviation]; 419 women) were evaluated in the internal data set, which included 667 benign bone tumors and 267 malignant bone tumors. Six hundred fifty-four patients were in the training set, 140 were in the validation set, and 140 were in the test set. One hundred eleven patients were in the external test set. The multitask DL model achieved 80.2% (89 of 111; 95% CI: 72.8, 87.6) accuracy, 62.9% (22 of 35; 95% CI: 47, 79) sensitivity, and 88.2% (67 of 76; CI: 81, 96) specificity in the classification of bone tumors as malignant or benign. The model achieved an IoU of 0.52 ± 0.34 for bounding box placements and a mean Dice score of 0.60 ± 0.37 for segmentations. The model accuracy was higher than that of two radiologic residents (71.2% and 64.9%; = .002 and < .001, respectively) and was comparable with that of two musculoskeletal fellowship-trained radiologists (83.8% and 82.9%; = .13 and = .25, respectively) in classifying a tumor as malignant or benign. Conclusion The developed multitask deep learning model allowed for accurate and simultaneous bounding box placement, segmentation, and classification of primary bone tumors on radiographs. © RSNA, 2021 See also the editorial by Carrino in this issue.

Authors

  • Claudio E von Schacky
    From the Department of Radiology and Biomedical Imaging (C.E.v.S., J.H.S., E.O., P.M.J., M.P., S.C.F., T.M.L., V.P.) and Department of Epidemiology and Biostatistics (F.L., M.C.N.), University of California, San Francisco, 185 Berry St, Suite 350, San Francisco, CA 94107; Department of Diagnostic and Interventional Radiology, Technische Universität München, Munich, Germany (C.E.v.S., S.C.F.); Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (P.M.J.); and Department of Radiology, University of California Davis Health, Sacramento, Calif (L.N.).
  • Nikolas J Wilhelm
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Valerie S Schäfer
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Yannik Leonhardt
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Felix G Gassert
    Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany.
  • Sarah C Foreman
    From the Department of Radiology and Biomedical Imaging (C.E.v.S., J.H.S., E.O., P.M.J., M.P., S.C.F., T.M.L., V.P.) and Department of Epidemiology and Biostatistics (F.L., M.C.N.), University of California, San Francisco, 185 Berry St, Suite 350, San Francisco, CA 94107; Department of Diagnostic and Interventional Radiology, Technische Universität München, Munich, Germany (C.E.v.S., S.C.F.); Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (P.M.J.); and Department of Radiology, University of California Davis Health, Sacramento, Calif (L.N.).
  • Florian T Gassert
    Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany.
  • Matthias Jung
    Department of Electrical Engineering and Computer Science, University of Siegen, Hölderlinstr. 3, Siegen, Germany.
  • Pia M Jungmann
    From the Department of Radiology and Biomedical Imaging (C.E.v.S., J.H.S., E.O., P.M.J., M.P., S.C.F., T.M.L., V.P.) and Department of Epidemiology and Biostatistics (F.L., M.C.N.), University of California, San Francisco, 185 Berry St, Suite 350, San Francisco, CA 94107; Department of Diagnostic and Interventional Radiology, Technische Universität München, Munich, Germany (C.E.v.S., S.C.F.); Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (P.M.J.); and Department of Radiology, University of California Davis Health, Sacramento, Calif (L.N.).
  • Maximilian F Russe
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Carolin Mogler
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Carolin Knebel
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Rüdiger von Eisenhart-Rothe
    Klinik und Poliklinik für Orthopädie und Sportorthopädie, Klinikum rechts der Isar der TU München, München, Deutschland.
  • Marcus R Makowski
    School of Medicine and Health, Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, TUM University Hospital, Technical University of Munich, Munich, Germany.
  • Klaus Woertler
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Rainer Burgkart
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).
  • Alexandra S Gersing
    From the Department of Radiology (C.E.v.S., V.S.S., Y.L., F.G.G., S.C.F., F.T.G., M.R.M., K.W., A.S.G.), Department for Orthopedics and Orthopedic Sports Medicine (N.J.W., C.K., R.v.E., R.B.), and Institute of Pathology (C.M.), Klinikum Rechts der Isar, Technische Universität München, Ismaninger Str 22, 81675 Munich, Germany; and the Department of Diagnostic and Interventional Radiology, Medical Center-University of Freiburg, Faculty of Medicine, Freiburg, Germany (M.J., P.M.J., M.F.R.).

Keywords

External Resources

Popular Topics
Recent Journals