Characterizing the Impact of Training Data on Generalizability: Application in Deep Learning to Estimate Lung Nodule Malignancy Risk.

Journal: Radiology. Artificial intelligence
Published Date: Aug 20, 2025

Abstract

. Purpose To investigate the relationship between training data volume and performance of a deep learning AI algorithm developed to assess the malignancy risk of pulmonary nodules detected on low-dose CT scans in lung cancer screening. Materials and Methods This retrospective study used a dataset of 16077 annotated nodules (1249 malignant, 14828 benign) from the National Lung Screening Trial (NLST) to systematically train an AI algorithm for pulmonary nodule malignancy risk prediction across various stratified subsets ranging from 1.25% to the full dataset. External testing was conducted using data from the Danish Lung Cancer Screening Trial (DLCST) to determine the amount of training data at which the performance of the AI was statistically non-inferior to the AI trained on the full NLST cohort. A size-matched cancer-enriched subset of DLCST, where each malignant nodule had been paired in diameter with the closest two benign nodules, was used to investigate the amount of training data at which the performance of the AI algorithm was statistically non-inferior to the average performance of 11 clinicians. Results The external testing set included 599 participants (mean age 57.65 (SD 4.84) for females and mean age 59.03 (SD 4.94) for males) with 883 nodules (65 malignant, 818 benign). The AI achieved a mean AUC of 0.92 [95% CI: 0.88, 0.96] on the DLCST cohort when trained on the full NLST dataset. Training with 80% of NLST data resulted in non-inferior performance (mean AUC 0.92 [95%CI: 0.89, 0.96], = .005). On the size-matched DLCST subset (59 malignant, 118 benign), the AI reached non-inferior clinician-level performance (mean AUC 0.82 [95% CI: 0.77, 0.86]) with 20% of the training data ( = .02). Conclusion The deep learning AI algorithm demonstrated excellent performance in assessing pulmonary nodule malignancy risk, achieving clinical level performance with a fraction of the training data and reaching peak performance before utilizing the full dataset. ©RSNA, 2025.

Authors

  • Bogdan Obreja
    Department of Medical Imaging, Radboud University Medical Center, Geert Grooteplein Zuid 10, 525 GA Nijmegen, the Netherlands.
  • Joeran Bosma
    Diagnostic Image Analysis Group, Department of Medical Imaging, Radboud University Medical Center, Nijmegen, 6525 GA, The Netherlands.
  • Kiran Vaidhya Venkadesh
    From the Department of Radiology, Nuclear Medicine and Anatomy, Radboud University Medical Center, P.O. Box 9101, 6500 HB Nijmegen, the Netherlands (N.L., C.I.S., L.H.B., M.B., E.C., W.M.v.E., P.K.G., B.G., M.G., N.H., W.H., H.J.H., C.J., R.K., M.K., K.v.L., J.M., M.O., R.S., C. Schaefer-Prokop, S.S., E.T.S., C. Sital, J.T., K.V.V., C.d.V., W.X., B.d.W., M.P., B.v.G.); Department of Radiology, Academic Medical Center, Amsterdam, the Netherlands (L.B.); Thirona, Nijmegen, the Netherlands (J.P.C., E.M.v.R.); Departments of Internal Medicine (T.D.) and Radiology (M.V.), Canisius-Wilhelmina Ziekenhuis, Nijmegen, the Netherlands; Department of Radiology and Nuclear Medicine, Maastricht University Medical Center, Maastricht, the Netherlands (H.A.G.); GROW School of Oncology and Developmental Biology, Maastricht, the Netherlands (H.A.G.); Departments of Biomedical Physics and Engineering and Radiology and Nuclear Medicine, Amsterdam University Medical Center, Amsterdam, the Netherlands (L.v.H., I.I.); Department of Radiology, Zuyderland Medical Center, Heerlen, the Netherlands (J.K.); Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany (B.L.); Department of Radiology and Nuclear Medicine, Haaglanden Medical Center, The Hague, the Netherlands (T.v.R.V.); Department of Radiology, Meander Medical Center, Amersfoort, the Netherlands (C. Schaefer-Prokop, S.S.); and Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands (J.L.S.).
  • Zaigham Saghir
  • Mathias Prokop
    Department of Radiology, Radboud University Medical Center, Nijmegen, The Netherlands.
  • Colin Jacobs
    Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals