Performance across different versions of an artificial intelligence model for screen-reading of mammograms.

Journal: European radiology
Published Date:

Abstract

OBJECTIVES: Studies have reported promising results regarding artificial intelligence (AI) as a tool for improved mammographic screening interpretive performance. We analyzed AI malignancy risk scores from two versions of the same commercial AI model. MATERIALS AND METHODS: This retrospective cohort study used data from 117,709 screening examinations performed in BreastScreen Norway 2009-2018. The mammograms were processed by two versions of the commercially available AI model, Transpara (version 1.7 and 2.1). The distributions of exam-level risk scores (AI score 1-10) and risk categories were evaluated for both AI versions on all examinations, including 737 screen-detected and 200 interval cancers. Scores between 1-7 were categorized as low risk, 8-9 as intermediate risk, and 10 as high risk of malignancy. RESULTS: Area under the receiver operating curve was 0.908 (95% CI: 0.986-0.920) for version 1.7 and 0.928 (95% CI: 0.917-0.939) for 2.1 when screen-detected and interval cancers were considered as positive cases (p < 0.001). A total of 87.1% (642/737) and 93.5% (689/737) of the screen-detected cancers had an AI score of 10 with version 1.7 and 2.1, respectively. Among interval cancers, 45.0% (90/200) had AI score 10 with version 1.7 and 44.5% (89/200) had AI score 10 with version 2.1. CONCLUSION: A higher proportion of screen-detected breast cancers had the highest AI score of 10 with the newer version of the AI model compared to the older version. For interval cancers, there was no difference in the proportion of cases assigned to the highest score between the two versions. KEY POINTS: Question Studies have reported promising results regarding the use of AI in mammography screening, but comparisons of updated versus older versions are less studied. Findings In our study, 87.1% (642/737) of the screen-detected cancers were classified with a high malignancy risk score by the old version, while it was 93.5% (689/737) for the newer version. Clinical relevance Understanding how version updates of AI models might impact screening mammography performance will be important for future quality assurance and validation of AI models.

Authors

  • Marthe Larsen
    Section for Breast Cancer Screening, Cancer Registry of Norway, Oslo, Norway.
  • Christoph I Lee
    Department of Radiology, University of Washington School of Medicine, Seattle, WA, USA.
  • Marie B Bergan
    Section for Breast Cancer Screening, Cancer Registry of Norway, Norwegian Institute of Public Health, PO Box 5313, 0304, Oslo, Norway.
  • Åsne S Holen
    Section for Breast Cancer Screening, Cancer Registry of Norway, Oslo, Norway.
  • Håkon Lund-Hanssen
    From the Section for Breast Cancer Screening (M.L., C.F.A., S.H.) and Department of Register Informatics (J.F.N.), Cancer Registry of Norway (G.U.), P.O. Box 5313, 0304 Oslo, Norway; Department of Health and Care Sciences, Faculty of Health Sciences, The Arctic University of Norway, Tromsø, Norway (S.H.); Department of Radiology, University of Washington School of Medicine, Seattle, Wash (C.I.L.); Department of Health Systems and Population Health, University of Washington School of Public Health, Seattle, Wash (C.I.L.); Department of Radiology, Ålesund Hospital, Møre og Romsdal Hospital Trust, Ålesund, Norway (S.R.H.); Department of Circulation and Medical Imaging, Faculty of Medicine and Health Sciences, National University for Science and Technology, Trondheim, Norway (S.R.H.); Department of Radiology and Nuclear Medicine, St Olavs University Hospital, Trondheim, Norway (H.L.H.); Department of Translational Medicine, Lund University, Lund, Sweden (K.L.); and Unilabs Mammography Unit, Skåne University Hospital, Malmö, Sweden (K.L.).
  • Solveig R Hoff
    From the Section for Breast Cancer Screening (M.L., C.F.A., S.H.) and Department of Register Informatics (J.F.N.), Cancer Registry of Norway (G.U.), P.O. Box 5313, 0304 Oslo, Norway; Department of Health and Care Sciences, Faculty of Health Sciences, The Arctic University of Norway, Tromsø, Norway (S.H.); Department of Radiology, University of Washington School of Medicine, Seattle, Wash (C.I.L.); Department of Health Systems and Population Health, University of Washington School of Public Health, Seattle, Wash (C.I.L.); Department of Radiology, Ålesund Hospital, Møre og Romsdal Hospital Trust, Ålesund, Norway (S.R.H.); Department of Circulation and Medical Imaging, Faculty of Medicine and Health Sciences, National University for Science and Technology, Trondheim, Norway (S.R.H.); Department of Radiology and Nuclear Medicine, St Olavs University Hospital, Trondheim, Norway (H.L.H.); Department of Translational Medicine, Lund University, Lund, Sweden (K.L.); and Unilabs Mammography Unit, Skåne University Hospital, Malmö, Sweden (K.L.).
  • Steinar Auensen
    From the Section for Breast Cancer Screening (M.L., C.F.O., S.H.) and Department of Register Informatics (S.A., J.F.N.), Cancer Registry of Norway, Norwegian Institute of Public Health, PO 5313, Majorstuen, 0304 Oslo, Norway; Department of Radiology, University of Washington School of Medicine, Seattle, Wash (C.I.L.); Department of Health Systems and Population Health, University of Washington School of Public Health, Seattle, Wash (C.I.L.); Department of Radiology, Vestre Viken Hospital Trust, Drammen, Norway (T.H.); Department of Radiology, Ålesund Hospital, Møre og Romsdal Hospital Trust, Ålesund, Norway (S.R.H.); Department of Circulation, Medical Imaging, Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology, Trondheim, Norway (S.R.H.); Department of Radiology, Østfold Hospital Trust, Kalnes, Norway (M.A.M.); Institute of Clinical Medicine, University of Oslo, Oslo, Norway (M.A.M.); SPKI-The Norwegian Centre for Clinical Artificial Intelligence, University Hospital of North Norway, Tromsø, Norway (K.Ø.M.); Department of Clinical Medicine, Faculty of Health Sciences (K.Ø.M.), Department of Physics and Technology, Faculty of Science and Technology (J.F.N.), and Department of Health and Care Sciences, Faculty of Health Sciences (S.H.), UiT-The Arctic University of Norway, Tromsø, Norway; Department of Radiology and Nuclear Medicine, St Olavs University Hospital, Trondheim, Norway (H.L.H.); Department of Radiology, Hospital of Southern Norway, Kristiansand, Norway (H.S.S.); Department of Radiology, Innlandet Hospital Trust, Hamar, Norway (M.S.); and Department of Radiology, Innlandet Hospital Trust, Lillehammer, Norway (Å.Ø.S.).
  • Jan F Nygård
    From the Section for Breast Cancer Screening (M.L., C.F.A., S.H.) and Department of Register Informatics (J.F.N.), Cancer Registry of Norway (G.U.), P.O. Box 5313, 0304 Oslo, Norway; Department of Health and Care Sciences, Faculty of Health Sciences, The Arctic University of Norway, Tromsø, Norway (S.H.); Department of Radiology, University of Washington School of Medicine, Seattle, Wash (C.I.L.); Department of Health Systems and Population Health, University of Washington School of Public Health, Seattle, Wash (C.I.L.); Department of Radiology, Ålesund Hospital, Møre og Romsdal Hospital Trust, Ålesund, Norway (S.R.H.); Department of Circulation and Medical Imaging, Faculty of Medicine and Health Sciences, National University for Science and Technology, Trondheim, Norway (S.R.H.); Department of Radiology and Nuclear Medicine, St Olavs University Hospital, Trondheim, Norway (H.L.H.); Department of Translational Medicine, Lund University, Lund, Sweden (K.L.); and Unilabs Mammography Unit, Skåne University Hospital, Malmö, Sweden (K.L.).
  • Kristina Lång
    Institute for Biomedical Engineering, ETH Zurich, Gloriastrasse 35, 8092, Zürich, Switzerland.
  • Yan Chen
    Department of Respiratory and Critical Care Medicine, Shanghai Pulmonary Hospital, School of Medicine, Tongji University, Shanghai, China.
  • Giske Ursin
    From the Section for Breast Cancer Screening (M.L., C.F.A., S.H.) and Department of Register Informatics (J.F.N.), Cancer Registry of Norway (G.U.), P.O. Box 5313, 0304 Oslo, Norway; Department of Health and Care Sciences, Faculty of Health Sciences, The Arctic University of Norway, Tromsø, Norway (S.H.); Department of Radiology, University of Washington School of Medicine, Seattle, Wash (C.I.L.); Department of Health Systems and Population Health, University of Washington School of Public Health, Seattle, Wash (C.I.L.); Department of Radiology, Ålesund Hospital, Møre og Romsdal Hospital Trust, Ålesund, Norway (S.R.H.); Department of Circulation and Medical Imaging, Faculty of Medicine and Health Sciences, National University for Science and Technology, Trondheim, Norway (S.R.H.); Department of Radiology and Nuclear Medicine, St Olavs University Hospital, Trondheim, Norway (H.L.H.); Department of Translational Medicine, Lund University, Lund, Sweden (K.L.); and Unilabs Mammography Unit, Skåne University Hospital, Malmö, Sweden (K.L.).
  • Solveig Hofvind
    Section for Breast Cancer Screening, Cancer Registry of Norway, Oslo, Norway.

Keywords

No keywords available for this article.