Automated Real-time Assessment of Intracranial Hemorrhage Detection AI Using an Ensembled Monitoring Model (EMM).

Journal: Research square
Published Date: May 26, 2025

Abstract

Artificial intelligence (AI) tools for radiology are commonly unmonitored once deployed. The lack of real-time case-by-case assessments of AI prediction confidence requires users to independently distinguish between trustworthy and unreliable AI predictions, which increases cognitive burden, reduces productivity, and potentially leads to misdiagnoses. To address these challenges, we introduce Ensembled Monitoring Model (EMM), a framework inspired by clinical consensus practices using multiple expert reviews. Designed specifically for black-box commercial AI products, EMM operates independently without requiring access to internal AI components or intermediate outputs, while still providing robust confidence measurements. Using intracranial hemorrhage detection as our test case on a large, diverse dataset of 2919 studies, we demonstrate that EMM successfully categorizes confidence in the AI-generated prediction, suggesting different actions and helping improve the overall performance of AI tools to ultimately reduce cognitive burden. Importantly, we provide key technical considerations and best practices for successfully translating EMM into clinical settings.

Authors

  • Zhongnan Fang
    LVIS Corporation, Palo Alto, California.
  • Andrew Johnston
  • Lina Cheuy
  • Hye Sun Na
    Department of Radiology, Stanford University School of Medicine, 453 Quarry Rd, MC 5659, Stanford, CA 94304.
  • Magdalini Paschali
    Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA.
  • Camila Gonzalez
    Informatics, TU Darmstadt, Germany.
  • Bonnie A Armstrong
  • Arogya Koirala
    Department of Radiology, Stanford University School of Medicine, 453 Quarry Rd, MC 5659, Stanford, CA 94304.
  • Derrick Laurel
  • Andrew Walker Campion
  • Michael Iv
    Department of Radiology, Stanford University, Stanford, California, USA.
  • Akshay S Chaudhari
    Department of Radiology, Stanford University, Stanford, California.
  • David B Larson
    Department of Radiology, Warren Alpert Medical School, Brown University, 593 Eddy St, Providence, RI 02903 (I.P.); Department of Diagnostic Imaging, Rhode Island Hospital, Providence, RI (I.P.); Visiana, Hørsholm, Denmark (H.H.T.); Department of Radiology, Stanford University, Palo Alto, Calif (S.S.H., D.B.L.); and Department of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.).

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals