Assessing workflow impact and clinical utility of AI-assisted brain aneurysm detection: a multi-reader study
Journal:
arXiv
Published Date:
Mar 22, 2025
Abstract
Despite the plethora of AI-based algorithms developed for anomaly detection
in radiology, subsequent integration into clinical setting is rarely evaluated.
In this work, we assess the applicability and utility of an AI-based model for
brain aneurysm detection comparing the performance of two readers with
different levels of experience (2 and 13 years). We aim to answer the following
questions: 1) Do the readers improve their performance when assisted by the AI
algorithm? 2) How much does the AI algorithm impact routine clinical workflow?
We reuse and enlarge our open-access, Time-Of-Flight Magnetic Resonance
Angiography dataset (N=460). We use 360 subjects for training/validating our
algorithm and 100 as unseen test set for the reading session. Even though our
model reaches state-of-the-art results on the test set (sensitivity=74%, false
positive rate=1.6), we show that neither the junior nor the senior reader
significantly increase their sensitivity (p=0.59, p=1, respectively). In
addition, we find that reading time for both readers is significantly higher in
the "AI-assisted" setting than in the "Unassisted" (+15 seconds, on average;
p=3x10^(-4) junior, p=3x10^(-5) senior). The confidence reported by the readers
is unchanged across the two settings, indicating that the AI assistance does
not influence the certainty of the diagnosis. Our findings highlight the
importance of clinical validation of AI algorithms in a clinical setting
involving radiologists. This study should serve as a reminder to the community
to always examine the real-word effectiveness and workflow impact of proposed
algorithms.
