Managing AI-Enabled Uncertainty in Clinical AI Deployment: Mixed-Methods Study of Governance, Workflow, and Organizational Learning in an ICU Decision Support Pilot
Journal:
medRxiv
Published Date:
Jul 2, 2026
Abstract
BackgroundHealth care organizations are increasingly required to make strategic decisions about artificial intelligence (AI) systems before their clinical value, operational consequences, governance requirements, and workforce implications are fully known. Clinical AI pilots can reduce this uncertainty only if they generate management-relevant evidence beyond retrospective model performance, including evidence on workflow fit, governance architecture, user interaction, accountability, and continuous learning.
ObjectiveThis study aimed to examine how a prospective intensive care unit (ICU) deployment of an AI-based clinical decision support system (CDSS) surfaced organizational and governance-related uncertainties, and to derive a transferable management toolbox for early-stage clinical AI deployment.
MethodsWe conducted a prospective, single-center mixed-methods implementation evaluation of a machine-learning-based CDSS for ICU length-of-stay prediction in a surgical ICU at a German university hospital. The study included 267 consecutive ICU stays and was approved by the LMU Munich Institutional Review Board (Project No. 24-0336) and registered with the German Clinical Trials Register (DRKS00037851). The evaluation combined five management-relevant domains: workflow integration and operational adoption, governance architecture and data-protection effects, live bedside benchmarking as a continuous learning mechanism, embedded ethics and accountability design, and human-AI interaction with user heterogeneity assessed using the Psychological Assessment of AI-based Decision Support Systems instrument.
ResultsThe deployment showed that AI-enabled uncertainty was generated primarily at the interface between the CDSS and its organizational setting. A low-burden "glance-and-judge" workflow enabled routine use among consultants and resident physicians, with usage proportions of 72% (191/267) and 61% (162/267), respectively. Governance requirements led to an air-gapped architecture with once-daily data refreshes, enabling compliant use but creating operational latency; outdated case-list entries occurred in 4 of 148 benchmarked stays (2.7%). Live benchmarking functioned as a feedback loop and supported exploratory model refinement, with CDSS mean absolute error decreasing from 5.95 to 4.12 days after the first model update. Embedded ethics informed onboarding, accountability framing, interface wording, and explainability design. Human-AI interaction assessment suggested heterogeneous user archetypes with distinct change-management needs.
ConclusionsClinical AI deployment should be managed as an organizational learning process rather than as a purely technical implementation. The relevant management object is the deployed sociotechnical system, including workflow, governance architecture, feedback loops, ethics, and user interaction. We propose a Governance-Aware AI Management toolbox that helps administrators and digital health officers distinguish solvable implementation-design uncertainty from true lack of clinical value, and supports staged, evidence-based decisions about scaling, investment, and definitive evaluation.
