Development and Evaluation of Artificial Intelligence-Assisted Decision Support System for Public Health Emergency Classification and Escalation in Kenya
Journal:
medRxiv
Published Date:
Jul 10, 2026
Abstract
Background Timely assessment, classification, and escalation of public health events are essential for effective outbreak response, yet decision-making after event detection remains challenging because of fragmented guidance and variable interpretation of escalation criteria.To strengthen public health emergency management, Kenya developed the Decision-Making Tool for Public Health Emergencies (DMT-PHE), a framework for event assessment, classification, notification, and escalation. An artificial intelligence (AI)-enabled version, the DMT-PHE AI Agent, was subsequently developed to operationalize the framework through decision support. This study describes the development of the DMT-PHE AI Agent and evaluates its performance, usability, safety, and user acceptability. Methods The DMT-PHE AI Agent was developed using a retrieval-augmented generation architecture supported by a curated knowledge base derived from the validated DMT-PHE framework and related public health guidance. A simulation-based pilot evaluation was conducted among 11 public health professionals who independently assessed three standardized outbreak scenarios. AI-generated recommendations were compared with expert-defined gold standards. Outcomes included concordance, response-action coverage, citation performance, safety, usability, and user acceptability. Results Thirty-three scenario evaluations were completed. The AI Agent achieved an overall weighted concordance score of 0.924, with exact agreement of 90.9% for Public Health Events of Initially Unknown Etiology, 81.8% for Rift Valley fever, and 90.9% for Mpox. Citation support was provided in 78.8% of interactions, with no incorrect citations or major safety concerns identified. The mean System Usability Scale score was 85.2, while participants reported high trust (4.27/5), contextual relevance (4.55/5), and perceived time savings (4.82/5). Conclusions The DMT-PHE AI Agent demonstrated that a nationally validated public health emergency decision framework can be successfully translated into an AI-enabled decision-support system. These findings provide early evidence that AI can augment public health emergency decision-making by delivering structured, transparent, and context-specific recommendations while maintaining human oversight, offering a practical model for operationalizing national public health guidance.
