Conformal Prediction and Venn-ABERS Calibration for Reliable Machine Learning-Based Prediction of Bacterial Infection Focus

Finding the focus of bacterial infections can be challenging, especially for hospitalised patients. Conventional microbiological diagnostic methods are either time consuming, expensive, or difficult to interpret due to contamination. The aim of this study was to apply machine learning (ML) to reliably predict the focus of bacterial infections. This study utilised a dataset including samples from 10,153 patients, collected from November 1, 2019, to June 3, 2023, at Rigshospitalet, Denmark. The dataset contains microbiological findings, biochemical data, and vital parameters. The dataset was analysed using ML. The ML-outputs were calibrated using Venn-ABERS calibration and model uncertainty was addressed using conformal risk control. The best performing model was the XGBoost model achieving a Log loss of 0.219 ± 0.050 (mean ± SD.) and an AUC of 0.93 ± 0.051. Combining the model with methods from the conformal prediction framework achieves predictive capabilities that surpasses similar studies, while also accounting for model uncertainty by providing statistically robust uncertainty estimates via calibrated probabilistic predictions.