Monitoring strategies for continuous evaluation of deployed clinical prediction models.
Journal:
Journal of biomedical informatics
Published Date:
Jun 5, 2025
Abstract
OBJECTIVE: As machine learning adoption in clinical practice continues to grow, deployed classifiers must be continuously monitored and updated (retrained) to protect against data drift that stems from inevitable changes, including evolving medical practices and shifting patient populations. However, successful clinical machine learning classifiers will lead to a change in care which may change the distribution of features, labels, and their relationship. For example, "high risk" cases that were correctly identified by the model may ultimately get labeled as "low risk" thanks to an intervention prompted by the model's alert. Classifier surveillance systems naive to such deployment-induced feedback loops will estimate lower model performance and lead to degraded future classifier retrains. The objective of this study is to simulate the impact of these feedback loops, propose feedback aware monitoring strategies as a solution, and assess the performance of these alternative monitoring strategies through simulations.
