Considering counterfactual prediction models in sleep research: A demonstration of incident stroke prediction in an observational cohort study.

OBJECTIVE: Data-driven risk prediction algorithms are increasingly common. However, studies suggesting that a single set of variables are most "important" within a given risk prediction model ignore the reality that other equally useful models containing entirely different features may exist. We aim to use a novel approach (Model Path Selection) to demonstrate that, for any given health outcome, there are likely many comparable counterfactual models including different sets of variables. We focus on incident stroke as a scientifically meaningful outcome to motivate this demonstration. METHODS: Model Path Selection uses an iterative model-building procedure to illustrate a set of counterfactual prediction models ("paths"). We applied Model Path Selection to Sleep Heart Health Study data (N = 4,512 adults) to predict 5-year incident stroke, considering 30 features comprising demographics, self-reported and measured medical risk factors, polysomnography, and self-reported sleep. Area Under the Curve was estimated for each resulting path based on 10-fold cross-validation. RESULTS: Model Path Selection identified 61 unique counterfactual models. 13 of these achieved Area Under the Curves between 0.81 and 0.82. Area Under the Curves of the remaining models ranged from 0.53 to 0.68 (median 0.61). All 13 top models included age plus various combinations of other features (polysomnography indices, body mass index, and/or the Short Form 36 mental and physical component scores). CONCLUSIONS: Model Path Selection can illuminate counterfactual prediction models, differentiate features that may be uniquely predictive from those that can be swapped out with other features, enhance reproducibility, inform the design of clinically practical predictive tools, and suggest sleep-related risk factors to be targeted for health promotion.