The importance of clinical context in evaluating algorithmic fairness: insights from a medication adherence prediction algorithm.

OBJECTIVE: Using AI algorithms can exacerbate health disparities if care or resources are allocated away from underserved populations. We evaluated an algorithm for its potential to worsen health disparities across different clinical use cases. MATERIALS AND METHODS: This was a retrospective study of patients with heart failure (HF) at an academic health system using an algorithm that predicts pharmacy fill nonadherence to evidence-based HF medications. We compared prediction performance metrics (accuracy, false positive rate, false negative rate), using rate-ratios (RRs), between subgroups with and without known HF care disparities: below vs above median neighborhood-level socioeconomic status (nSES) and Black vs White race. Results were then applied to 3 hypothetical clinical use cases. RESULTS: Among 34 697 patients (13% Black, 10% Hispanic, 65% White), algorithm accuracy was similar across nSES and racial subgroups. The algorithm assigned more false positives for medication nonadherence among low vs high nSES (RR [95%CI] 1.50 [1.44-1.56]) and Black vs White (2.05 [1.92-2.19]) subgroups. The algorithm also assigned fewer false negatives (0.63 [0.59-0.67]) to Black vs White subgroups. When applied to 3 hypothetical use cases, worsening of existing disparities was pertinent for clinical applications where false positives could be particularly harmful (e.g, if predictions of nonadherence prompted lower treatment priority). DISCUSSION: Although accuracy was similar across demographic groups, differences in false positive and false negative rates revealed that the same prediction may worsen disparities in some use cases, but not others. CONCLUSION: Evaluation of predictions in the context of clinical use is essential to avoid unintentionally worsening inequities.