Can co-designed educational interventions help consumers think critically about asking ChatGPT health questions? Results from a randomised-controlled trial

Generative artificial intelligence (AI) tools have clear potential health benefits for individuals (e.g. simplifying information) and risks (e.g. inaccurate information). To evaluate two brief co-designed health literacy interventions to help people critically reflect on health questions they ask ChatGPT. Three arm parallel-group randomised controlled trial Australian adults without university education who had used ChatGPT in the past 6 months, and recruited via online social research panel. (1) animation intervention, (2) image-based intervention, or (3) control. The primary outcomes were intention to ask ChatGPT a question in ‘lower risk’ and ‘higher risk’ scenarios, where higher risk scenarios would typically require clinical interpretation. Secondary outcomes were ChatGPT knowledge, trust in ChatGPT’s responses, and intervention acceptability. Of the 619 participants, 592 were included in the analysis sample. Average age was 47.0 years (SD=16.4), 42.6% identified as man or male, and 17.4% had limited/marginal health literacy. Participants in the animation group (n=191) reported lower intention to use ChatGPT for higher risk scenarios (M=2.42/5, 95%CI: 2.27 to 2.56) compared to those in the images group (n=203, M=2.69/5, 95%CI: 2.54 to 2.83, p=0.010). Participants in both intervention groups reported lower intentions to use ChatGPT for higher risk scenarios compared to control (n=205, M=3.12/5, 95%CI: 2.98 to 3.27, p<0.001). There was no effect of intervention group on intention to use ChatGPT for lower risk scenarios (p=0.800). Participants in the intervention groups had higher knowledge of ChatGPT (p<0.001) and reported lower trust (p<0.001), compared to those in the control group. Brief health literacy interventions may help improve knowledge of ChatGPT and reduce intentions to ask riskier health questions. This study represents an initial step towards addressing AI health literacy and highlights the kinds of health literacy skills that can support people to navigate AI tools safely.