Assessment of the accuracy and readability of artificial generative intelligence for patient questions on heavy menstrual bleeding.

OBJECTIVES: To evaluate the quality, patient-centeredness, clinician endorsement, and readability of generative artificial intelligence (AI) responses to patient questions about heavy menstrual bleeding (HMB) compared with high-quality patient-facing websites. METHODS: A cross-sectional study compared responses generated by ChatGPT and Google Gemini to excerpts from the five highest-quality HMB patient resources identified through a Google Trends-informed search and the QUality Evaluation Scoring Tool (QUEST). Five layperson-style questions representing HMB subtopics (definition, causes, investigations, management, and safety) were submitted to each model. Responses were de-identified and independently evaluated by five gynecologists, blinded to source, using 5-point Likert scales for accuracy/comprehensiveness (quality), empathetic and validating communication style (patient-centeredness), and expert comfort with patient use of the resource to guide understanding (clinician endorsement). Readability was assessed using the Flesch-Kincaid Grade Level (FKGL). Inter-rater reliability was measured using intraclass correlation coefficients (ICCs). RESULTS: Thirty-six responses were reviewed (16 AI-generated and 20 web-based). Compared with web-based excerpts, AI responses showed similar quality (3.94/5±0.70 vs. 3.53/5±1.21; p=0.21), lower patient-centeredness (2.81/5±0.82 vs. 3.52/5±1.13; p=0.04), and comparable clinician endorsement (3.56/5±0.92 vs. 3.44/5±1.35; p=0.52). Without impacting content quality, AI responses were written at a lower grade level (7.6±2.4) than web-based excerpts (11.1±5.2; p=0.042). Inter-rater reliability was high (ICC=0.82-0.84). CONCLUSIONS: Compared with high-quality web-based resources, AI-generated responses to HMB questions were more inclusive for varying health literacy levels, comparable in quality and clinician endorsement, but performed worse in patient-centeredness. In an evolving digital landscape for health information acquisition, AI-generated responses represent a valuable and safe supplementary or first-line resource for patients.