Assessing Quality, Readability, and Transparency of Online and AI-Generated Information on Type 2 Diabetes Mellitus: Cross-Sectional Exploratory Study.

BACKGROUND: Type 2 diabetes mellitus (T2DM) affects approximately 590 million people worldwide, and its management relies heavily on patient education. With the emergence of online health information and artificial intelligence (AI) large language models, patients are increasingly sourcing medical information independently. OBJECTIVE: This study compared the quality, readability, and transparency of websites and AI-generated leaflets (AIGLs) related to T2DM. METHODS: Four predefined search terms ("type 2 diabetes," "type 2 diabetes mellitus," "T2DM," and "adult diabetes") were entered into 3 major search engines (Google, Yahoo, and Bing), and the top 20 search results were retrieved. AIGLs with patient information about T2DM were produced using a standardized prompt in 4 AI large language models (ChatGPT, Gemini, DeepSeek, and Grok). Information quality was assessed using the DISCERN score, calculated by 3 independent raters and ChatGPT. The Journal of the American Medical Association (JAMA) benchmarks were used to measure reliability and transparency. The Flesch-Kincaid Grade Level was used to determine readability. RESULTS: Seventy-five websites and 4 AIGLs were evaluated. Mean author-rated DISCERN scores were 42.6 (SD 11.3) for websites and 43.9 (SD 1.74) for AIGLs, corresponding to fair quality (DISCERN 41-51). In contrast, ChatGPT-rated mean DISCERN scores were higher, with 58.5 (SD 11.5) for websites and 61.0 (SD 2.94) for AIGLs, corresponding to good quality (DISCERN 52-63). Mean JAMA benchmark scores were 2.74 (SD 0.965) for websites, indicating moderate reliability (2-3 out of 4 points), whereas all AIGLs scored 0 out of 4 points. Mean Flesch-Kincaid Grade Level scores for websites were 8.67 (SD 2.23) and 8.30 (SD 1.92) for AIGLs, corresponding to an eighth- to ninth-grade comprehension level. Spearman rank correlation demonstrated minimal variability among the 3 independent raters but showed a significant difference between ChatGPT and the 3 independent raters. CONCLUSIONS: Given the high prevalence of T2DM, both websites and AIGLs demonstrated suboptimal quality, readability, and transparency. Increasing patient reliance on digital health information calls for improved readability standards and stronger safeguards for AI-generated content. Both websites and AIGLs require an eighth- to ninth-grade comprehension level, far above the average reading age of 9 years in the United Kingdom (fourth- to fifth-grade level). This reduces the accessibility of online health information. The landscape of medical consultations is evolving, with patients increasingly presenting with preconceived notions based on online health information; hence, health care professionals should adapt to this shift.