Comparing physician and artificial intelligence chatbot responses to preterm infant care questions posted to a public medical consultation forum: evaluation study.

OBJECTIVES: This study aims to compare the preterm infant care advice generated by ChatGPT with responses from neonatologists in an online medical consultation forum. STUDY DESIGN: This cross-sectional study involving 22 evaluators (12 experienced neonatologists and 10 lay users) compared the responses of an AI chatbot (ChatGPT 4o) and online human experts to 60 preterm infant care-related questions from the "Doctor DingXiang" platform accessed in August 2024. Quantitative indicators, including readability, helpfulness, understandability, intent capture, empathy, actionability, accuracy and safety, were used for comparison. Qualitative content analyses were conducted to reveal further more detailed situational information about the differences between them. RESULTS: The mean[SD] score for ChatGPT was higher than human experts on the dimensions of helpfulness (4.07 [0.78] vs 3.50 [1.04]), understandability (4.12 [0.73] vs 3.70 [1.01]), intent capture (4.12 [0.75] vs 3.62 [1.01]), empathy (3.98 [0.73] vs 3.45 [1.05]), actionability (3.97 [0.83] vs 3.37 [1.11]), accuracy (3.97 [0.73] vs 3.32 [1.01]) and safety (3.96 [0.72] vs 3.37 [0.97]). (all P <.001) ChatGPT's responses had few instances of low ratings than human experts' responses. ChatGPT provided more comprehensive and elaborative information, though its responses exhibited higher semantic richness and were more difficult to read. In contrast, human experts were better at offering personalized advice and emotional support, adjusting responses based on specific contexts. CONCLUSION: ChatGPT can provide satisfactory responses to neonatal medical inquiries. However, further improvements are necessary to enhance the precision, decision support, and contextual awareness.