Disparities in medical recommendations from AI-based chatbots across different countries/regions.

Journal: Scientific reports
PMID: 39048640

Abstract

This study explores disparities and opportunities in healthcare information provided by AI chatbots. We focused on recommendations for adjuvant therapy in endometrial cancer, analyzing responses across four regions (Indonesia, Nigeria, Taiwan, USA) and three platforms (Bard, Bing, ChatGPT-3.5). Utilizing previously published cases, we asked identical questions to chatbots from each location within a 24-h window. Responses were evaluated in a double-blinded manner on relevance, clarity, depth, focus, and coherence by ten experts in endometrial cancer. Our analysis revealed significant variations across different countries/regions (p < 0.001). Interestingly, Bing's responses in Nigeria consistently outperformed others (p < 0.05), excelling in all evaluation criteria (p < 0.001). Bard also performed better in Nigeria compared to other regions (p < 0.05), consistently surpassing them across all categories (p < 0.001, with relevance reaching p < 0.01). Notably, Bard's overall scores were significantly higher than those of ChatGPT-3.5 and Bing in all locations (p < 0.001). These findings highlight disparities and opportunities in the quality of AI-powered healthcare information based on user location and platform. This emphasizes the necessity for more research and development to guarantee equal access to trustworthy medical information through AI technologies.

Authors

  • Khanisyah E Gumilar
    Graduate Institute of Biomedical Science, China Medical University, Taichung, Taiwan. khanisyah@fk.unair.ac.id.
  • Birama R Indraprasta
    Department of Obstetrics and Gynecology, Dr. Soetomo General Hospital - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Yu-Cheng Hsu
    Department of Public Health, China Medical University, Taichung, Taiwan.
  • Zih-Ying Yu
    Department of Public Health, China Medical University, Taichung, Taiwan.
  • Hong Chen
    Department of Nephrology, The Second Affiliated Hospital of Guangxi Medical University, Nanning, Guangxi, China.
  • Budi Irawan
    Department of Obstetrics and Gynecology, Dr. Soetomo General Hospital - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Zulkarnain Tambunan
    Department of Obstetrics and Gynecology, Dr. Soetomo General Hospital - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Bagus M Wibowo
    Department of Obstetrics and Gynecology, Dr. Soetomo General Hospital - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Hari Nugroho
    Department of Obstetrics and Gynecology, Dr. Soetomo General Hospital - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Brahmana A Tjokroprawiro
    Department of Obstetrics and Gynecology, Dr. Soetomo General Hospital - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Erry G Dachlan
    Department of Obstetrics and Gynecology, Dr. Soetomo General Hospital-Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Pungky Mulawardhana
    Department of Obstetrics and Gynecology, Hospital of Universitas Airlangga - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Eccita Rahestyningtyas
    Department of Obstetrics and Gynecology, Hospital of Universitas Airlangga - Faculty of Medicine, Universitas Airlangga, Surabaya, Indonesia.
  • Herlangga Pramuditya
    Department of Obstetrics and Gynecology, Dr. Ramelan Naval Hospital, Surabaya, Indonesia.
  • Very Great E Putra
    Department of Obstetrics and Gynecology, Dr. Kariadi Central General Hospital, Semarang, Indonesia.
  • Setyo T Waluyo
    Department of Obstetrics and Gynecology, Ulin General Hospital, Banjarmasin, Indonesia.
  • Nathan R Tan
    Department of Modern and Classical Languages and Literature, University of South Alabama, Mobile, AL, USA.
  • Royhaan Folarin
    Department of Anatomy, Faculty of Basic Medical Sciences, Olabisi Onabanjo University, Sagamu, Nigeria.
  • Ibrahim H Ibrahim
    Graduate Institute of Biomedical Science, China Medical University, Taichung, Taiwan.
  • Cheng-Han Lin
    Graduate Institute of Biomedical Science, China Medical University, Taichung, Taiwan.
  • Tai-Yu Hung
    Graduate Institute of Biomedical Science, China Medical University, Taichung, Taiwan.
  • Ting-Fang Lu
    Department of Obstetrics and Gynecology, Taichung Veteran General Hospital, 1650 Taiwan Boulevard Sector. 4, Taichung, 40705, Taiwan, ROC.
  • Yen-Fu Chen
    Department of Obstetrics and Gynecology, Taichung Veteran General Hospital, 1650 Taiwan Boulevard Sector. 4, Taichung, 40705, Taiwan, ROC.
  • Yu-Hsiang Shih
    Department of Obstetrics and Gynecology, Taichung Veteran General Hospital, 1650 Taiwan Boulevard Sector. 4, Taichung, 40705, Taiwan, ROC.
  • Shao-Jing Wang
    Department of Obstetrics and Gynecology, Taichung Veteran General Hospital, 1650 Taiwan Boulevard Sector. 4, Taichung, 40705, Taiwan, ROC.
  • Jingshan Huang
    School of Computing, College of Medicine, University of South Alabama, Mobile, AL, USA.
  • Clayton C Yates
    Department of Pathology, Johns Hopkins University School of Medicine, Baltimore, MD, 21287, USA.
  • Chien-Hsing Lu
    Department of Obstetrics and Gynecology, Taichung Veteran General Hospital, Taichung, Taiwan.
  • Li-Na Liao
    Department of Public Health, China Medical University, Taichung, Taiwan.
  • Ming Tan
    Graduate Institute of Biomedical Science, China Medical University, Taichung, Taiwan.

Keywords

External Resources

Popular Topics
Recent Journals