Integrated image-based deep learning and language models for primary diabetes care.

Journal: Nature medicine
Published Date: Jul 19, 2024

Abstract

Primary diabetes care and diabetic retinopathy (DR) screening persist as major public health challenges due to a shortage of trained primary care physicians (PCPs), particularly in low-resource settings. Here, to bridge the gaps, we developed an integrated image-language system (DeepDR-LLM), combining a large language model (LLM module) and image-based deep learning (DeepDR-Transformer), to provide individualized diabetes management recommendations to PCPs. In a retrospective evaluation, the LLM module demonstrated comparable performance to PCPs and endocrinology residents when tested in English and outperformed PCPs and had comparable performance to endocrinology residents in Chinese. For identifying referable DR, the average PCP's accuracy was 81.0% unassisted and 92.3% assisted by DeepDR-Transformer. Furthermore, we performed a single-center real-world prospective study, deploying DeepDR-LLM. We compared diabetes management adherence of patients under the unassisted PCP arm (n = 397) with those under the PCP+DeepDR-LLM arm (n = 372). Patients with newly diagnosed diabetes in the PCP+DeepDR-LLM arm showed better self-management behaviors throughout follow-up (P < 0.05). For patients with referral DR, those in the PCP+DeepDR-LLM arm were more likely to adhere to DR referrals (P < 0.01). Additionally, DeepDR-LLM deployment improved the quality and empathy level of management recommendations. Given its multifaceted performance, DeepDR-LLM holds promise as a digital solution for enhancing primary diabetes care and DR screening.

Authors

  • Jiajia Li
    Shanghai Artificial Intelligence Research Institute Co., Ltd, Shanghai, China.
  • Zhouyu Guan
    Shanghai International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China.
  • Jing Wang
    Endoscopy Center, Peking University Cancer Hospital and Institute, Beijing, China.
  • Carol Y Cheung
    Department of Ophthalmology and Visual Sciences, Chinese University of Hong Kong, Hong Kong Special Administrative Region, China. Electronic address: carolcheung@cuhk.edu.hk.
  • Yingfeng Zheng
    State Key Laboratory of Ophthalmology, Clinical Research Center for Ocular Disease, Zhongshan Ophthalmic Centre, Sun Yat-sen University, Guangzhou, China.
  • Lee-Ling Lim
    Department of Medicine, Faculty of Medicine, University of Malaya, Kuala Lumpur, Malaysia.
  • Cynthia Ciwei Lim
    Department of Renal Medicine, Singapore General Hospital, 168753, Singapore.
  • Paisan Ruamviboonsuk
    Rajavithi Hospital, Bangkok, Thailand.
  • Rajiv Raman
    Shri Bhagwan Mahavir Vitreoretinal Services, Sankara Nethralaya, Chennai, Tamil Nadu, India.
  • Leonor Corsino
    Department of Medicine, Division of Endocrinology, Metabolism and Nutrition, and Department of Population Health Sciences, Duke University School of Medicine, Durham, NC, USA.
  • Justin B Echouffo-Tcheugui
    Department of Medicine, Division of Endocrinology, Diabetes and Metabolism, Johns Hopkins School of Medicine, Baltimore, MD, USA.
  • Andrea O Y Luk
    Department of Medicine and Therapeutics, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China.
  • Li Jia Chen
    Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong SAR.
  • Xiaodong Sun
    Shanghai Key Laboratory of Ocular Fundus Diseases, Shanghai General Hospital, Shanghai JiaoTong University, 200080 Shanghai, China.
  • Haslina Hamzah
    Singapore Eye Research Institute, Singapore National Eye Center, Singapore.
  • Qiang Wu
    Department of Radiotherapy, West China Hospital, Sichuan University, Chengdu, PR China; Lung Cancer Center, West China Hospital, Sichuan University, Chengdu, PR China.
  • Xiangning Wang
    Department of Ophthalmology, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai, 200233, China.
  • Ruhan Liu
    Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China.
  • Ya Xing Wang
    Beijing Visual Science and Translational Eye Research Institute (BERI), Eye Center of Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University, Beijing, China.
  • Tingli Chen
    Department of Ophthalmology, Huadong Sanatorium, Wuxi, China.
  • Xiao Zhang
    Merck & Co., Inc., Rahway, NJ, USA.
  • Xiaolong Yang
    Department of Ophthalmology, Huadong Sanatorium, Wuxi, China.
  • Jun Yin
    Institute of Surgery Research, Daping Hospital, Third Military Medical University, Chongqing, 400042, China.
  • Jing Wan
    State Key Laboratory of ASIC and System, School of Information Science and Technology, Fudan University, Shanghai 200433, China.
  • Wei Du
    Department of Respiratory and Critical Care Medicine, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China.
  • Ten Cheer Quek
    Singapore Eye Research Institute, Singapore National Eye Centre, Singapore, Singapore.
  • Jocelyn Hui Lin Goh
    Centre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore and National University Health System, Singapore.
  • Dawei Yang
    Department of Pulmonary and Critical Care Medicine, Zhongshan Hospital Fudan University, Shanghai, China.
  • Xiaoyan Hu
    Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China.
  • Truong X Nguyen
    Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China.
  • Simon K H Szeto
    Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China.
  • Peranut Chotcomwongse
    Vitreoretina Unit, Department of Ophthalmology, Rajavithi Hospital, Rungsit University, Bangkok, Thailand.
  • Rachid Malek
    Department of Internal Medicine, Setif University Ferhat Abbas, Setif, Algeria.
  • Nargiza Normatova
    Ophthalmology Department at Tashkent Advanced Training Institute for Doctors, Tashkent, Uzbekistan.
  • Nilufar Ibragimova
    Charity Union of Persons with Disabilities and People with Diabetes UMID, Tashkent, Uzbekistan.
  • Ramyaa Srinivasan
    Shri Bhagwan Mahavir Vitreoretinal Services, Medical Research Foundation, Sankara Nethralaya, Chennai, India.
  • Pingting Zhong
    Guangdong Eye Institute, Department of Ophthalmology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences, the Second School of Clinical Medicine, Southern Medical University, Guangzhou, China.
  • Wenyong Huang
    State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangdong Provincial Key Laboratory of Ophthalmology and Vision Science, Guangdong Provincial Clinical Research Center for Ocular Diseases, Guangzhou, China.
  • Chenxin Deng
    Department of Geriatrics, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
  • Lei Ruan
    Department of Geriatrics, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
  • Cuntai Zhang
    Department of Geriatrics, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
  • Chenxi Zhang
    Department of Ophthalmology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences, No. 1 Shuaifuyuan Wangfujing, Dongcheng District, Beijing, 100730, China.
  • Yan Zhou
    Department of Computer Science, University of Texas at Dallas, Richardson, Texas 75080, United States.
  • Chan Wu
    Department of Ophthalmology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences, Beijing, China.
  • Rongping Dai
    Department of Ophthalmology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences, and Peking Union Medical College, Beijing, China.
  • Sky Wei Chee Koh
    National University Polyclinics, National University Health System, Department of Family Medicine, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
  • Adina Abdullah
    Department of Primary Care Medicine, Faculty of Medicine, Universiti Malaya, Kuala Lumpur, Malaysia.
  • Nicholas Ken Yoong Hee
    Department of Medicine, University Malaya Medical Centre, Kuala Lumpur, Malaysia.
  • Hong Chang Tan
    Department of Endocrinology, Singapore General Hospital, Singapore, Singapore.
  • Zhong Hong Liew
    Department of Renal Medicine, Singapore General Hospital, SingHealth-Duke Academic Medical Centre, Singapore, Singapore.
  • Carolyn Shan-Yeu Tien
    Department of Renal Medicine, Singapore General Hospital, SingHealth-Duke Academic Medical Centre, Singapore, Singapore.
  • Shih Ling Kao
    Division of Endocrinology, University Medicine Cluster, National University Health System, Singapore, Singapore.
  • Amanda Yuan Ling Lim
    Division of Endocrinology, University Medicine Cluster, National University Health System, Singapore, Singapore.
  • Shao Feng Mok
    Division of Endocrinology, University Medicine Cluster, National University Health System, Singapore, Singapore.
  • Lina Sun
    State Key Laboratory for Molecular Virology and Genetic Engineering, National Institute for Viral Disease Control and Prevention, Chinese Center for Disease Control and Prevention, Beijing 102206, China.
  • Jing Gu
    Department of Epidemiology and Medical Statistics, School of Public Health, Sun Yat-sen University, Guangzhou, China.
  • Liang Wu
    Clinical and Research Center of AIDS, Beijing Ditan Hospital, Capital Medical University, Beijing, China.
  • Tingyao Li
    Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, China.
  • Di Cheng
    State Key Laboratory of Bioreactor Engineering, Shanghai Key Laboratory of Chemical Biology, School of Pharmacy, East China University of Science and Technology, 130 Meilong Road, Shanghai, 200237, PR China.
  • Zheyuan Wang
    Shanghai International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China; MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China.
  • Yiming Qin
    Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, China.
  • Ling Dai
    Department of Computer Science and Engineering, Shanghai Jiao Tong University, China; MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China.
  • Ziyao Meng
    Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, China.
  • Jia Shu
    College of Life Science, Northwest A&F University, 712100, Shaanxi, Yangling, China.
  • Yuwei Lu
    Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, China.
  • Nan Jiang
  • Tingting Hu
    People's Hospital of Deyang City, Deyang, 618000, Sichuan, China.
  • Shan Huang
    Tianjin Key Laboratory of Ionic-Molecular Function of Cardiovascular Disease, Department of Cardiology, Tianjin Institute of Cardiology, Second Hospital of Tianjin Medical University, 300211 Tianjin, China.
  • Gengyou Huang
    Shanghai Belt and Road International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai, China.
  • Shujie Yu
    Shanghai International Joint Laboratory of Intelligent Prevention and Treatment for Metabolic Diseases, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China.
  • Dan Liu
    Department of Bioengineering, Temple University, Philadelphia, PA, United States.
  • Weizhi Ma
    Institure for AI Industry Research, Tsinghua University, Beijing, China.
  • Minyi Guo
  • Xinping Guan
    Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China. Electronic address: xpguan@sjtu.edu.cn.
  • Xiaokang Yang
  • Covadonga Bascaran
    International Centre for Eye Health, London School of Hygiene and Tropical Medicine, University of London, London, UK.
  • Charles R Cleland
    International Centre for Eye Health, London School of Hygiene and Tropical Medicine, University of London, London, UK.
  • Yuqian Bao
    Department of Endocrinology and Metabolism, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai Clinical Center for Diabetes, Shanghai Key Laboratory of Diabetes Mellitus, Shanghai Diabetes Institute Shanghai 200233, China.
  • Elif I Ekinci
    Department of Endocrinology, Austin Health, Melbourne, Victoria, Australia.
  • Alicia Jenkins
    Australian Centre for Accelerating Diabetes Innovations, The University of Melbourne, Parkville, Victoria, Australia.
  • Juliana C N Chan
    Department of Medicine and Therapeutics, Prince of Wales Hospital, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China.
  • Yong Mong Bee
    1Department of Endocrinology, Singapore General Hospital, Singapore.
  • Sobha Sivaprasad
    Moorfields Eye Hospital City Road Campus, London, UK.
  • Jonathan E Shaw
    Department of Medicine, The University of Melbourne (Austin Health), Melbourne, Victoria, Australia.
  • Rafael Simó
    Endocrinology and Nutrition Department, Hospital Universitari Vall D´Hebron, Barcelona, Spain.
  • Pearse A Keane
    National Institute for Health Research Biomedical Research Centre for Ophthalmology, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK.
  • Ching-Yu Cheng
    Singapore National Eye Centre, Singapore Eye Research Institute, Singapore, Singapore.
  • Gavin Siew Wei Tan
    Singapore Eye Research Institute, Singapore National Eye Center, Singapore.
  • Weiping Jia
    Department of Endocrinology and Metabolism, Shanghai Jiao Tong University Affiliated Sixth People's Hospital, Shanghai Clinical Center for Diabetes, Shanghai Key Laboratory of Diabetes Mellitus, Shanghai Diabetes Institute Shanghai 200233, China.
  • Yih-Chung Tham
    Singapore Eye Research Institute, Singapore National Eye Centre, Singapore; Ophthalmology and Visual Sciences Academic Clinical Program (Eye ACP), Duke-NUS Medical School, Singapore.
  • Huating Li
    Shanghai Jiao Tong University Affiliated Sixth People's Hospital, China. Electronic address: huarting99@sjtu.edu.cn.
  • Bin Sheng
    MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China.
  • Tien Yin Wong
    Singapore National Eye Center, Duke-National University of Singapore Medical School, Singapore 168751, Singapore; National Institutes of Health Research Biomedical Research Centre Biomedical Centre, Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, London, UK.

Keywords

External Resources

Popular Topics
Recent Journals