Enhancing AI reliability: A foundation model with uncertainty estimation for optical coherence tomography-based retinal disease diagnosis.

Journal: Cell reports. Medicine
PMID: 39706192

Abstract

Inability to express the confidence level and detect unseen disease classes limits the clinical implementation of artificial intelligence in the real world. We develop a foundation model with uncertainty estimation (FMUE) to detect 16 retinal conditions on optical coherence tomography (OCT). In the internal test set, FMUE achieves a higher F1 score of 95.74% than other state-of-the-art algorithms (92.03%-93.66%) and improves to 97.44% with threshold strategy. The model achieves similar excellent performance on two external test sets from the same and different OCT machines. In human-model comparison, FMUE achieves a higher F1 score of 96.30% than retinal experts (86.95%, p = 0.004), senior doctors (82.71%, p < 0.001), junior doctors (66.55%, p < 0.001), and generative pretrained transformer 4 with vision (GPT-4V) (32.39%, p < 0.001). Besides, FMUE predicts high uncertainty scores for >85% images of non-target-category diseases or with low quality to prompt manual checks and prevent misdiagnosis. Our FMUE provides a trustworthy method for automatic retinal anomaly detection in a clinical open-set environment.

Authors

  • Yuanyuan Peng
    School of Electronics and Information Engineering, Soochow University, Suzhou, China.
  • Aidi Lin
    Department of Women's Oncology, Shuangyu Campus, Wenzhou Central Hospital, Wenzhou, Zhejiang, China.
  • Meng Wang
    State Key Laboratory of Urban Water Resource and Environment, School of Environment, Harbin Institute of Technology, Harbin 150001, China.
  • Tian Lin
    Guangxi Medical University, Nanning Guangxi, 530021, P.R.China.
  • Linna Liu
    Department of Laboratory Medicine, The Second Affiliated Hospital and Yuying Children's Hospital of Wenzhou Medical University, Wenzhou, Zhejiang 325000, China.
  • Jianhua Wu
  • Ke Zou
    National Key Laboratory of Fundamental Science on Synthetic Vision and the College of Computer Science, Sichuan University, Chengdu, Sichuan 610065, China.
  • Tingkun Shi
    Joint Shantou International Eye Centre of Shantou University and The Chinese University of Hong Kong, Shantou, Guangdong, China.
  • Lixia Feng
    Department of Ophthalmology, First Affiliated Hospital of Anhui Medical University, Hefei, Anhui, China.
  • Zhen Liang
    Graduate School of Informatics, Kyoto University, Kyoto 606-8501, Japan; School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, 518060, China. Electronic address: jane-l@sys.i.kyoto-u.ac.jp.
  • Tao Li
    Department of Emergency Medicine, Jining No.1 People's Hospital, Jining, China.
  • Dan Liang
    First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, People's Republic of China (D.L.); Department of Radiology, Guangzhou First People's Hospital, School of Medicine, South China University of Technology, Guangzhou, Guangdong, People's Republic of China (D.L., Y.L., D.C., A.C., J.D., X.W.).
  • Shanshan Yu
    State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Centre, Sun Yat-sen University, Guangzhou, Guangdong, China.
  • Dawei Sun
    The Affiliated Hospital of Qingdao University, PR China.
  • Jing Luo
    Department of Ophthalmology, The Second Xiangya Hospital of Central South University, 139 Middle Renmin RD, Changsha, Hunan, China.
  • Ling Gao
  • Xinjian Chen
    Medical Image Processing, Analysis, and Visualization (MIVAP) Lab, School of Electronics and Information Engineering, Soochow University, Suzhou, China.
  • Ching-Yu Cheng
    Singapore National Eye Centre, Singapore Eye Research Institute, Singapore, Singapore.
  • Huazhu Fu
    A*STAR, Singapore, Singapore.
  • Haoyu Chen
    Joint Shantou International Eye Center, Shantou University and the Chinese University of Hong Kong, Shantou, China.

Keywords

External Resources

Popular Topics
Recent Journals