Self-improving generative foundation model for synthetic medical image generation and clinical applications.

Journal: Nature medicine
PMID: 39663467

Abstract

In many clinical and research settings, the scarcity of high-quality medical imaging datasets has hampered the potential of artificial intelligence (AI) clinical applications. This issue is particularly pronounced in less common conditions, underrepresented populations and emerging imaging modalities, where the availability of diverse and comprehensive datasets is often inadequate. To address this challenge, we introduce a unified medical image-text generative model called MINIM that is capable of synthesizing medical images of various organs across various imaging modalities based on textual instructions. Clinician evaluations and rigorous objective measurements validate the high quality of MINIM's synthetic images. MINIM exhibits an enhanced generative capability when presented with previously unseen data domains, demonstrating its potential as a generalist medical AI (GMAI). Our findings show that MINIM's synthetic images effectively augment existing datasets, boosting performance across multiple medical applications such as diagnostics, report generation and self-supervised learning. On average, MINIM enhances performance by 12% for ophthalmic, 15% for chest, 13% for brain and 17% for breast-related tasks. Furthermore, we demonstrate MINIM's potential clinical utility in the accurate prediction of HER2-positive breast cancer from MRI images. Using a large retrospective simulation analysis, we demonstrate MINIM's clinical potential by accurately identifying targeted therapy-sensitive EGFR mutations using lung cancer computed tomography images, which could potentially lead to improved 5-year survival rates. Although these results are promising, further validation and refinement in more diverse and prospective settings would greatly enhance the model's generalizability and robustness.

Authors

  • Jinzhuo Wang
    Department of Big Data and Biomedical AI, College of Future Technology, Peking University, Beijing, China.
  • Kai Wang
    Department of Rheumatology, The Affiliated Huai'an No. 1 People's Hospital of Nanjing Medical University, Huai'an, Jiangsu, China.
  • Yunfang Yu
    Guangdong Provincial Key Laboratory of Malignant Tumor Epigenetics and Gene Regulation, Guangdong-Hong Kong Joint Laboratory for RNA Medicine, Department of Medical Oncology, Breast Tumor Centre, Phase I Clinical Trial Centre, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, China; Guangdong Provincial Key Laboratory of Cancer Pathogenesis and Precision Diagnosis and Treatment, Joint Big Data Laboratory, Department of Medical Oncology, Shenshan Medical Center, Memorial Hospital of Sun Yat-sen University, Shanwei, China; Institute for AI in Medicine and faculty of Medicine, Macau University of Science and Technology, Taipa, Macao, China; Department of Breast Surgery, The First Affiliated Hospital, Jinan University, Guangzhou, China.
  • Yuxing Lu
    Department of Big Data and Biomedical AI, College of Future Technology, Peking University, Beijing 100091, China.
  • Wenchao Xiao
    Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China.
  • Zhuo Sun
    State Key Laboratory of Eye Health, Eye Hospital and Institute for Advanced Study on Eye Health and Diseases, Wenzhou Medical University, Wenzhou, China; Department of Ophthalmology, The Third People's Hospital of Changzhou, Changzhou, China.
  • Fei Liu
    Department of Interventional Radiology, Qinghai Red Cross Hospital, Xining, Qinghai, China.
  • Zixing Zou
    Institute for AI in Medicine and Faculty of Medicine, Macau University of Science and Technology, Macau, China; State Key Laboratory of Eye Health, Eye Hospital and Institute for Advanced Study on Eye Health and Diseases, Wenzhou Medical University, Wenzhou, China; National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China; Guangzhou National Laboratory, Guangzhou, China.
  • Yuanxu Gao
    Institute for AI in Medicine and Faculty of Medicine, Macau University of Science and Technology, Macau, China; State Key Laboratory of Eye Health, Eye Hospital and Institute for Advanced Study on Eye Health and Diseases, Wenzhou Medical University, Wenzhou, China; National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China; Department of Big Data and Biomedical AI, College of Future Technology, Peking University and Peking-Tsinghua Center for Life Sciences, Beijing, China.
  • Lei Yang
    George Mason University.
  • Hong-Yu Zhou
    Department of Respiratory and Critical Care Medicine, National Clinical Research Center for Geriatrics, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu 610041, P.R. China; Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong.
  • Hanpei Miao
    Dongguan Hospital, Southern Medical University, Dongguan, China.
  • Wenting Zhao
    First Clinical College, Hubei University of Chinese Medicine, Wuhan, China.
  • Lisha Huang
    Department of Bioinformatics, Zhejiang Shaoxing Topgen Biomedical Technology CO., LTD, Shanghai, China.
  • Lingchao Zeng
    Department of Ophthalmology, Zhuhai People's Hospital and the First Affiliated Hospital of Faculty of Medicine, Macau University of Science and Technology, Zhuhai, China.
  • Rui Guo
    College of Chemistry&Chemical Engineering, Xiamen University, Xiamen 361005, China.
  • Ieng Chong
    Faculty of Innovation Engineering, Macau University of Science and Technology, Macau, China.
  • Boyu Deng
    Guangzhou National Laboratory, Guangzhou, China.
  • Linling Cheng
    Instutite for Artificial Intelligence in Medicine and Faculty of Medicine, Macau University of Science and Technology, Macau, China.
  • Xiaoniao Chen
    National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China.
  • Jing Luo
    Department of Ophthalmology, The Second Xiangya Hospital of Central South University, 139 Middle Renmin RD, Changsha, Hunan, China.
  • Meng-Hua Zhu
    Research Center of Blood Transfusion Medicine, Education Ministry Key Laboratory of Laboratory Medicine, Zhejiang Provincial People's Hospital, Hangzhou, 310014, China. Electronic address: mmizjpph89@163.com.
  • Daniel Baptista-Hon
    Institute for AI in Medicine, Faculty of Medicine, Macau University of Science and Technology, Taipa, Macau, China.
  • Olivia Monteiro
    Center for Biomedicine and Innovations, Faculty of Medicine, Macau University of Science and Technology, Macao 999078, China.
  • Ming Li
    Radiology Department, Huadong Hospital, Affiliated with Fudan University, Shanghai, China.
  • Yu Ke
    Dongguan People's Hospital, Southern Medical University, Dongguan, China.
  • Jiahui Li
    College of Communication Engineering, Jilin University, Changchun, Jilin China.
  • Simiao Zeng
    Department of Reproductive Medicine, Guangzhou Women and Children's Medical Center, Guangzhou Medical University, Guangzhou, Guangdong 510623, China.
  • Taihua Guan
    Guangzhou National Laboratory, Guangzhou, China.
  • Jin Zeng
    Department of Spine Surgery, The Third Xiangya Hospital of Central South University, 138 Tongzipo Rd, Changsha, 410013, Hunan, China.
  • Kanmin Xue
    Nuffield Department of Neuroscience, Oxford University, Oxford, UK.
  • Eric Oermann
    NYU Langone Medical Center, New York University, New York, NY, USA.
  • Huiyan Luo
    Department of Medical Oncology, State Key Laboratory of Oncology in South China, Collaborative Innovation Center for Cancer Medicine, Sun Yat-sen University Cancer Center, Guangzhou, China.
  • Yun Yin
    Faculty of Health and Wellness, Faculty of Business, City University of Macau, Macau, China.
  • Kang Zhang
    Xifeng District People's Hospital, Qingyang, China.
  • Jia Qu
    National Clinical Research Center for Ocular Diseases, Eye Hospital, Wenzhou Medical University, Wenzhou, China. Electronic address: jia.qu@163.com.

Keywords

External Resources

Popular Topics
Recent Journals