Foundation Model-Enabled Multimodal Deep Learning for Prognostic Prediction in Colorectal Cancer with Incomplete Modalities: A Multi-Institutional Retrospective Study.

Journal: Advanced science (Weinheim, Baden-Wurttemberg, Germany)
Published Date: Jan 20, 2026

Abstract

Accurate prognostic prediction for colorectal cancer is essential for optimizing personalized treatment strategies and improving patient outcomes. Current unimodal approaches encounter significant limitations in effectively leveraging multimodal data and confront challenges with the issue of missing modalities. A novel multimodal deep learning framework named FLARE, which integrates pathological images, radiological imaging, and clinical text reports, is introduced to provide accurate risk assessments for colorectal cancer survival and progression. FLARE employs foundation models to achieve efficient feature extraction, utilizes an attention-based multi-branch framework to enhance synergy and distinctiveness across modalities, and incorporates a diversity-promoting loss function. To address the issue of incomplete data, FLARE integrates modality and missing-aware prompts, pseudo embeddings, and a modality-level augmentation strategy, thereby effectively mitigating potential performance degradation. The performance of FLARE is retrospectively assessed using a dataset of 1679 colorectal cancer patients from four independent clinical centers. Its superior prognostic capability is demonstrated through Kaplan-Meier analysis and the concordance index. FLARE effectively stratified patients into high- and low-risk groups. It achieved the highest concordance index across all validation cohorts, significantly outperforming traditional clinical models and existing multimodal methods, thereby highlighting its robust generalizability. Interpretability was enhanced by the comprehensive analyses of clinical factors, immune infiltration patterns, and gene pathways, as well as visualizations of feature importance across multiple modalities. In summary, FLARE establishes a comprehensive and robust framework for multimodal deep learning in medical prognostics, providing an advanced Artificial intelligence, Multimodal Deep Learning, Prognosis prediction, colorectal cancer, foundation modeltool for precision cancer prognosis and intelligent diagnosis.

Authors

  • Linhao Qu
    Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China.
  • Chengsheng Zhang
    Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China.
  • Yingyong Hou
    Department of Pathology, Zhongshan Hospital, Fudan University, Shanghai, 200032, China.
  • Feng Tang
    Department of Materials Science and Technology, Laser Fusion Research Center, China Academy of Engineering Physics, Mianyang 621900, China.
  • Weiqi Sheng
    Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China; Department of Oncology, Fudan University Shanghai Medical College, Shanghai, China; Institute of Pathology, Fudan University, Shanghai, China. Electronic address: [email protected].
  • Dan Huang
    Department of Anesthesiology, The Second Affiliated Hospital of Soochow University, Suzhou 215004, China.; Department of Anesthesiology, Renji Hospital, School of Medicine, Shanghai Jiaotong University, Shanghai 200127, China.
  • Zhijian Song
    Digital Medical Research Center, Fudan University, Shanghai, China.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals