ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathology
Journal:
arXiv
Published Date:
Mar 21, 2025
Abstract
Prediction tasks in digital pathology are challenging due to the massive size
of whole-slide images (WSIs) and the weak nature of training signals. Advances
in computing, data availability, and self-supervised learning (SSL) have paved
the way for slide-level foundation models (SLFMs) that can improve prediction
tasks in low-data regimes. However, working with these models is challenging,
with issues such as catastrophic forgetting during fine-tuning and
under-utilization of shared information between tasks and modalities. To
overcome these two challenges, we propose ModalTune, a novel fine-tuning
framework which introduces the Modal Adapter to integrate new modalities
without modifying SLFM weights. Additionally, we use large-language models
(LLMs) to encode labels as text, capturing semantic relationships and enhancing
generalization across multiple tasks and cancer types in a single training
recipe. ModalTune achieves state-of-the-art (SOTA) results against both
uni-modal and multi-modal models across four cancer types, jointly improving
survival and cancer subtype prediction while remaining competitive in
pan-cancer settings. Additionally, we show ModalTune is highly generalizable to
two out-of-distribution (OOD) datasets. To our knowledge, this is the first
unified fine-tuning framework for multi-modal, multi-task, and pan-cancer
modeling in digital pathology.
