MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classification
Journal:
arXiv
Published Date:
Feb 11, 2025
Abstract
Whole slide pathology image classification presents challenges due to
gigapixel image sizes and limited annotation labels, hindering model
generalization. This paper introduces a prompt learning method to adapt large
vision-language models for few-shot pathology classification. We first extend
the Prov-GigaPath vision foundation model, pre-trained on 1.3 billion pathology
image tiles, into a vision-language model by adding adaptors and aligning it
with medical text encoders via contrastive learning on 923K image-text pairs.
The model is then used to extract visual features and text embeddings from
few-shot annotations and fine-tunes with learnable prompt embeddings. Unlike
prior methods that combine prompts with frozen features using prefix embeddings
or self-attention, we propose multi-granular attention that compares
interactions between learnable prompts with individual image patches and groups
of them. This approach improves the model's ability to capture both
fine-grained details and broader context, enhancing its recognition of complex
patterns across sub-regions. To further improve accuracy, we leverage
(unbalanced) optimal transport-based visual-text distance to secure model
robustness by mitigating perturbations that might occur during the data
augmentation process. Empirical experiments on lung, kidney, and breast
pathology modalities validate the effectiveness of our approach; thereby, we
surpass several of the latest competitors and consistently improve performance
across diverse architectures, including CLIP, PLIP, and Prov-GigaPath
integrated PLIP. We release our implementations and pre-trained models at this
MGPATH.
