CLIP-IT: CLIP-based Pairing for Histology Images Classification
Journal:
arXiv
Published Date:
Apr 22, 2025
Abstract
Multimodal learning has shown significant promise for improving medical image
analysis by integrating information from complementary data sources. This is
widely employed for training vision-language models (VLMs) for cancer detection
based on histology images and text reports. However, one of the main
limitations in training these VLMs is the requirement for large paired
datasets, raising concerns over privacy, and data collection, annotation, and
maintenance costs. To address this challenge, we introduce CLIP-IT method to
train a vision backbone model to classify histology images by pairing them with
privileged textual information from an external source. At first, the modality
pairing step relies on a CLIP-based model to match histology images with
semantically relevant textual report data from external sources, creating an
augmented multimodal dataset without the need for manually paired samples.
Then, we propose a multimodal training procedure that distills the knowledge
from the paired text modality to the unimodal image classifier for enhanced
performance without the need for the textual data during inference. A
parameter-efficient fine-tuning method is used to efficiently address the
misalignment between the main (image) and paired (text) modalities. During
inference, the improved unimodal histology classifier is used, with only
minimal additional computational complexity. Our experiments on challenging
PCAM, CRC, and BACH histology image datasets show that CLIP-IT can provide a
cost-effective approach to leverage privileged textual information and
outperform unimodal classifiers for histology.
