Evaluating Vision Language Models (VLMs) for Radiology: A Comprehensive Analysis
Journal:
arXiv
Published Date:
Apr 22, 2025
Abstract
Foundation models, trained on vast amounts of data using self-supervised
techniques, have emerged as a promising frontier for advancing artificial
intelligence (AI) applications in medicine. This study evaluates three
different vision-language foundation models (RAD-DINO, CheXagent, and
BiomedCLIP) on their ability to capture fine-grained imaging features for
radiology tasks. The models were assessed across classification, segmentation,
and regression tasks for pneumothorax and cardiomegaly on chest radiographs.
Self-supervised RAD-DINO consistently excelled in segmentation tasks, while
text-supervised CheXagent demonstrated superior classification performance.
BiomedCLIP showed inconsistent performance across tasks. A custom segmentation
model that integrates global and local features substantially improved
performance for all foundation models, particularly for challenging
pneumothorax segmentation. The findings highlight that pre-training methodology
significantly influences model performance on specific downstream tasks. For
fine-grained segmentation tasks, models trained without text supervision
performed better, while text-supervised models offered advantages in
classification and interpretability. These insights provide guidance for
selecting foundation models based on specific clinical applications in
radiology.
