VisTA: Vision-Text Alignment Model with Contrastive Learning using Multimodal Data for Evidence-Driven, Reliable, and Explainable Alzheimer's Disease Diagnosis
Journal:
arXiv
Published Date:
Feb 3, 2025
Abstract
Objective: Assessing Alzheimer's disease (AD) using high-dimensional
radiology images is clinically important but challenging. Although Artificial
Intelligence (AI) has advanced AD diagnosis, it remains unclear how to design
AI models embracing predictability and explainability. Here, we propose VisTA,
a multimodal language-vision model assisted by contrastive learning, to
optimize disease prediction and evidence-based, interpretable explanations for
clinical decision-making.
Methods: We developed VisTA (Vision-Text Alignment Model) for AD diagnosis.
Architecturally, we built VisTA from BiomedCLIP and fine-tuned it using
contrastive learning to align images with verified abnormalities and their
descriptions. To train VisTA, we used a constructed reference dataset
containing images, abnormality types, and descriptions verified by medical
experts. VisTA produces four outputs: predicted abnormality type, similarity to
reference cases, evidence-driven explanation, and final AD diagnoses. To
illustrate VisTA's efficacy, we reported accuracy metrics for abnormality
retrieval and dementia prediction. To demonstrate VisTA's explainability, we
compared its explanations with human experts' explanations.
Results: Compared to 15 million images used for baseline pretraining, VisTA
only used 170 samples for fine-tuning and obtained significant improvement in
abnormality retrieval and dementia prediction. For abnormality retrieval, VisTA
reached 74% accuracy and an AUC of 0.87 (26% and 0.74, respectively, from
baseline models). For dementia prediction, VisTA achieved 88% accuracy and an
AUC of 0.82 (30% and 0.57, respectively, from baseline models). The generated
explanations agreed strongly with human experts' and provided insights into the
diagnostic process. Taken together, VisTA optimize prediction, clinical
reasoning, and explanation.
