D-Rax: Domain-specific Radiologic assistant leveraging multi-modal data and eXpert model predictions
Journal:
arXiv
Published Date:
Jul 2, 2024
Abstract
Large vision language models (VLMs) have progressed incredibly from research
to applicability for general-purpose use cases. LLaVA-Med, a pioneering large
language and vision assistant for biomedicine, can perform multi-modal
biomedical image and data analysis to provide a natural language interface for
radiologists. While it is highly generalizable and works with multi-modal data,
it is currently limited by well-known challenges that exist in the large
language model space. Hallucinations and imprecision in responses can lead to
misdiagnosis which currently hinder the clinical adaptability of VLMs. To
create precise, user-friendly models in healthcare, we propose D-Rax -- a
domain-specific, conversational, radiologic assistance tool that can be used to
gain insights about a particular radiologic image. In this study, we enhance
the conversational analysis of chest X-ray (CXR) images to support radiological
reporting, offering comprehensive insights from medical imaging and aiding in
the formulation of accurate diagnosis. D-Rax is achieved by fine-tuning the
LLaVA-Med architecture on our curated enhanced instruction-following data,
comprising of images, instructions, as well as disease diagnosis and
demographic predictions derived from MIMIC-CXR imaging data, CXR-related visual
question answer (VQA) pairs, and predictive outcomes from multiple expert AI
models. We observe statistically significant improvement in responses when
evaluated for both open and close-ended conversations. Leveraging the power of
state-of-the-art diagnostic models combined with VLMs, D-Rax empowers
clinicians to interact with medical images using natural language, which could
potentially streamline their decision-making process, enhance diagnostic
accuracy, and conserve their time.
