Bridging Vision and Language: Optimal Transport-Driven Radiology Report Generation via LLMs
Journal:
arXiv
Published Date:
Jul 5, 2025
Abstract
Radiology report generation represents a significant application within
medical AI, and has achieved impressive results. Concurrently, large language
models (LLMs) have demonstrated remarkable performance across various domains.
However, empirical validation indicates that general LLMs tend to focus more on
linguistic fluency rather than clinical effectiveness, and lack the ability to
effectively capture the relationship between X-ray images and their
corresponding texts, thus resulting in poor clinical practicability. To address
these challenges, we propose Optimal Transport-Driven Radiology Report
Generation (OTDRG), a novel framework that leverages Optimal Transport (OT) to
align image features with disease labels extracted from reports, effectively
bridging the cross-modal gap. The core component of OTDRG is Alignment \&
Fine-Tuning, where OT utilizes results from the encoding of label features and
image visual features to minimize cross-modal distances, then integrating image
and text features for LLMs fine-tuning. Additionally, we design a novel disease
prediction module to predict disease labels contained in X-ray images during
validation and testing. Evaluated on the MIMIC-CXR and IU X-Ray datasets, OTDRG
achieves state-of-the-art performance in both natural language generation (NLG)
and clinical efficacy (CE) metrics, delivering reports that are not only
linguistically coherent but also clinically accurate.
