Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2
Journal:
arXiv
Published Date:
Jan 21, 2025
Abstract
Radiology plays a pivotal role in modern medicine due to its non-invasive
diagnostic capabilities. However, the manual generation of unstructured medical
reports is time consuming and prone to errors. It creates a significant
bottleneck in clinical workflows. Despite advancements in AI-generated
radiology reports, challenges remain in achieving detailed and accurate report
generation. In this study we have evaluated different combinations of
multimodal models that integrate Computer Vision and Natural Language
Processing to generate comprehensive radiology reports. We employed a
pretrained Vision Transformer (ViT-B16) and a SWIN Transformer as the image
encoders. The BART and GPT-2 models serve as the textual decoders. We used
Chest X-ray images and reports from the IU-Xray dataset to evaluate the
usability of the SWIN Transformer-BART, SWIN Transformer-GPT-2, ViT-B16-BART
and ViT-B16-GPT-2 models for report generation. We aimed at finding the best
combination among the models. The SWIN-BART model performs as the
best-performing model among the four models achieving remarkable results in
almost all the evaluation metrics like ROUGE, BLEU and BERTScore.
