UMIT: Unifying Medical Imaging Tasks via Vision-Language Models
Journal:
arXiv
Published Date:
Mar 20, 2025
Abstract
With the rapid advancement of deep learning, particularly in the field of
medical image analysis, an increasing number of Vision-Language Models (VLMs)
are being widely applied to solve complex health and biomedical challenges.
However, existing research has primarily focused on specific tasks or single
modalities, which limits their applicability and generalization across diverse
medical scenarios. To address this challenge, we propose UMIT, a unified
multi-modal, multi-task VLM designed specifically for medical imaging tasks.
UMIT is able to solve various tasks, including visual question answering,
disease detection, and medical report generation. In addition, it is applicable
to multiple imaging modalities (e.g., X-ray, CT and PET), covering a wide range
of applications from basic diagnostics to complex lesion analysis. Moreover,
UMIT supports both English and Chinese, expanding its applicability globally
and ensuring accessibility to healthcare services in different linguistic
contexts. To enhance the model's adaptability and task-handling capability, we
design a unique two-stage training strategy and fine-tune UMIT with designed
instruction templates. Through extensive empirical evaluation, UMIT outperforms
previous methods in five tasks across multiple datasets. The performance of
UMIT indicates that it can significantly enhance diagnostic accuracy and
workflow efficiency, thus providing effective solutions for medical imaging
applications.
