Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers
Journal:
arXiv
Published Date:
Jan 30, 2025
Abstract
A patient undergoes multiple examinations in each hospital stay, where each
provides different facets of the health status. These assessments include
temporal data with varying sampling rates, discrete single-point measurements,
therapeutic interventions such as medication administration, and images. While
physicians are able to process and integrate diverse modalities intuitively,
neural networks need specific modeling for each modality complicating the
training procedure. We demonstrate that this complexity can be significantly
reduced by visualizing all information as images along with unstructured text
and subsequently training a conventional vision-text transformer. Our approach,
Vision Transformer for irregular sampled Multi-modal Measurements (ViTiMM), not
only simplifies data preprocessing and modeling but also outperforms current
state-of-the-art methods in predicting in-hospital mortality and phenotyping,
as evaluated on 6,175 patients from the MIMIC-IV dataset. The modalities
include patient's clinical measurements, medications, X-ray images, and
electrocardiography scans. We hope our work inspires advancements in
multi-modal medical AI by reducing the training complexity to (visual) prompt
engineering, thus lowering entry barriers and enabling no-code solutions for
training. The source code will be made publicly available.
