The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review.

Journal: Diagnostic and interventional radiology (Ankara, Turkey)
Published Date: Oct 1, 2024

Abstract

With the ongoing revolution of artificial intelligence (AI) in medicine, the impact of AI in radiology is more pronounced than ever. An increasing number of technical and clinical AI-focused studies are published each day. As these tools inevitably affect patient care and physician practices, it is crucial that radiologists become more familiar with the leading strategies and underlying principles of AI. Multimodal AI models can combine both imaging and clinical metadata and are quickly becoming a popular approach that is being integrated into the medical ecosystem. This narrative review covers major concepts of multimodal AI through the lens of recent literature. We discuss emerging frameworks, including graph neural networks, which allow for explicit learning from non-Euclidean relationships, and transformers, which allow for parallel computation that scales, highlighting existing literature and advocating for a focus on emerging architectures. We also identify key pitfalls in current studies, including issues with taxonomy, data scarcity, and bias. By informing radiologists and biomedical AI experts about existing practices and challenges, we hope to guide the next wave of imaging-based multimodal AI research.

Authors

  • Benjamin D Simon
    Molecular Imaging Branch, NCI, NIH, Bethesda, Maryland, USA (B.D.S., K.M.M., S.A.H., E.C.Y., P.L.C., B.T.); Institute of Biomedical Engineering, Department Engineering Science, University of Oxford, UK (B.D.S.).
  • Kutsev Bengisu Ozyoruk
    Institute of Biomedical Engineering, Bogazici University, Istanbul, Turkey.
  • David G Gelikman
    Molecular Imaging Branch, National Cancer Institute, National Institutes of Health, 10 Center Dr., MSC 1182, Building 10, Room B3B85, Bethesda, MD, 20892, USA.
  • Stephanie A Harmon
    Clinical Research Directorate, Frederick National Laboratory for Cancer Research sponsored by the National Cancer Institute, Frederick, Maryland, USA.
  • Barış Türkbey
    Molecular Imaging Program, National Cancer Institute, National Institutes of Health, Bethesda, Maryland, USA.

Keywords

External Resources

Popular Topics
Recent Journals