Reading the (functional) writing on the (structural) wall: Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia.

Journal: NeuroImage
Published Date: Jul 25, 2018

Abstract

This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns "alignments" (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.

Authors

  • Sergey M Plis
    Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, 30303, USA.
  • Md Faijul Amin
    The Mind Research Network, 1101 Yale Blvd, Albuquerque, NM, 87106, USA; Intel Corporation, 5000 W Chandler Blvd, Chandler, AZ, 85226, USA.
  • Adam Chekroud
    Human Neuroscience Laboratory, Yale University, New Haven, CT, 06520, USA.
  • Devon Hjelm
    The Mind Research Network, 1101 Yale Blvd, Albuquerque, NM, 87106, USA.
  • Eswar Damaraju
    The Mind Research Network, 1101 Yale Blvd. NE, Albuquerque, NM 87106, USA; Department of Electrical and Computer Engineering, University of New Mexico, Albuquerque, NM 87106, USA.
  • Hyo Jong Lee
    Division of Computer Science and Engineering, CAIIT, Chonbuk National University, Jeonju, South Korea.
  • Juan R Bustillo
    Department of Psychiatry and Neurosciences, University of New Mexico, Albuquerque, NM, 87131, USA.
  • Kyunghyun Cho
    Department of Information and Computer Science, Aalto University School of Science, Finland.
  • Godfrey D Pearlson
    Olin Neuropsychiatry Research Center, Hartford Hospital (IOL Campus), Hartford, CT, USA; Department of Psychiatry and Neurobiology, Yale University School of Medicine, New Haven, CT, USA.
  • Vince D Calhoun
    Mind Research Network & Lovelace Biomedical and Environmental Research Institute, Albuquerque, New Mexico; Department of Psychiatry and Behavioral Sciences, University of New Mexico, Albuquerque, New Mexico; Department of Electrical and Computer Engineering, University of New Mexico, Albuquerque, New Mexico; Department of Neurosciences, University of New Mexico, Albuquerque, New Mexico.

Keywords

External Resources

Popular Topics
Recent Journals