Bio-Plausible Multimodal Learning with Emerging Neuromorphic Devices.

Journal: Advanced science (Weinheim, Baden-Wurttemberg, Germany)
Published Date: Sep 11, 2024

Abstract

Multimodal machine learning, as a prospective advancement in artificial intelligence, endeavors to emulate the brain's multimodal learning abilities with the objective to enhance interactions with humans. However, this approach requires simultaneous processing of diverse types of data, leading to increased model complexity, longer training times, and higher energy consumption. Multimodal neuromorphic devices have the capability to preprocess spatio-temporal information from various physical signals into unified electrical signals with high information density, thereby enabling more biologically plausible multimodal learning with low complexity and high energy-efficiency. Here, this work conducts a comparison between the expression of multimodal machine learning and multimodal neuromorphic computing, followed by an overview of the key characteristics associated with multimodal neuromorphic devices. The bio-plausible operational principles and the multimodal learning abilities of emerging devices are examined, which are classified into heterogeneous and homogeneous multimodal neuromorphic devices. Subsequently, this work provides a detailed description of the multimodal learning capabilities demonstrated by neuromorphic circuits and their respective applications. Finally, this work highlights the limitations and challenges of multimodal neuromorphic computing in order to hopefully provide insight into potential future research directions.

Authors

  • Haonan Sun
    School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China.
  • Haoxiang Tian
    State Key Laboratory of Electronic Thin Film and Integrated Devices, University of Electronic Science and Technology of China, Chengdu, 611731, China.
  • Yihao Hu
    School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China.
  • Yi Cui
    Global Station for Quantum Medical Science and Engineering, Global Institution for Collaborative Research and Education (GI-CoRE), Proton Beam Therapy Center, North 14 West 5 Kita-ku, Sapporo, Hokkaido, 060-8648, Japan.
  • Xinrui Chen
    State Key Laboratory of Electronic Thin Film and Integrated Devices, University of Electronic Science and Technology of China, Chengdu, 611731, China.
  • Minyi Xu
    State Key Laboratory of Electronic Thin Film and Integrated Devices, University of Electronic Science and Technology of China, Chengdu, 611731, China.
  • Xianfu Wang
    State Key Laboratory of Electronic Thin Film and Integrated Devices, University of Electronic Science and Technology of China, Chengdu, 611731, China.
  • Tao Zhou
    Department of Otorhinolaryngology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, Hubei, China.

Keywords

External Resources

Popular Topics
Recent Journals