BC-PMJRS: A Brain Computing-inspired Predefined Multimodal Joint Representation Spaces for enhanced cross-modal learning.
Journal:
Neural networks : the official journal of the International Neural Network Society
Published Date:
Apr 10, 2025
Abstract
Multimodal learning faces two key challenges: effectively fusing complex information from different modalities, and designing efficient mechanisms for cross-modal interactions. Inspired by neural plasticity and information processing principles in the human brain, this paper proposes BC-PMJRS, a Brain Computing-inspired Predefined Multimodal Joint Representation Spaces method to enhance cross-modal learning. The method learns the joint representation space through two complementary optimization objectives: (1) minimizing mutual information between representations of different modalities to reduce redundancy and (2) maximizing mutual information between joint representations and sentiment labels to improve task-specific discrimination. These objectives are balanced dynamically using an adaptive optimization strategy inspired by long-term potentiation (LTP) and long-term depression (LTD) mechanisms. Furthermore, we significantly reduce the computational complexity of modal interactions by leveraging a global-local cross-modal interaction mechanism, analogous to selective attention in the brain. Experimental results on the IEMOCAP, MOSI, and MOSEI datasets demonstrate that BC-PMJRS outperforms state-of-the-art models in both complete and incomplete modality settings, achieving up to a 1.9% improvement in weighted-F1 on IEMOCAP, a 2.8% gain in 7-class accuracy on MOSI, and a 2.9% increase in 7-class accuracy on MOSEI. These substantial improvements across multiple datasets demonstrate that incorporating brain-inspired mechanisms, particularly the dynamic balance of information redundancy and task relevance through neural plasticity principles, effectively enhances multimodal learning. This work bridges neuroscience principles with multimodal machine learning, offering new insights for developing more effective and biologically plausible models.
