Interpretable Cross-Modal Alignment Network for EEG Visual Decoding With Algorithm Unrolling.

Journal: IEEE transactions on neural networks and learning systems
Published Date: Jul 30, 2025

Abstract

Accurate decoding in electroencephalography (EEG) technology, particularly for rapid visual stimuli, remains challenging due to the low signal-to-noise ratio (SNR). Additionally, existing neural networks struggle with issues related to generalization and interpretability. This article proposes a cross-modal aligned network, E2IVAE, which leverages shared information from multiple modalities for self-supervised alignment of EEG to images for extracting visual perceptual information and features a novel EEG encoder, ISTANet, based on algorithm unrolling. This network framework significantly enhances the accuracy and stability of EEG decoding for object recognition in novel classes while reducing the extensive neural data typically required for training neural decoders. The proposed ISTANet employs algorithm unrolling to transform the multilayer sparse coding algorithm into an end-to-end format, extracting features from noisy EEG signals while incorporating the interpretability of traditional machine learning. The experimental results demonstrate that our method achieves SOTA top-1 accuracy of 62.39% and top-5 accuracy of 88.98% on a comprehensive rapid serial visual presentation (RSVP) dataset for public comparison in a 200-class zero-shot neural decoding task. Additionally, ISTANet enables visualization and analysis of multiscale atom features and overall reconstruction features, exploring biological plausibility across temporal, spatial, and spectral dimensions. On another more challenging RSVP large-scale dataset, the proposed framework also achieves significantly above chance-level performance, proving its robustness and generalization. This research provides critical insights into neural decoding and brain-computer interfaces (BCIs) within the fields of cognitive science and artificial intelligence.

Authors

  • Daowen Xiong
    School of Life and Health Information Science and Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China.
  • Liangliang Hu
    College of Computer Science and Technology, ChongQing University of Posts and Telecommunications, ChongQing 400065, China.
  • Jiahao Jin
  • Yikang Ding
    School of Life and Health Information Science and Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China.
  • Congming Tan
    College of Computer Science and Technology, ChongQing University of Posts and Telecommunications, ChongQing 400065, China.
  • Jing Zhang
    MOEMIL Laboratory, School of Optoelectronic Information, University of Electronic Science and Technology of China, Chengdu, China.
  • Yin Tian

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals