Driver Emotion Recognition Using Multimodal Signals by Combining Conformer and Autoformer.

Journal: International journal of neural systems
Published Date: Sep 25, 2025

Abstract

This study aims to develop a multimodal driver emotion recognition system that accurately identifies a driver's emotional state during the driving process by integrating facial expressions, ElectroCardioGram (ECG) and ElectroEncephaloGram (EEG) signals. Specifically, this study proposes a model that employs a Conformer for analyzing facial images to extract visual cues related to the driver's emotions. Additionally, two Autoformers are utilized to process ECG and EEG signals. The embeddings from these three modalities are then fused using a cross-attention mechanism. The integrated features from the cross-attention mechanism are passed through a fully connected layer and classified to determine the driver's emotional state. The experimental results demonstrate that the fusion of visual, physiological and neurological modalities significantly improves the reliability and accuracy of emotion detection. The proposed approach not only offers insights into the emotional processes critical for driver assistance systems and vehicle safety but also lays the foundation for further advancements in emotion recognition area.

Authors

  • Weiguang Wang
    Decision, Operations and Information Technologies Department, Robert H. Smith School of Business, University of Maryland, College Park, Maryland, USA.
  • Jian Lian
    School of Intelligence Engineering, Shandong Management University, Jinan, China.
  • Chuanjie Xu
    Shandong Provincial Key Laboratory for Novel Distributed Computer Software Technology, Jinan, China.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals