DMF2Mel: A Dynamic Multiscale Fusion Network for EEG-Driven Mel Spectrogram Reconstruction
Journal:
arXiv
Published Date:
Jul 10, 2025
Abstract
Decoding speech from brain signals is a challenging research problem.
Although existing technologies have made progress in reconstructing the mel
spectrograms of auditory stimuli at the word or letter level, there remain core
challenges in the precise reconstruction of minute-level continuous imagined
speech: traditional models struggle to balance the efficiency of temporal
dependency modeling and information retention in long-sequence decoding. To
address this issue, this paper proposes the Dynamic Multiscale Fusion Network
(DMF2Mel), which consists of four core components: the Dynamic Contrastive
Feature Aggregation Module (DC-FAM), the Hierarchical Attention-Guided
Multi-Scale Network (HAMS-Net), the SplineMap attention mechanism, and the
bidirectional state space module (convMamba). Specifically, the DC-FAM
separates speech-related "foreground features" from noisy "background features"
through local convolution and global attention mechanisms, effectively
suppressing interference and enhancing the representation of transient signals.
HAMS-Net, based on the U-Net framework,achieves cross-scale fusion of
high-level semantics and low-level details. The SplineMap attention mechanism
integrates the Adaptive Gated Kolmogorov-Arnold Network (AGKAN) to combine
global context modeling with spline-based local fitting. The convMamba captures
long-range temporal dependencies with linear complexity and enhances nonlinear
dynamic modeling capabilities. Results on the SparrKULee dataset show that
DMF2Mel achieves a Pearson correlation coefficient of 0.074 in mel spectrogram
reconstruction for known subjects (a 48% improvement over the baseline) and
0.048 for unknown subjects (a 35% improvement over the baseline).Code is
available at: https://github.com/fchest/DMF2Mel.
