FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecasting
Journal:
arXiv
Published Date:
Jan 23, 2025
Abstract
This paper presents \textbf{FreEformer}, a simple yet effective model that
leverages a \textbf{Fre}quency \textbf{E}nhanced Trans\textbf{former} for
multivariate time series forecasting. Our work is based on the assumption that
the frequency spectrum provides a global perspective on the composition of
series across various frequencies and is highly suitable for robust
representation learning. Specifically, we first convert time series into the
complex frequency domain using the Discrete Fourier Transform (DFT). The
Transformer architecture is then applied to the frequency spectra to capture
cross-variate dependencies, with the real and imaginary parts processed
independently. However, we observe that the vanilla attention matrix exhibits a
low-rank characteristic, thus limiting representation diversity. This could be
attributed to the inherent sparsity of the frequency domain and the
strong-value-focused nature of Softmax in vanilla attention. To address this,
we enhance the vanilla attention mechanism by introducing an additional
learnable matrix to the original attention matrix, followed by row-wise L1
normalization. Theoretical analysis~demonstrates that this enhanced attention
mechanism improves both feature diversity and gradient flow. Extensive
experiments demonstrate that FreEformer consistently outperforms
state-of-the-art models on eighteen real-world benchmarks covering electricity,
traffic, weather, healthcare and finance. Notably, the enhanced attention
mechanism also consistently improves the performance of state-of-the-art
Transformer-based forecasters.
