Spectral State Space Model for Rotation-Invariant Visual Representation Learning
Journal:
arXiv
Published Date:
Mar 9, 2025
Abstract
State Space Models (SSMs) have recently emerged as an alternative to Vision
Transformers (ViTs) due to their unique ability of modeling global
relationships with linear complexity. SSMs are specifically designed to capture
spatially proximate relationships of image patches. However, they fail to
identify relationships between conceptually related yet not adjacent patches.
This limitation arises from the non-causal nature of image data, which lacks
inherent directional relationships. Additionally, current vision-based SSMs are
highly sensitive to transformations such as rotation. Their predefined scanning
directions depend on the original image orientation, which can cause the model
to produce inconsistent patch-processing sequences after rotation. To address
these limitations, we introduce Spectral VMamba, a novel approach that
effectively captures the global structure within an image by leveraging
spectral information derived from the graph Laplacian of image patches. Through
spectral decomposition, our approach encodes patch relationships independently
of image orientation, achieving rotation invariance with the aid of our
Rotational Feature Normalizer (RFN) module. Our experiments on classification
tasks show that Spectral VMamba outperforms the leading SSM models in vision,
such as VMamba, while maintaining invariance to rotations and a providing a
similar runtime efficiency.
